Disk Monitoring

Disk Monitoring

Disk monitoring covers two distinct problems: disk space usage (running out of space crashes applications and corrupts writes) and disk I/O performance (slow disks cause processes to wait in uninterruptible sleep, degrading the entire system). Both need monitoring but require different tools. A third consideration is disk health: SMART monitoring provides early warning of hardware failure before a disk dies unexpectedly.

Disk space usage

df -h              # Disk space on all mounted filesystems
df -h /var/log     # Specific filesystem
du -sh /var/log/*  # Top-level directory sizes

df -h output

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        50G   38G   10G  80% /
/dev/sda2       200G  145G   45G  73% /var
tmpfs           3.9G  2.1G  1.8G  54% /dev/shm

→ / is at 80% — set alert at 85%, investigate at 90%

# Find the largest directories (when disk is filling):
du -sh /* 2>/dev/null | sort -rh | head -10    # Top-level
du -sh /var/* 2>/dev/null | sort -rh | head -10

# Find recently modified large files:
find / -size +100M -mtime -7 -type f 2>/dev/null | xargs ls -lh 2>/dev/null

Inode exhaustion

# Inode exhaustion: filesystem can't create new files even with free space
df -i    # Check inode usage (-i flag)

df -i output showing inode exhaustion

Filesystem      Inodes  IUsed  IFree IUse% Mounted on
/dev/sda1      3276800 3276799      1  100% /var/spool/mail
→ 100% inodes used! Cannot create any new files despite having disk space

# Find directory with most files (the inode hog):
find /var/spool -type d | while read dir; do
    count=$(ls "$dir" | wc -l)
    echo "$count $dir"
done | sort -rn | head -10

iostat for I/O performance

# Install sysstat for iostat:
sudo apt install -y sysstat

# Monitor disk I/O in real time:
iostat -xz 1 3    # Extended stats, skip idle devices, 1-second interval

iostat -xz output

Device   r/s   w/s  rkB/s  wkB/s  await  svctm  %util
sda     45.2  23.1  892.1  445.2   8.42   1.23   8.5%  ← healthy
sdb      1.2 892.4    4.5 8924.1  245.3  18.72  99.8%  ← saturated!

await: average I/O wait time in ms (>20ms for HDD = overloaded, >1ms for SSD = overloaded)
%util: disk utilization (>80% = bottleneck)

SMART health monitoring

# Install smartmontools for disk health monitoring:
sudo apt install -y smartmontools

# Check disk health:
sudo smartctl -H /dev/sda    # Quick health check

SMART health status

SMART overall-health self-assessment test result: PASSED
# If FAILED: disk is predicting imminent failure — backup immediately

# View SMART attributes (warning signs):
sudo smartctl -A /dev/sda | grep -E "Reallocated|Pending|Uncorrectable"
# Reallocated_Sector_Ct > 0 = sectors failed and replaced — disk is degrading
# Current_Pending_Sector > 0 = sectors about to fail

# Enable automatic SMART monitoring:
sudo systemctl enable --now smartd

Conclusion

Set up monitoring alerts for disk space above 85%, inode usage above 85% (run df -i in your monitoring, not just df -h), and SMART failures. For production servers, enable smartd for automatic disk health monitoring — it emails when a drive reports degraded SMART attributes, giving you time to replace the drive before it fails completely. Check iostat %util and await when applications are slow and CPU looks idle: the bottleneck may be disk I/O.