Monitoring disk usage is one of those tasks every IT team “does,” yet disk-related incidents still account for a disproportionate share of outages, performance degradations, and after-hours firefights. The reason is that basic capacity checks—“is the drive full?”—only cover a small part of what makes storage reliable. Most application-impacting storage failures and slowdowns show up first as rising latency, sustained queueing, uneven per-volume utilization, or silent error patterns long before you see a red disk icon.
This article lays out a practical framework for what to measure, why it matters, and how to interpret it across common environments (bare metal, virtualization, SAN/NAS, cloud volumes). It also provides concrete collection examples using native tools and common telemetry stacks. The aim is to help you build monitoring that answers the question you actually care about: “Is storage becoming a risk to availability or performance, and how soon?”
Start with a mental model: capacity, performance, and health
To monitor disk usage well, it helps to separate storage signals into three categories that interact but are not interchangeable.
Capacity metrics tell you how much space is allocated and consumed. They drive operational risk (running out of space), but they also influence performance because many filesystems and storage systems degrade as they fill.
Performance metrics describe how fast storage responds to I/O requests. The most actionable performance signals are latency (how long requests take), IOPS (how many operations per second), and throughput (how many bytes per second). These are tied to user-facing outcomes like page load time, job runtime, and database commit rates.
Health metrics indicate whether the underlying media and path are behaving reliably: error counters, SMART attributes, controller status, and symptoms like repeated timeouts or resets. Health is where you catch “it’s slow because the disk is dying” or “it’s slow because the path is flapping.”
If your monitoring overweights capacity and underweights performance and health, you end up reacting when it’s too late. The rest of this guide builds upward: first capacity (the most obvious risk), then performance (the most common source of “mystery slowness”), then health (the signals that prevent data-loss and unstable systems).
Define what you’re monitoring: device, filesystem, volume, or workload
A recurring problem in disk monitoring is mixing layers. A “disk” can mean a physical drive, a RAID virtual disk, a SAN LUN, a cloud block volume, a virtual disk (VMDK/VHDX), or a filesystem mount. Each layer can be the bottleneck, and each reports different metrics.
At the filesystem or mount level you care about free space, inode usage (on Linux), fragmentation patterns, and whether the filesystem is read-only due to errors. At the block device level you care about I/O timing, queueing, and errors/timeouts. At the storage array or cloud service level you care about provisioned limits (IOPS/throughput caps), throttling, and backend health.
When you set up monitoring, be explicit about which layer each metric comes from and how it maps to impact. A web server with /var on a separate volume might show plenty of free space on /, yet be minutes away from failing because /var (logs, temp files, caches) is at 99%. Similarly, a virtual machine might show normal “disk busy” inside the guest while the datastore is saturated and latency is exploding at the hypervisor layer.
A practical rule is to monitor capacity at the filesystem level (because applications fail when filesystems fill) and performance at both the filesystem/block level in the OS and at the upstream layer you don’t control (hypervisor datastore, SAN, cloud volume). This layering is essential when you need to prove where the bottleneck lives.
Capacity metrics: more than “percent full”
Most teams already alert on percent used, but percent used is a coarse proxy. Capacity-related incidents usually come from one of four patterns: fast growth, uneven distribution across mounts, hidden consumption (deleted-but-open files, snapshots), or metadata exhaustion (inodes). To catch these patterns early, you need a few additional metrics and some contextual thresholds.
Filesystem utilization and free space (absolute and relative)
Monitor both percentage used and absolute free space. Percent used is useful because some filesystems slow down near full (and storage arrays can behave poorly when thin pools are nearly exhausted). Absolute free space is useful because a 2 TB volume at 95% still has 100 GB free, while a 50 GB system disk at 95% has only 2.5 GB free and may fail imminently.
The most reliable alerting scheme combines a percentage threshold and an absolute threshold, for example: alert if used% > 85% AND free < 20 GB for large data volumes, and used% > 90% OR free < 5 GB for OS volumes. The exact numbers depend on your workloads and how long it takes you to remediate.
On Linux, basic usage is straightforward:
df -hT
On Windows, PowerShell can query logical disks:
powershell
Get-CimInstance Win32_LogicalDisk -Filter "DriveType=3" |
Select-Object DeviceID, VolumeName,
@{n='SizeGB';e={[math]::Round($_.Size/1GB,1)}},
@{n='FreeGB';e={[math]::Round($_.FreeSpace/1GB,1)}},
@{n='UsedPct';e={[math]::Round((1-($_.FreeSpace/$_.Size))*100,1)}} |
Sort-Object UsedPct -Descending
Those commands are not “monitoring” by themselves, but they show what your monitoring system should continuously collect: size, free, and percent used per mount/drive.
Growth rate (slope) and time-to-full
Alerting at 90% used catches the problem late; alerting on the rate of change catches it early. Growth rate is the derivative of used bytes over time. Time-to-full estimates when a filesystem will hit a threshold given the current growth trend.
Time-to-full isn’t perfect—growth can be bursty—but it’s operationally useful. If /var grows 30 GB/day due to debug logging, you want an alert that says “7 days to 95%,” not “95% reached at 3 AM.” Many monitoring platforms can compute slope or prediction; if yours cannot, you can still approximate with scheduled scripts exporting “used bytes” and doing the math centrally.
The key is to pick a window that matches reality. For logs and backups, 24–72 hour trends often capture the effect. For steady database growth, weekly trends might be better. Avoid mixing short-term spikes with long-term trends in a single predictor.
Inodes (Linux) and metadata exhaustion
On Linux filesystems like ext4 and XFS, you can run out of inodes (file metadata structures) even when there’s plenty of space. This happens with workloads that create huge numbers of small files: mail spools, container layers, build caches, package managers, or misbehaving apps dumping per-request temp files.
Monitor inode utilization where it matters:
bash
df -ih
If you’ve never alerted on inodes and you operate CI servers, container hosts, or mail systems, inode monitoring is low effort and high payoff. When inode exhaustion hits, common symptoms include “No space left on device” even though df -h shows free space.
Reserved space, thin provisioning, and snapshots
Many filesystems and storage systems reserve space that isn’t visible as “free.” ext4 typically reserves a percentage for root; LVM thin pools have metadata and data space; copy-on-write systems (ZFS, btrfs) and many SAN/NAS platforms have snapshot space that can grow quickly.
In virtualization, thin-provisioned disks and thin pools introduce a second capacity limit: the backing datastore or pool. A VM can report 60% used while the datastore is 98% and one large write triggers “no space” at the hypervisor layer.
The practical takeaway is that “monitor disk usage” should include at least one upstream capacity metric for shared pools: datastore free space, thin pool data %, thin pool metadata %, and snapshot reserve. Where available, collect these from the authoritative layer (vCenter, storage array, cloud provider metrics) rather than inferring them from guest OS data.
Deleted-but-open files (Linux) and log rotation pitfalls
A subtle but common capacity trap in Linux is deleted files that remain open by a process. Disk space isn’t reclaimed until the last file handle closes, so df shows the filesystem still full even after you delete large logs.
You can detect this condition when investigating unexpected usage by checking open deleted files:
bash
sudo lsof +L1
This is not a metric you alert on continuously in most environments, but it is a capability you should have ready because it explains a class of “we deleted files but nothing changed” incidents.
Performance metrics: the signals that predict user impact
Capacity tells you when you might run out of space; performance metrics tell you when users will feel pain. In practice, most “disk issues” presented to on-call engineers are performance issues: slow queries, timeouts, backlogged jobs, delayed logins. The fastest way to triage is to look at latency, IOPS, throughput, queueing, and saturation indicators.
Latency: the most important disk performance metric
Latency is the time it takes to complete an I/O request. It is typically measured separately for reads and writes, and often summarized as an average and percentiles (p95, p99). Percentiles matter because averages hide spikes that cause tail-latency problems in applications.
You can have “acceptable” average latency while p99 latency is terrible, which is exactly the pattern that causes intermittent timeouts in databases and microservices. For storage monitoring, aim to collect at least average and p95, and if your tooling supports it, p99.
What counts as “good” latency depends on the storage type and workload. NVMe local disks might deliver sub-millisecond reads under moderate load, while network-attached storage could be several milliseconds even when healthy. Databases and virtualization are especially sensitive to write latency. Rather than hard-coding a single “2 ms is good” target, establish a baseline for each storage tier and alert on deviation plus absolute guardrails.
On Linux, iostat provides a quick look at average await time (roughly, average time for I/O requests):
bash
iostat -x 1 5
Key columns include r_await, w_await, and %util (utilization). Treat these as indicators, not perfect truth; they’re still among the most useful first checks.
On Windows, PerfMon counters such as \LogicalDisk(*)\Avg. Disk sec/Read and \LogicalDisk(*)\Avg. Disk sec/Write are commonly used. In PowerShell, you can sample them:
powershell
Get-Counter '\LogicalDisk(*)\Avg. Disk sec/Read','\LogicalDisk(*)\Avg. Disk sec/Write' -SampleInterval 1 -MaxSamples 5
These counters report seconds; 0.005 is 5 ms. For many server workloads, sustained average latencies above ~20 ms usually indicate a problem, but treat that as a starting point and validate against your baselines.
IOPS: how many operations you’re driving
IOPS (I/O operations per second) measures the number of read/write operations completed per second. It’s central because many platforms have explicit IOPS limits: cloud block volumes, midrange arrays, and even local RAID controllers under certain conditions.
But IOPS alone can mislead. 5,000 IOPS of 4K random reads is a very different workload from 5,000 IOPS of 256K sequential writes. That’s why IOPS needs to be interpreted alongside throughput and request size.
On Linux, iostat -x shows r/s and w/s. On Windows, PerfMon includes \LogicalDisk(*)\Disk Reads/sec and \LogicalDisk(*)\Disk Writes/sec.
In monitoring systems, tracking IOPS helps you answer: is the system suddenly doing more I/O than usual, and if so, why? It also helps with capacity planning for cloud volumes where you must choose a size or tier to achieve the required IOPS.
Throughput: bytes per second and the sequential story
Throughput measures how many bytes per second are read/written. It is the metric that reveals large scans, backups, replications, and ETL jobs. Many “midnight slowdowns” are throughput-driven: a backup job saturates throughput on a volume, raising latency for everything else.
On Linux, iostat provides rMB/s and wMB/s (or rkB/s and wkB/s depending on version). On Windows, \LogicalDisk(*)\Disk Read Bytes/sec and \LogicalDisk(*)\Disk Write Bytes/sec are the equivalents.
Throughput also matters because some storage platforms have separate throughput caps independent of IOPS. Cloud volumes and managed disks often publish both numbers. If you hit a throughput ceiling, latency rises sharply even if IOPS seems moderate.
Queue depth and saturation: when requests are waiting
Queue depth is the number of outstanding I/O requests waiting to be served. High queue depth can be normal under heavy throughput if latency remains stable, but rising queue depth combined with rising latency indicates saturation.
In Windows PerfMon, the classic indicator is \LogicalDisk(*)\Current Disk Queue Length. On Linux, iostat -x provides avgqu-sz. Neither is a perfect apples-to-apples value across devices, but both show whether I/O is backing up.
An important nuance is that modern storage stacks can queue aggressively, and some devices perform best with some queueing. That’s why queue length should be interpreted alongside latency and utilization. A queue length of 10 on a single SATA disk might be a red flag; on a high-end NVMe or an array front-end, it might be normal.
Utilization (%busy) and iowait: understand what they do (and don’t) mean
Linux %util from iostat is often read as “disk is maxed.” On a single physical disk, sustained %util near 100% does suggest saturation. On device-mapper layers, RAID, SAN LUNs, and especially NVMe, %util can be less intuitive.
Similarly, CPU iowait (%iowait in top/vmstat) means the CPU has threads waiting on I/O, but it doesn’t tell you which disk or whether the bottleneck is local disk, network storage, or a filesystem lock. iowait is a symptom metric: it is useful to confirm that storage is a likely contributor to slowness, but it’s not enough to pinpoint.
Use iowait and utilization as supporting signals. Lead with latency and queueing when diagnosing impact.
Read/write mix, request size, and randomness: why workloads behave differently
Two environments can show the same IOPS and throughput and still behave very differently because of the I/O pattern. Random I/O generally costs more than sequential I/O. Small I/O tends to be latency-sensitive; large I/O tends to be throughput-sensitive.
Many tools expose “average request size” or “average I/O size.” In iostat -x, look at rareq-sz and wareq-sz (or avgrq-sz on some variants). A sudden drop in request size can indicate a workload shift from sequential to random patterns, often driven by index rebuilds, new query plans, or virtualization host contention.
If you operate databases or virtualization platforms, track read/write ratios and request sizes per volume. Over time you’ll develop an intuition: “this datastore normally does 128K writes at night; today it’s doing 4K random writes and latency is up.” That kind of pattern recognition is what turns metrics into operational advantage.
Health and reliability metrics: errors, SMART, and the path to storage
Performance metrics tell you that storage is slow. Health metrics tell you whether storage is failing or the path to it is unstable. Collecting health signals is also how you prevent latent data corruption and repeated performance incidents caused by retries.
SMART and media indicators (where applicable)
SMART (Self-Monitoring, Analysis, and Reporting Technology) is a set of attributes reported by many SATA/SAS drives. NVMe has its own SMART log via nvme-cli. While SMART is not perfect, certain attributes are strongly associated with impending drive failure: reallocated sectors, pending sectors, media errors, and increasing error rates.
On Linux, smartctl (from smartmontools) is a standard tool:
bash
sudo smartctl -a /dev/sda
For NVMe:
bash
sudo nvme smart-log /dev/nvme0
In many data center environments (RAID controllers, SANs), the OS may not see physical drives directly, so SMART from the host is limited or unavailable. In those cases, rely on the controller or array’s health telemetry.
Filesystem error states and remounts
Filesystems may remount read-only after encountering errors. That is a reliability event, and it often looks like an application outage rather than a “disk issue.” Monitor syslog/journal and Windows Event Logs for storage-related errors, and monitor mount states when possible.
On Linux, dmesg and journalctl provide evidence of I/O errors, resets, and timeouts:
bash
sudo journalctl -k --since "1 hour ago" | egrep -i 'error|timeout|reset|I/O'
These logs should be fed into your logging pipeline and correlated with latency spikes. A single timeout might be noise; repeated timeouts plus rising latency is a strong signal of a path or device issue.
Path health: multipath, HBA, and network storage
In SAN environments, multipathing and HBA (Host Bus Adapter) behavior matters. A path flapping can double latency due to failover and retries. Similarly, iSCSI/NFS/SMB storage depends on network health; packet loss and microbursts show up as storage latency.
If you rely on network-attached storage, do not treat disk metrics in isolation. Collect network interface errors, drops, and latency, and correlate them with storage latency. This is especially important for VM datastores, shared file shares, and container persistent volumes backed by network storage.
Establish baselines before you set aggressive alerts
The most common failure mode in disk monitoring is either no alerts until disaster, or too many noisy alerts that engineers ignore. Both happen when thresholds are chosen without baselines.
A baseline is what “normal” looks like for a specific system, volume, and time of day. Storage is often cyclical: nightly backups, weekly batch jobs, end-of-month reporting. If you set a single static threshold on throughput or IOPS, you will page people every night for expected behavior.
Start by collecting metrics at reasonable resolution (for example, 10–60 seconds for performance counters, 5 minutes for capacity) and keep at least a few weeks of history. Then define alerting in layers.
First, define “hard stop” alerts: low free space on system volumes, sustained high write latency on database volumes, thin pool nearly full. These should be rare and urgent.
Second, define “degradation” alerts based on baseline deviation: latency 2–3× normal for the time-of-day, queue depth rising alongside latency, or error rates exceeding a low baseline.
Third, define “planning” alerts: time-to-full under 14 or 30 days, growing snapshot usage, steady increase in average latency month over month.
This layered approach is how you keep pages meaningful while still improving long-term reliability.
Collecting disk usage metrics on Linux in a repeatable way
Linux offers many tools, but it’s easy to create inconsistent metrics if each engineer runs a different command. Standardize on a small set of collection methods that can be automated and shipped to your monitoring system.
Use df for filesystem capacity, not devices
df reports filesystem usage, which aligns with application failure modes. Collect size, used, available, and used% per mountpoint. Also collect filesystem type (-T) so you can interpret behavior (ext4 vs xfs vs nfs).
bash
df -PT -B1
-B1 outputs bytes, which is best for monitoring pipelines.
Use iostat -x for device-level performance overview
iostat -x provides extended device stats. It’s a good first layer for identifying which device is slow and whether the problem is read-heavy, write-heavy, or mixed.
bash
iostat -dx 5 3
Use the interval and count parameters to capture a short sample. For continuous monitoring, use an agent (node_exporter, Telegraf, etc.) rather than running iostat repeatedly via cron.
Use pidstat/iotop tactically to identify noisy processes
When you see high I/O, the next question is “who is doing it?” Tools like iotop and pidstat -d can identify processes generating I/O. They’re not always suitable for continuous monitoring, but they are useful to keep in your runbook.
bash
sudo pidstat -d 1 10
This helps bridge metrics (latency is high) to action (which service is causing the writes).
Collecting disk usage metrics on Windows with PerfMon and PowerShell
Windows storage monitoring typically uses PerfMon counters and event logs. The challenge is choosing counters that map to the right layer (logical disk vs physical disk) and accounting for caching.
Prefer logical disk counters for application impact
Logical disk counters map to volumes (drive letters or mount points), which is usually what you want when an application depends on a specific volume.
Commonly collected counters include:
\LogicalDisk(*)\% Free Space\LogicalDisk(*)\Free Megabytes\LogicalDisk(*)\Avg. Disk sec/Read\LogicalDisk(*)\Avg. Disk sec/Write\LogicalDisk(*)\Disk Reads/secand\LogicalDisk(*)\Disk Writes/sec\LogicalDisk(*)\Disk Read Bytes/secand\LogicalDisk(*)\Disk Write Bytes/sec
You can query a subset quickly:
powershell
$paths = @(
'\LogicalDisk(*)\% Free Space',
'\LogicalDisk(*)\Free Megabytes',
'\LogicalDisk(*)\Avg. Disk sec/Read',
'\LogicalDisk(*)\Avg. Disk sec/Write'
)
Get-Counter -Counter $paths -SampleInterval 5 -MaxSamples 3
For long-term monitoring, export these counters via your monitoring agent rather than running ad-hoc sampling.
Use physical disk counters when you need to map to a device
Physical disk counters can help correlate to a specific disk/LUN, especially on systems with multiple storage devices. They can be harder to interpret in virtualization or SAN setups where the “physical disk” is abstract.
When you do use them, keep the mapping between volume and disk clear (Disk Management, Get-Disk, Get-Partition, Get-Volume). A repeatable mapping is crucial during incidents.
powershell
Get-Disk | Select-Object Number, FriendlyName, SerialNumber, HealthStatus, OperationalStatus
Get-Volume | Select-Object DriveLetter, FileSystemLabel, FileSystemType, Size, SizeRemaining
Watch the right Windows events for storage reliability
Storage-related reliability issues often show up first in the System log. While event IDs vary by driver and stack, the operational practice is consistent: centralize System logs, alert on recurring disk/controller warnings and errors, and correlate timestamps with performance spikes.
Avoid relying only on “disk is slow” counters. A rising rate of resets/timeouts can manifest as latency spikes with no obvious capacity changes.
Monitoring in virtualized environments: where guest metrics stop being enough
Virtualization changes how you should interpret disk metrics because multiple guests share the same underlying storage, and the hypervisor can introduce its own queues and caching behaviors.
Inside a VM, you might see normal IOPS and modest latency while the datastore is overloaded and the hypervisor is delaying I/O. Conversely, you might see high latency inside the guest caused by an application’s sync writes even if the datastore is healthy.
The practical approach is to monitor at both layers. At the guest layer, collect volume free space, latency, and IOPS. At the hypervisor layer, collect datastore latency and queueing, plus any contention metrics your platform provides. The exact counters differ by vendor, but the concept is the same: without the upstream view, you can’t distinguish “this VM is noisy” from “the shared datastore is saturated.”
This becomes especially important with shared datastores hosting mixed workloads. A single VM running a log-heavy job can drive random writes that elevate latency for dozens of other VMs. Guest-only monitoring will show everyone slow, but not why.
Monitoring cloud block storage: provisioned limits and throttling signals
Cloud environments add another twist: performance is often governed by provisioned limits (IOPS, throughput) and burst models. Monitoring must include those limits and the indicators that you’re approaching or exceeding them.
For example, with managed block volumes you typically need to monitor:
- Volume read/write IOPS and throughput
- Average read/write latency
- Queue length (or equivalent)
- Burst balance/credit (where applicable)
- Throttling events or “stalled I/O” indicators
If your workloads are in AWS, Azure, or GCP, collect provider metrics (CloudWatch, Azure Monitor, Cloud Monitoring) in addition to OS counters. Provider metrics are the authoritative source for throttling and backend limits.
As an illustration, Azure CLI can list metrics for a resource (the exact metric names depend on resource type and API version, so validate against Azure Monitor for your disk type):
bash
# Example pattern: query Azure Monitor metrics (validate names for your resource)
az monitor metrics list \
--resource /subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.Compute/disks/<diskname> \
--interval PT1M \
--aggregation Average
The point isn’t the specific metric name in this snippet; it’s that cloud storage requires you to watch the platform’s view of limits and throttling, not just the guest’s iostat.
Thresholds that work in practice: combining absolute limits with deviation
Once you have a baseline, you can set thresholds that are both sensitive and stable. The trick is to avoid single-metric alerting. Storage problems are multi-symptom: latency rises, queue depth rises, and throughput/IOPS patterns shift.
Capacity thresholds: percent, free space, and time-to-full
For capacity, combine percent used with absolute free space and time-to-full. Percent used catches “near full,” absolute catches “small disk nearly full,” and time-to-full catches “it will be full soon.”
A workable pattern is:
- Warning: time-to-90% under 30 days, or used% over 80% with consistent growth.
- Critical: used% over 90% or free space under a fixed number appropriate for the volume.
Tune by volume role. Database volumes often need more headroom for maintenance tasks; log volumes need headroom for bursts.
Performance thresholds: require persistence and correlation
For performance, prefer alerts that require sustained conditions (for example, 10 minutes) and that correlate latency with queueing. This avoids paging on short spikes and catches real saturation.
A common alert condition is “write latency above X ms for Y minutes AND queue depth above Z.” Another is “latency above baseline by 3× for 15 minutes.”
If you can only choose one performance metric to alert on, choose latency. It is the closest metric to application experience.
Health thresholds: low tolerance for repeated errors
For health, repeated errors matter more than single events. Alert on increasing counts of I/O errors, resets, or SMART failures. If you see repeated controller timeouts, you want a ticket even if latency seems fine right now, because the next failover might not be clean.
Real-world scenario 1: The “disk full” outage that wasn’t about disk size
A common incident pattern involves a Linux application server where root has plenty of space, yet the service fails and logs say “No space left on device.” The on-call engineer checks df -h and sees 40% used on /. Confusion follows.
The missing metric is inode usage or a separate mount. In one real-world pattern, /var/lib/docker is on the root filesystem and the host runs frequent image builds. Layers accumulate into hundreds of thousands of small files. The filesystem still has free bytes, but inode utilization reaches 100%. Package installs start failing, containers can’t write, and the node is effectively down.
If you had been monitoring df -ih on the relevant mounts, you would have seen inode usage rising steadily days before impact. You also would have been able to correlate it with increased filesystem object counts, and you could have implemented retention policies (pruning unused images) before the outage.
The operational lesson is that “monitor disk usage” must include the resources that can be exhausted first, not only bytes. Bytes are only one dimension of capacity.
Real-world scenario 2: A database slowdown caused by a backup job’s write pattern
Consider a Windows SQL Server host where users report intermittent slow queries every night around 01:00. CPU is fine, memory is stable, and network latency is normal. Disk usage (free space) is healthy.
When you look at disk performance counters, you see that Avg. Disk sec/Write on the data volume jumps from a baseline of 2–4 ms to 40–80 ms for about 30 minutes. At the same time, Disk Write Bytes/sec spikes, and Current Disk Queue Length climbs. The pattern repeats nightly.
The culprit is often a scheduled job—database backup, log shipping, antivirus scan, or ETL export—that changes the I/O mix. If the backup writes large sequential streams to the same underlying storage pool used by the database, it can saturate throughput and increase write latency for the database’s small random writes. Users feel it as slow transactions.
If you only monitor capacity, you won’t catch this. If you monitor latency, throughput, and queue depth per volume and correlate it with job schedules, you can either move the job to a different time, write to a different volume/tier, or tune the job (compression, throttling, or using backup targets that don’t contend).
The operational lesson is that performance monitoring must be time-aware and per-volume, and you should treat recurring patterns as candidates for scheduling and placement fixes.
Real-world scenario 3: VM fleet degradation from datastore contention
In a virtualized cluster, you might get reports that “everything is slow” across multiple application VMs. Each VM shows moderate CPU usage, and inside the guest the disk counters don’t look dramatic. You might see slightly elevated latency but nothing that screams saturation.
At the hypervisor layer, however, datastore latency is high and increasing, and a small number of VMs are generating a burst of random writes due to a reindexing job or a logging storm. The shared nature of the datastore means the tail latency affects everyone. Even VMs that are mostly idle experience delayed I/O because the storage backend is servicing a backlog.
In this scenario, the right metrics are not just guest OS counters. You need datastore-level latency, queueing, and per-VM I/O contribution. Once you identify the top talkers, you can apply controls: move noisy workloads, apply storage policies/QoS if available, or isolate workloads onto separate datastores.
The operational lesson is that virtualization requires upstream visibility; otherwise you end up chasing “slow disks” inside every VM when the real bottleneck is shared.
Building an actionable dashboard: what to group together
Dashboards are only useful if they answer operational questions quickly. For disk monitoring, the most useful layout is per-host and per-volume, with capacity and performance on the same page so you can correlate cause and effect.
A practical dashboard grouping is:
- Capacity panel per mount/drive: used%, free bytes, growth rate, time-to-full.
- Performance panel per device/volume: read/write latency (avg and p95 if available), IOPS, throughput, queue depth.
- Health panel: error counts/timeouts, SMART/drive status where applicable, filesystem read-only events, multipath state.
Keep the panels aligned by time so you can visually correlate events. If latency spikes at the same time free space drops sharply, you might be seeing log storms, runaway temp files, or snapshots. If latency spikes with no capacity change but with error logs, suspect a failing path or device.
Avoid dashboards that only show %util and “disk busy.” Those often lead to false conclusions.
Correlating disk metrics with application symptoms
Disk metrics become far more actionable when you correlate them with application-level signals. Storage issues often manifest as:
- Increased request latency in web apps
- Database commit delays and lock timeouts
- Increased retry rates or circuit breaker trips in microservices
- Backup jobs exceeding their windows
- Message queues building up
When you see an application SLO degradation, you should be able to quickly answer whether storage latency increased at the same time, and on which volume. That requires consistent labels and mapping: the monitoring system should know that “SQL Data (D:)” corresponds to the volume hosting database files, not just “Disk 1.”
If you operate in Kubernetes, also consider the mapping between pods and persistent volumes. A storage backend saturation can affect multiple namespaces, and pod-level CPU/memory metrics won’t explain it.
Choosing sampling intervals and retention for disk metrics
Disk performance problems can be spiky. A 5-minute average can hide 30-second latency bursts that cause timeouts. At the same time, collecting every second for everything can be expensive.
A balanced approach is:
- Capacity: 5 minutes is usually sufficient, with daily rollups for long-term growth analysis.
- Performance: 10–30 seconds for latency/IOPS/throughput/queue depth on critical volumes (databases, VM datastores), and 30–60 seconds on less critical systems.
- Health/log-derived signals: near real-time ingestion with alerting based on counts over windows (e.g., 5 minutes, 1 hour).
Retention depends on your planning horizon. If you want to forecast growth and detect slow performance regressions, keep at least 30–90 days of high-level rollups and a few weeks of high-resolution performance metrics.
Integrating with Prometheus (Linux) using node_exporter
Many teams standardize on Prometheus for infrastructure metrics. For disk usage, node_exporter exposes filesystem metrics and disk I/O statistics derived from /proc.
The key is to scope metrics correctly. Filesystem metrics are labeled by mountpoint and filesystem type, which is ideal. Disk I/O metrics are labeled by device, which requires you to know which device corresponds to which mount.
A common approach is to build recording rules for:
- Filesystem used% per mount (excluding ephemeral mounts like
tmpfsif appropriate) - Disk read/write latency approximations (Prometheus derives time spent and operations)
- Disk throughput and IOPS per device
Because Prometheus calculations can be subtle, validate the derived metrics against iostat during initial rollout. The goal is consistency over time rather than perfect equivalence with every OS tool.
Integrating with the Elastic/Opensearch stack or Splunk for log-based storage signals
Not all storage signals are numeric counters. Error logs, resets, and filesystem warnings often provide the earliest evidence of instability. Centralized logging lets you alert on patterns like:
- Repeated “I/O error” messages for a device
- Multipath path down/up events
- NTFS warnings and storage driver resets
- NFS “stale file handle” or timeout messages
The value comes from correlation: an error burst that aligns with latency spikes tells a stronger story than either alone. If you already run a logging platform, treat storage-related log alerts as first-class, not as an afterthought.
Capacity planning: turning monitoring history into decisions
Once you’ve collected capacity and performance metrics for a few weeks, you can use them for capacity planning rather than just alerting.
For capacity, use growth trends per volume to forecast when you need to expand. Also look for uneven growth: if one mount grows rapidly because it holds logs or user uploads, plan to separate it or implement lifecycle management.
For performance, use p95/p99 latency trends and peak IOPS/throughput to decide when to move to a faster tier, separate workloads, or increase provisioned performance (in cloud). A single “max IOPS observed” number is less useful than understanding how often and how long you operate near a limit.
Capacity planning is also where you account for maintenance overhead. Databases need free space for index operations; filesystems need headroom to avoid fragmentation and metadata pressure; snapshot-based backup systems need reserve capacity for deltas.
Common metric interpretation pitfalls (and how to avoid them)
Disk metrics are notorious for being misread. Avoiding a few common traps improves both alert quality and incident response.
First, don’t interpret high throughput as “bad” by itself. Throughput might be expected during backups or ETL. It becomes a problem when it coincides with rising latency and user impact.
Second, don’t treat %util as definitive saturation in layered storage stacks. Use latency and queue depth as primary indicators.
Third, don’t mix device-level and filesystem-level metrics without mapping. Alerting “/dev/sdb is slow” doesn’t help if your operators think in mountpoints. Conversely, alerting “/var is slow” doesn’t make sense because filesystems aren’t “slow” in isolation—the underlying device is.
Fourth, beware caching effects. Some reads may be served from page cache, making read latency look great until cache misses occur. That’s not a reason to ignore metrics; it’s a reason to measure percentiles and correlate with memory pressure.
Finally, avoid global thresholds across heterogeneous storage. A single latency threshold for NVMe, network storage, and cloud volumes will either miss problems or generate noise. Use baselines and per-tier thresholds.
Putting it all together: a minimal but effective metric set
If you need to start simple, you can still build a strong monitoring foundation by focusing on a small number of high-signal metrics at the right layers.
For each host and each important filesystem/volume, collect: used%, free bytes, growth rate/time-to-full, and (Linux) inode usage where relevant.
For each underlying device (or volume, if your platform reports at that level), collect: read latency, write latency, read/write IOPS, read/write throughput, and queue depth.
For health, collect: storage-related error logs, disk/controller status where available, and SMART/NVMe health for directly attached drives.
Then, for virtualization and cloud, add the upstream metrics: datastore/pool usage and latency, and provider-side throttling/limits. Without these, you will spend too much time arguing about where the bottleneck lives.
By designing monitoring around capacity, performance, and health—with clear mapping between layers—you move from reactive checks to a system that tells you what’s changing, why it matters, and how close you are to an incident.