"The VMs are slow" is not a fault you can act on. Before you go anywhere near the array, one PowerCLI line against vCenter will check latency for every VM at once — and the answer usually points at two or three machines rather than at the storage.
Everything below runs from a PowerCLI session connected to vCenter: Install-Module VMware.PowerCLI once, then Connect-VIServer vcenter.example.com.
One PowerCLI command to check latency for every VM
Get-VM | ? {$_.PowerState -eq "PoweredOn"} | Select Name, @{n="AVG Max Latency (ms)";e={(get-stat -Entity $_ -Stat Disk.MaxTotalLatency.Latest -Start (Get-Date).AddDays(-7) | Measure Value -Average ).Average }},@{n="Max Latency (ms)";e={(get-stat -Entity $_ -Stat Disk.MaxTotalLatency.Latest -Start (Get-Date).AddDays(-7) | Measure Value -Maximum ).Maximum }}
Name AVG Max Latency (ms) Max Latency (ms)
---- -------------------- ----------------
vm-app-01 106.491017964072 19020
vm-app-02 76.9730538922156 10673
vm-app-03 54.6467065868263 5611
vm-app-04 34.0149700598802 5310
vm-app-05 28.3682634730539 1877
vm-backup-01 26.6467065868263 593
vm-app-06 25.1916167664671 3162
vm-app-07 20.0718562874251 2407
vm-db-01 10.1137724550898 318
vm-vc-01 9.38023952095808 283
vCLS (1) 6.58682634730539 354
vm-sql-01 6.1377245508982 201
vm-ntp-01 2.76946107784431 175
vm-test-01 0.646706586826347 101
Sorted here for readability — the raw command returns inventory order. The version further down sorts for you.
What Disk.MaxTotalLatency.Latest actually measures
At each sample, the highest latency across the storage devices backing that VM's disks — VMkernel time plus device time, as the guest's SCSI commands see it. So the first column is the average of a series of worst-case readings, not the average latency the VM experienced. It usually reads higher than a true average, and that is the point: it surfaces a VM whose disks sit behind one badly behaving device.
It also means a noisy neighbour on the same LUN can lift a VM's number, so a whole column of high values points at the storage rather than at the VMs.
The second column is the worst sample vCenter kept. This counter rolls up as latest rather than maximum, so each stored value is a snapshot taken at the end of its interval, not the peak within it — the real weekly worst is at least this bad and may be worse. (Broadcom's disk counter reference has the definitions.)
Each average above is built from 334 samples: seven days of 30-minute rollups, minus the two at the edge of the window.
Reading the numbers
As rules of thumb for the average-of-peaks column: under 10 ms is healthy, 10–20 ms is normal under load, 20–50 ms deserves a look, and anything sustained above 50 ms is a VM someone is complaining about.
Those numbers assume a hybrid or spinning array. On all-flash, normal is 1–2 ms and a sustained 10 ms is already a fault — halve every figure above.
The peak column needs the opposite instinct: do not chase a single spike. A 19-second peak looks alarming and is usually a snapshot being consolidated, a backup proxy hot-adding disks, or a storage path failing over. Read it together with the average — a high average and a high peak is a sick VM, while a low average with one enormous peak is an event with a timestamp you can go and find.
If the whole cluster reads high rather than two VMs, the problem is underneath them; reclaiming space on the datastore and the array is one of the things worth checking there.
Scope it to one cluster, and sort it
The command above will check latency for every VM in the connected vCenter, and calls Get-Stat twice per VM, which on a few hundred machines is slow enough to make you think it has hung. This version asks once, sorts, and does not lie about missing data:
$start = (Get-Date).AddDays(-7)
Get-Cluster "Cluster-01" | Get-VM | Where-Object {$_.PowerState -eq "PoweredOn"} | ForEach-Object {
$s = Get-Stat -Entity $_ -Stat Disk.MaxTotalLatency.Latest -Start $start -ErrorAction SilentlyContinue
[pscustomobject]@{
Name = $_.Name
AvgMaxMs = if ($s) { [math]::Round(($s | Measure-Object Value -Average).Average, 2) } else { $null }
MaxMs = ($s | Measure-Object Value -Maximum).Maximum
Samples = ($s | Measure-Object).Count
}
} | Sort-Object AvgMaxMs -Descending
The if ($s) matters. [math]::Round() turns a missing average into 0, which sorts to the bottom of the list and reads as the healthiest VM on the cluster rather than the one with no data at all. The Samples column is there for the same reason — a confident-looking average built from nine samples is not an average.
The same Get-Cluster | Get-... shape works for host-level jobs, such as enabling SSH on every host in a cluster.
Append | Export-Csv latency.csv -NoTypeInformation to hand the result to someone, or | Select -First 10 when you only want the offenders.
When Get-Stat comes back empty
- The VM was powered off for part of the period, or created after it. Note that the power-state filter is about now, not about the window: a VM booted an hour ago gets an average from a handful of samples, and one that was terrible all week but is powered off today never appears at all.
- You swapped in a counter that is not level 1.
Disk.MaxTotalLatency.Latestis collected at every rollup interval out of the box. Per-VMDK counters are not — anything in theVirtualDiskgroup needs statistics level 3, and at the default level those exist only in the realtime interval, twenty-second samples kept for about an hour. Check vCenter → Configure → General → Statistics before blaming the script. - Your query sits on a retention boundary. vCenter keeps 30-minute rollups for about 7 days and 2-hour rollups for a month, so
AddDays(-7)straddles the edge: the sample count, and therefore the average, shifts between runs depending on which interval answers. It never errors —-Startsilently returns whatever exists.
Run the same query with -Start (Get-Date).AddDays(-1) to bisect it. If the day works and the week does not, it is the statistics configuration, not your script.
