You deleted a few hundred gigabytes inside a Linux virtual machine, MinIO finally expired the old object versions, and df -h in the guest is delighted. The VMware datastore is not: it shows exactly the same usage as before, and so does the array behind it. Nothing is broken — nobody told the lower layers. This is how to reclaim space all the way down, on RHEL 9, Rocky Linux 9, AlmaLinux 9 and any other modern Linux guest.

This article is about Linux virtual machines. On Windows the same work is normally automatic: the OS issues TRIM to the underlying storage on its own schedule.

The problem: three layers, each with its own idea of "free"

Deleting a file makes the filesystem — ext4, XFS, whatever — mark its blocks as free inside the guest. The data itself stays where it is, and nothing is sent downwards. So you end up with three views of the same disk:

Layer What it thinks Who tells it otherwise
Guest filesystem space is free the rm you already ran
Thin VMDK on the datastore still allocated guest-issued TRIM/UNMAP
Thin LUN on the array still allocated UNMAP from the ESXi host

If the LUN is thin-provisioned and the storage supports SCSI UNMAP, TRIM or space reclamation, the guest has to say so explicitly. That is the whole job.

Step 1: does the virtual disk even support TRIM?

Run lsblk -D in the guest:

[root@minio-prod-01 ~]# lsblk -D
NAME        DISC-ALN DISC-GRAN DISC-MAX DISC-ZERO
sda                0        1M      32M         0
├─sda1             0        1M      32M         0
├─sda2             0        1M      32M         0
├─sda3             0        1M      32M         0
  ├─rl-root        0        1M      32M         0
  ├─rl-swap        0        1M      32M         0
  └─rl-home        0        1M      32M         0
sdb                0        1M      32M         0
sdc                0        1M      32M         0
sdd                0        1M      32M         0
sde                0        1M      32M         0
sdf                0        1M      32M         0
sdg                0        1M      32M         0
sdh                0        1M      32M         0
sdi                0        1M      32M         0
sr0                0        0B       0B         0
[root@minio-prod-01 ~]#

Look at DISC-GRAN and DISC-MAX:

  • Both non-zero → TRIM/UNMAP is supported and the passthrough to the VMDK is in place. Here they are 1M and 32M, so this guest is good to go.
  • Both zero → the discard path does not exist and nothing below will help. The usual cause is a thick-provisioned VMDK: a thick disk has no blocks to give back, so ESXi advertises no discard support to the guest. Converting the disk to thin is the prerequisite, not a tuning step. If the disk is already thin and the columns are still zero, look next at the virtual SCSI controller and the VM hardware version — the older LSI Logic Parallel controller and low hardware versions do not carry UNMAP through to the VMDK. VMware Paravirtual is the controller you want.

This one command is a better test than any version checklist, because it reports what the guest was actually offered rather than what the documentation says it should have been.

Step 2: reclaim space from inside the guest

Two ways to do it.

Method 1 — real-time TRIM with the discard mount option

Add discard in /etc/fstab:

/dev/sdb  /mnt/disk1  xfs  defaults,discard  0  0

Every delete then issues TRIM immediately. It works, but it puts the discard in the path of every unlink, and on a busy volume that costs measurable latency. For large volumes of data the scheduled approach below is the better default — and it is what the distributions themselves have moved to.

Method 2 — scheduled TRIM with fstrim

Trim a mounted filesystem on demand:

[root@minio-prod-01 ~]# fstrim -v /mnt/disk1
/mnt/disk1: 76.1 GiB (81664561152 bytes) trimmed

That number is the one you came for: 76.1 GiB just went back to the datastore.

Do not read too much into it on a re-run, though. fstrim(8) is explicit that it "will report the same potential discard bytes each time, but only sectors which had been written to between the discards would actually be discarded by the storage device". The figure is the free space the filesystem offered for discard, recalculated from scratch on every run — not a tally of what was newly released. An unchanged number on a second run therefore tells you nothing either way; the datastore is where you confirm the result.

For several disks at once:

for i in {1..8}; do
  echo "Trimming /mnt/disk$i"
  fstrim -v /mnt/disk$i
done

The simplest schedule: fstrim.timer

Before writing any cron script, check whether the distribution already ships one. RHEL 9, Rocky Linux 9 and AlmaLinux 9 all include a systemd timer that runs fstrim --all weekly:

systemctl enable --now fstrim.timer
systemctl list-timers fstrim.timer

fstrim --all walks every mounted filesystem that supports discard, so it covers disks you add later without editing anything. For most systems this is the whole answer, and the sections below are for when you want a daily run and a log you can point at.

Automating with cron

A daily script that trims /mnt/disk1 to /mnt/disk8 and logs to /var/log/minio-trim/:

cat <<'EOF' > /etc/cron.daily/minio-trim
#!/bin/bash

LOGDIR="/var/log/minio-trim"
LOGFILE="$LOGDIR/$(date +%F).log"

mkdir -p "$LOGDIR"

echo "=== fstrim started at $(date) ===" >> "$LOGFILE"

for disk in /mnt/disk{1..8}; do
  if mountpoint -q "$disk"; then
    echo "Trimming $disk..." >> "$LOGFILE"
    fstrim -v "$disk" >> "$LOGFILE" 2>&1
  else
    echo "Skipping $disk (not mounted)" >> "$LOGFILE"
  fi
done

echo "=== fstrim finished at $(date) ===" >> "$LOGFILE"
echo "" >> "$LOGFILE"
EOF

chmod +x /etc/cron.daily/minio-trim

The mountpoint -q guard matters: running fstrim against a directory that is not a mount point trims the parent filesystem instead, which is not what the log will claim happened.

Optional: clean up old logs

And a second job to delete logs older than 60 days:

cat <<'EOF' > /etc/cron.daily/minio-trim-logrotate
#!/bin/bash
find /var/log/minio-trim/ -type f -name "*.log" -mtime +60 -delete
EOF

chmod +x /etc/cron.daily/minio-trim-logrotate

Step 3: check the datastore, not just the guest

This is the step the guest-side guides stop before, and it is where the space actually shows up. fstrim hands the blocks back to the VMDK; whether they reach the array depends on the datastore.

On VMFS6 (ESXi 6.5 or newer — VMFS6 does not exist before that), ESXi reclaims automatically in the background. Check the current setting per datastore from the host with esxcli storage vmfs reclaim config get:

esxcli storage vmfs reclaim config get -l DATASTORE_NAME

On VMFS5 there is no automatic reclamation — esxcli storage vmfs unmap has to be run by hand:

esxcli storage vmfs unmap -l DATASTORE_NAME

Then confirm from the top down, in this order — each layer can be the one that has not caught up:

# 1. the guest
df -h /mnt/disk1

# 2. the VMDK, from the host - the first column is the blocks actually used
ls -lsh /vmfs/volumes/DATASTORE_NAME/VM_NAME/VM_NAME-flat.vmdk

# 3. the datastore
esxcli storage filesystem list

On a thin disk it is that blocks-used figure that shrinks, not the logical size — plain ls -lh reports the provisioned size and looks identical before and after, which is an easy way to convince yourself nothing happened when it did.

Give it time. Automatic reclamation on VMFS6 is deliberately unhurried so it does not compete with production I/O, and thin-LUN accounting on the array often updates on its own schedule on top of that. Checking five minutes later and concluding it did not work is the most common mistake here.

Bonus: package it as an RPM

If you run this on more than a couple of servers, repeating the setup by hand gets tedious and drifts. Build it once as an RPM and install it everywhere.

1. Install the build tools

dnf install rpm-build rpmdevtools -y

2. Set up the build tree

rpmdev-setuptree

This creates the standard directory structure under ~/rpmbuild.

3. Create the .spec file

cat > ~/rpmbuild/SPECS/minio-trim.spec <<'EOF'
Name:           minio-trim
Version:        1.0
Release:        1%{?dist}
Summary:        Daily fstrim for MinIO volumes with logging and log rotation

License:        MIT
BuildArch:      noarch

%description
Runs daily fstrim on /mnt/disk1 to /mnt/disk8, logs output to /var/log/minio-trim/,
and removes logs older than 60 days.

%prep

%build

%install
mkdir -p %{buildroot}/etc/cron.daily
mkdir -p %{buildroot}/var/log/minio-trim

cat > %{buildroot}/etc/cron.daily/minio-trim << 'EOS'
#!/bin/bash

LOGDIR="/var/log/minio-trim"
LOGFILE="$LOGDIR/$(date +%F).log"

mkdir -p "$LOGDIR"

echo "=== fstrim started at $(date) ===" >> "$LOGFILE"

for disk in /mnt/disk{1..8}; do
  if mountpoint -q "$disk"; then
    echo "Trimming $disk..." >> "$LOGFILE"
    fstrim -v "$disk" >> "$LOGFILE" 2>&1
  else
    echo "Skipping $disk (not mounted)" >> "$LOGFILE"
  fi
done

echo "=== fstrim finished at $(date) ===" >> "$LOGFILE"
echo "" >> "$LOGFILE"
EOS

chmod 0755 %{buildroot}/etc/cron.daily/minio-trim

cat > %{buildroot}/etc/cron.daily/minio-trim-logrotate << 'EOS'
#!/bin/bash
find /var/log/minio-trim/ -type f -name "*.log" -mtime +60 -delete
EOS

chmod 0755 %{buildroot}/etc/cron.daily/minio-trim-logrotate

%files
%attr(0755,root,root) /etc/cron.daily/minio-trim
%attr(0755,root,root) /etc/cron.daily/minio-trim-logrotate
%dir %attr(0755,root,root) /var/log/minio-trim

%changelog
* Fri Aug 01 2025 You <[email protected]> - 1.0-1
- Initial version with daily trim and log cleanup
EOF

Keep the retention in the %description and in the find command the same. They drift apart the first time someone edits one of them, and then the package documents a policy it does not implement.

4. Build it

rpmbuild -ba ~/rpmbuild/SPECS/minio-trim.spec

The result lands at ~/rpmbuild/RPMS/noarch/minio-trim-1.0-1.noarch.rpm.

5. Install it on the target servers

dnf install minio-trim-1.0-1.noarch.rpm

That is the whole loop automated: the guest gives the blocks back daily, the datastore picks them up, and you can reclaim space across a fleet with one package instead of eight mount points times however many servers.

If the answer turns out to be more space rather than less, the same stack is walked in the other direction — see how to extend an LVM partition online.

One Comment

Leave a Reply