TL;DR — A VM’s qcow2 disk got its metadata marked corrupt after the thin-provisioned datastore beneath it ran out of real space mid-write. QEMU refuses to write to a corrupt-flagged image, so the VM was down. Recovery: take the VM offline, back up the raw image file first, run qemu-img check, repair with qemu-img check -r all, verify, boot, then fsck inside the guest. Root cause and prevention below.

The Symptom

A production VM went unresponsive. The guest console showed I/O errors, and the VM eventually paused. On the hypervisor, the QEMU log for the VM (/var/log/libvirt/qemu/<vm>.log, or journalctl on Proxmox) had these lines:

qcow2_free_clusters failed: Invalid argument
qcow2: Marking image as corrupt: Cluster allocation offset 0x1b2f10000 unaligned (L2 offset: 0x2a0000, L2 index: 0x4a); further corruption events will be suppressed

Once QEMU marks a qcow2 image corrupt, it sets the corrupt bit in the image header and refuses all further writes to protect what’s left — the VM either pauses or every write inside the guest fails. Restarting the VM does not help; you get:

qcow2: Image is corrupt; cannot be opened read/write

That error string is the whole reason you found this post. Here’s the way out.

Environment

  • Hypervisor: Proxmox VE 8.x (QEMU/KVM), single node of a small cluster
  • VM disk: qcow2 on a file-based datastore, thin-provisioned
  • Guest: Linux application VM
  • Trigger event: datastore hit 100% while the VM was writing

Everything below applies to any qcow2-on-KVM setup — plain libvirt, Proxmox, or OpenStack — not just this topology.

Root Cause Analysis

A qcow2 file is not a flat disk: it’s a small filesystem of its own, with an L1/L2 table tree mapping guest offsets to file offsets, and refcount tables tracking which clusters are in use. Every guest write that allocates a new cluster updates metadata in more than one place.

That multi-step update is the weakness. If the host loses power — or the underlying storage returns an error mid-allocation, which is exactly what happens when a thin-provisioned datastore runs out of real space — the L2 table can end up pointing at a cluster offset that was never properly allocated. On the next access, QEMU spots the inconsistency (an unaligned or out-of-range cluster offset), logs qcow2_free_clusters failed / Marking image as corrupt, sets the corrupt flag, and fences off writes.

In our case, the backing datastore filled to 100% while the guest was under write load; QEMU got ENOSPC mid-cluster-allocation and the L2 metadata was left pointing at a half-allocated cluster.

Two aggravating factors worth knowing:

  • lazy_refcounts=on (a qcow2 performance option) defers refcount updates, widening the crash window. Fine with cache=none + clean shutdowns; risky on flaky storage.
  • Host page-cache write-back (cache=writeback without a battery-backed path) means QEMU thinks metadata hit disk when it didn’t.

The Fix

Step 0 — stop, and copy the image before touching it. Every repair below rewrites metadata in place. If the repair goes sideways you want the original bytes.

# Proxmox: stop the VM (or confirm it's dead)
qm stop <vmid>

# Copy the image AS-IS to somewhere with space
cp --sparse=always /var/lib/vz/images/<vmid>/vm-<vmid>-disk-0.qcow2 /backup/

Step 1 — assess the damage (read-only):

qemu-img check /var/lib/vz/images/<vmid>/vm-<vmid>-disk-0.qcow2

Typical output for this incident class:

ERROR cluster 55321 refcount=0 reference=1
Leaked cluster 60911 refcount=1 reference=0
...
2 errors were found on the image.
Data may be corrupted, or further writes to the image may corrupt it.
1 leaked clusters were found on the image.

Read this carefully before repairing:

  • Leaked clusters are harmless to data — space marked used that nothing references. -r leaks fixes only these; zero risk.
  • ERROR (refcount mismatches / invalid offsets) are real metadata damage. -r all will fix the metadata to be consistent, which is not the same as recovering every byte — a cluster the L2 table lost is gone.

Step 2 — repair:

qemu-img check -r all /var/lib/vz/images/<vmid>/vm-<vmid>-disk-0.qcow2

This also clears the corrupt bit if the image comes out consistent. Confirm:

qemu-img check /var/lib/vz/images/<vmid>/vm-<vmid>-disk-0.qcow2
# expect: No errors were found on the image.
qemu-img info /var/lib/vz/images/<vmid>/vm-<vmid>-disk-0.qcow2 | grep corrupt
# expect: corrupt: false (or no corrupt line at all)

Step 3 — boot and check the guest filesystem. qcow2 consistency ≠ guest-filesystem consistency. The guest saw I/O errors, so:

qm start <vmid>
# inside the guest:
sudo fsck -f /dev/vda1        # or xfs_repair for XFS — from rescue mode if it's the root fs
# for a database VM, run the engine's own checks too
# (mysqlcheck / innodb_force_recovery ladder — that's its own post)

In our incident, -r all produced a clean image, fsck fixed a handful of orphaned inodes, and the VM came back with no application-level data loss — the failed writes had never been acknowledged to the guest.

If -r all cannot produce a clean image, stop iterating on the original. Your options, in order: restore from backup; or salvage what’s readable with qemu-img convert -O qcow2 corrupt.qcow2 salvaged.qcow2 (convert reads what’s reachable and writes a fresh, consistent image), then fsck the result hard.

Prevention

  1. Never let a thin-provisioned datastore hit 100%. This is the #1 real-world cause. Alert at 85% on the storage, not just inside guests — thin provisioning means guests can’t see the real free space.
  2. qemu-img check in maintenance windows. A leaked-cluster count that grows across reboots is an early warning. (Image must be offline or use --force-share read-only, results are advisory while running.)
  3. Audit cache modes. cache=none (O_DIRECT) for anything you care about, unless you know exactly why not.
  4. Reconsider lazy_refcounts on storage that can drop writes.
  5. Backups you have restore-tested. The repair above worked; the plan can’t be that it always will.

What I’d do differently

The uncomfortable part: the datastore usage was visible the whole time — nobody was alerting on it at the storage layer, only inside guests, and thin provisioning made every guest think it had plenty of room. The repair worked, but it didn’t have to. Capacity alerts on the datastore itself and a restore-tested backup are what turn this from a war story into a non-event.