[calvix]

Description

This fixes a bug in apache#13556 (encrypted RBD volumes).

It creates an encrypted root disk as a thin clone of the template: whatever the guest has not written yet is read straight from the template. What it missed is that this only works if the template has no holes (empty, never-written ranges), and templates usually have them (the Ubuntu 24.04 template tested below has 2.2 of 4 GiB allocated).

Ceph stores a disk in 4 MiB objects. The first guest write into an object, even 4 KiB, copies that whole object from the template into the clone and encrypts it - but the template's holes inside it are not copied. From then on those ranges are read through decryption, and decrypting empty space gives random bytes, not zeros.

The result: VMs with encrypted roots boot and run fine, and get corrupted later - places the filesystem expects to read as zeros return garbage. A mostly empty separate /boot usually breaks first.

The fix

apache#13556 already prepares each template once before the first encrypted clone: it grows the template a bit (room for the LUKS2 header) and takes a snapshot to clone from. This PR fills the template's holes right before that snapshot is taken, by rewriting the template from its own cloudstack-base-snap with qemu-img convert -n -S 0 (-S 0 writes zeros out instead of skipping them). The snapshot then has no holes, so encrypted clones read correctly everywhere. They stay thin clones.

• The snapshot gets a new name, cloudstack-base-snap-luks2 instead of -luks, so templates already prepared by storage: KVM - enable RBD/Ceph volume encryption support apache/cloudstack#13556 (with holes) get a new snapshot without holes. Roots already cloned from the old -luks snapshot keep using it.
• Unencrypted clones are not affected: they are cloned from cloudstack-base-snap, which is only read.
• If two hosts prepare the same template at the same time, the second one uses the snapshot the first one made instead of failing the deploy.
• QemuImg gets setWriteZeroRanges() to pass -S 0.

Cost: a template used for encrypted roots becomes fully allocated in its pool (once per template per pool).

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• Build/CI
• Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

How Has This Been Tested?

Unit tests on this branch (main): RbdEncryptionTest (10), LibvirtStorageAdaptorTest (23) and KVMStorageProcessorTest (51) pass. RbdEncryptionTest checks that the copy asks for -S 0, reads from the template's snapshot and writes into the template image itself, unencrypted. QemuImgTest covers what -S 0 does to a local file, but only runs where qemu-img and libvirt are available.

On a 3-node KVM cluster with RBD primary storage (the same change on 4.23.0.0), with an Ubuntu 24.04 template that already had the old -luks snapshot with holes (2.2 of 4 GiB allocated):

• Deploying an encrypted root created cloudstack-base-snap-luks2 (~31 s for the 4 GiB template), fully allocated (4.0 of 4.0 GiB). The old -luks snapshot was left alone.
• The root is a clone of @cloudstack-base-snap-luks2 and the VM booted from it.
• A plaintext VM from the same template is still a clone of @cloudstack-base-snap and runs.

To check the actual bug I made two clones of that template, one from the old -luks snapshot and one from the new -luks2, both with a LUKS2 header. In 10 objects that hold data but also have holes, I read a hole sector, wrote 4 KiB of zeros elsewhere in the same object (so Ceph copies the object into the clone), and read the hole sector again:

                    hole reads zeros   hole reads zeros
                    before the write   after the write
                    ----------------   ----------------
old -luks (holes)         10/10              0/10
new -luks2 (filled)       10/10             10/10

Not tested: two hosts preparing the same template at the same time, and templates bigger than 4 GiB.