Summary
Make RTMR[1] independent of the QEMU version by normalizing the Linux setup
header, so dstack-mr can go back to a single kernel digest.
Follow-up to #1183, which fixes the immediate breakage by recording two
kernel Authenticode digests and picking one from the host-declared
vm_config.qemu_version. That works, but it leaves the verifier carrying a
QEMU-version branch forever. This issue removes the branch instead.
Background
QEMU rewrites part of the Linux setup header before serving the kernel over
fw_cfg, and OVMF measures the result into RTMR[1]. QEMU commit a7542a38f399
("x86/loader: Don't update kernel header for CoCo VMs", first in 10.2.0) stopped
doing that for confidential guests, so the same image measures differently on
QEMU <= 10.1 vs >= 10.2. See #1183 for the full analysis.
Measured empirically on a real dstack image, QEMU's rewrite touches 4 regions,
11 bytes:
0x0210..0x0212 orig=[00,01] patched=[b0,81] type_of_loader, loadflags
0x0219..0x021f orig=[00 x6] patched=[10,54,7f,c0,62,a9] ramdisk_image, ramdisk_size
0x0224..0x0226 orig=[a0,50] patched=[00,fe] heap_end_ptr
0x022a..0x022b orig=[00] patched=[02] cmd_line_ptr
Proposal
Normalize the setup header on both sides so the measured bytes are
version-independent and still equal the shipped file byte for byte:
- Image build — normalize the shipped
bzImage in place.
- OVMF — apply the same normalization to the
kernel blob before it is
measured and loaded.
- dstack-mr — drop
patched_kernel_authenticode, patch_kernel(), the
version branch, the qemu_patches_kernel_header escape hatch, and the
memory_size == 2 GiB || >= 0xB0000000 restriction (that guard only exists
because the patched digest depends on guest RAM).
Normalization is idempotent, so it produces the same bytes whether or not QEMU
rewrote anything. Because the shipped file is normalized too, the measured value
remains a plain Authenticode hash of the artifact in sha256sum.txt — no
dstack-specific derived digest.
Normalization rule
Defined against the boot protocol, not against QEMU's behavior, so it does
not go stale when QEMU changes:
Zero every field Documentation/arch/x86/boot.rst types as write
(bootloader writes it; the kernel supplies no value), and clear
CAN_USE_HEAP (0x80) in loadflags. Leave every modify field alone.
That covers type_of_loader (0x210), ramdisk_image/ramdisk_size
(0x218/0x21c), heap_end_ptr (0x224), ext_loader_ver/ext_loader_type
(0x226/0x227), cmd_line_ptr (0x228), hardware_subarch[_data] (0x23c/0x240)
and setup_data (0x250).
Do not widen this to "all bootloader-writable fields". modify fields carry
real kernel-supplied values and zeroing them breaks the kernel:
| field |
type |
offset |
build-time value |
code32_start |
modify (optional, reloc) |
0x214 |
0x100000 (protected-mode entry) |
setup_move_size |
modify (obligatory) |
0x212 |
0x8000 |
vid_mode |
modify (obligatory) |
0x1fa |
0xffff |
loadflags |
modify (obligatory) |
0x211 |
0x01 |
root_flags |
modify (optional) |
0x1f2 |
0x01 |
vid_mode is the one modify field QEMU can write, but only when the cmdline
contains vga= — and the cmdline is measured into RTMR[2], so it cannot be
smuggled in. Leaving it alone is safe.
Impact on the shipped kernel: 2 bytes
Every write field is already zero in a freshly built bzImage except
heap_end_ptr (0x224 = 0x50a0). So normalization changes exactly two bytes.
This is safe for non-TDX boot:
boot.rst types it write (obligatory): the bootloader writes it, the kernel
supplies no value.
init_heap() reads it only when the bootloader has set CAN_USE_HEAP, and
that flag is itself bootloader-set (zero at build time). Any bootloader that
sets the flag is obliged by the spec to set heap_end_ptr too.
- On the EFI-stub path the real-mode setup code never runs at all.
- The PE headers occupy
0x40..0x170; every setup-header field is outside them,
so the PE structure and EFI-stub entry are untouched.
Secure Boot is not planned, so there is no signature to invalidate and no
sign-vs-normalize ordering constraint.
Implementation
1. Image build (shared by Yocto and mkosi)
Both backends already funnel through os/image/assemble.sh
(os/yocto/mkimage.sh execs it; mkosi calls it from mkosi.postoutput), so one
insertion point covers both. Add a helper next to the existing
authenticode_hash.py / kernel-cmdline.sh, e.g.
os/image/normalize-kernel-header.py, and call it right after the kernel is
staged:
verbose cp "$KERNEL_IMAGE" "${OUTPUT_DIR}/bzImage"
+ verbose "$(dirname "${BASH_SOURCE[0]}")/normalize-kernel-header.py" "${OUTPUT_DIR}/bzImage"
assemble.sh:413 runs before tdx-measurement-cbor (:469) and before
CHECKSUM_FILES (:590), so the measurement document and sha256sum.txt
automatically cover the normalized kernel.
The script should be idempotent and should fail loudly if the setup header does
not look like a supported boot protocol version.
Open question: the optional UKI image embeds the kernel in a .linux PE
section. Check whether it needs the same treatment or is unaffected because it
is not loaded via -kernel.
2. OVMF
OvmfPkg/QemuKernelLoaderFsDxe builds the virtual kernel file from
QemuFwCfgItemKernelSetupData ++ QemuFwCfgItemKernelData (see
mKernelBlobItems). That blob is what gBS->LoadImage measures and runs on
the EFI-stub path, so normalizing it there keeps measured bytes == loaded bytes
== file bytes — no "attested something other than what ran" gap.
Note X86QemuLoadImageLib keeps a separate SetupBuf for the legacy
LoadLinux* path; dstack does not use it and it should not be touched.
Worth attempting upstream: this is the missing half of Gerd Hoffmann's 2024
series (f2594d92844, "x86/loader: expose unpatched kernel", whose commit
message ends "Needs OVMF changes too to be actually useful") and it fixes the
same class of Secure Boot breakage that motivated that series. Note that the
etc/boot/kernel fw_cfg file it added only exists in QEMU >= 10.1, which is why
normalization is preferable for us: it works on 8.2.2 too.
3. dstack-mr / dstack-types
Only after images with the new OVMF are the floor. Until then #1183's
patch_kernel_header switch is still needed to verify CVMs running older
firmware.
Documentation
os/image/README.md — what the script does and why, so a bit-for-bit
reproducibility check does not trip over bzImage differing from the raw
kernel build output.
docs/security/security-model.md — replace the "A host-declared QEMU version
selects between digests" section: once this lands the host no longer has that
choice at all, which is a strictly stronger property.
os/yocto/repro-build/check.sh — make sure the normalization step is part of
the reproducibility story.
Ordering
Land after #1183. That PR is the fix for the currently broken production path
and it provides the patch_kernel_header switch this transition depends on.
Summary
Make RTMR[1] independent of the QEMU version by normalizing the Linux setup
header, so
dstack-mrcan go back to a single kernel digest.Follow-up to #1183, which fixes the immediate breakage by recording two
kernel Authenticode digests and picking one from the host-declared
vm_config.qemu_version. That works, but it leaves the verifier carrying aQEMU-version branch forever. This issue removes the branch instead.
Background
QEMU rewrites part of the Linux setup header before serving the kernel over
fw_cfg, and OVMF measures the result into RTMR[1]. QEMU commit
a7542a38f399("x86/loader: Don't update kernel header for CoCo VMs", first in 10.2.0) stopped
doing that for confidential guests, so the same image measures differently on
QEMU <= 10.1 vs >= 10.2. See #1183 for the full analysis.
Measured empirically on a real dstack image, QEMU's rewrite touches 4 regions,
11 bytes:
Proposal
Normalize the setup header on both sides so the measured bytes are
version-independent and still equal the shipped file byte for byte:
bzImagein place.kernelblob before it ismeasured and loaded.
patched_kernel_authenticode,patch_kernel(), theversion branch, the
qemu_patches_kernel_headerescape hatch, and thememory_size == 2 GiB || >= 0xB0000000restriction (that guard only existsbecause the patched digest depends on guest RAM).
Normalization is idempotent, so it produces the same bytes whether or not QEMU
rewrote anything. Because the shipped file is normalized too, the measured value
remains a plain Authenticode hash of the artifact in
sha256sum.txt— nodstack-specific derived digest.
Normalization rule
Defined against the boot protocol, not against QEMU's behavior, so it does
not go stale when QEMU changes:
That covers
type_of_loader(0x210),ramdisk_image/ramdisk_size(0x218/0x21c),
heap_end_ptr(0x224),ext_loader_ver/ext_loader_type(0x226/0x227),
cmd_line_ptr(0x228),hardware_subarch[_data](0x23c/0x240)and
setup_data(0x250).Do not widen this to "all bootloader-writable fields".
modifyfields carryreal kernel-supplied values and zeroing them breaks the kernel:
code32_startsetup_move_sizevid_modeloadflagsroot_flagsvid_modeis the onemodifyfield QEMU can write, but only when the cmdlinecontains
vga=— and the cmdline is measured into RTMR[2], so it cannot besmuggled in. Leaving it alone is safe.
Impact on the shipped kernel: 2 bytes
Every
writefield is already zero in a freshly built bzImage exceptheap_end_ptr(0x224 =0x50a0). So normalization changes exactly two bytes.This is safe for non-TDX boot:
boot.rsttypes itwrite (obligatory): the bootloader writes it, the kernelsupplies no value.
init_heap()reads it only when the bootloader has setCAN_USE_HEAP, andthat flag is itself bootloader-set (zero at build time). Any bootloader that
sets the flag is obliged by the spec to set
heap_end_ptrtoo.0x40..0x170; every setup-header field is outside them,so the PE structure and EFI-stub entry are untouched.
Secure Boot is not planned, so there is no signature to invalidate and no
sign-vs-normalize ordering constraint.
Implementation
1. Image build (shared by Yocto and mkosi)
Both backends already funnel through
os/image/assemble.sh(
os/yocto/mkimage.shexecs it; mkosi calls it frommkosi.postoutput), so oneinsertion point covers both. Add a helper next to the existing
authenticode_hash.py/kernel-cmdline.sh, e.g.os/image/normalize-kernel-header.py, and call it right after the kernel isstaged:
verbose cp "$KERNEL_IMAGE" "${OUTPUT_DIR}/bzImage" + verbose "$(dirname "${BASH_SOURCE[0]}")/normalize-kernel-header.py" "${OUTPUT_DIR}/bzImage"assemble.sh:413runs beforetdx-measurement-cbor(:469) and beforeCHECKSUM_FILES(:590), so the measurement document andsha256sum.txtautomatically cover the normalized kernel.
The script should be idempotent and should fail loudly if the setup header does
not look like a supported boot protocol version.
Open question: the optional UKI image embeds the kernel in a
.linuxPEsection. Check whether it needs the same treatment or is unaffected because it
is not loaded via
-kernel.2. OVMF
OvmfPkg/QemuKernelLoaderFsDxebuilds the virtualkernelfile fromQemuFwCfgItemKernelSetupData ++ QemuFwCfgItemKernelData(seemKernelBlobItems). That blob is whatgBS->LoadImagemeasures and runs onthe EFI-stub path, so normalizing it there keeps measured bytes == loaded bytes
== file bytes — no "attested something other than what ran" gap.
Note
X86QemuLoadImageLibkeeps a separateSetupBuffor the legacyLoadLinux*path; dstack does not use it and it should not be touched.Worth attempting upstream: this is the missing half of Gerd Hoffmann's 2024
series (
f2594d92844, "x86/loader: expose unpatched kernel", whose commitmessage ends "Needs OVMF changes too to be actually useful") and it fixes the
same class of Secure Boot breakage that motivated that series. Note that the
etc/boot/kernelfw_cfg file it added only exists in QEMU >= 10.1, which is whynormalization is preferable for us: it works on 8.2.2 too.
3. dstack-mr / dstack-types
Only after images with the new OVMF are the floor. Until then #1183's
patch_kernel_headerswitch is still needed to verify CVMs running olderfirmware.
Documentation
os/image/README.md— what the script does and why, so a bit-for-bitreproducibility check does not trip over
bzImagediffering from the rawkernel build output.
docs/security/security-model.md— replace the "A host-declared QEMU versionselects between digests" section: once this lands the host no longer has that
choice at all, which is a strictly stronger property.
os/yocto/repro-build/check.sh— make sure the normalization step is part ofthe reproducibility story.
Ordering
Land after #1183. That PR is the fix for the currently broken production path
and it provides the
patch_kernel_headerswitch this transition depends on.