Skip to content

Normalize the Linux setup header so RTMR[1] stops depending on the QEMU version #1185

Description

@kvinwang

Summary

Make RTMR[1] independent of the QEMU version by normalizing the Linux setup
header, so dstack-mr can go back to a single kernel digest.

Follow-up to #1183, which fixes the immediate breakage by recording two
kernel Authenticode digests and picking one from the host-declared
vm_config.qemu_version. That works, but it leaves the verifier carrying a
QEMU-version branch forever. This issue removes the branch instead.

Background

QEMU rewrites part of the Linux setup header before serving the kernel over
fw_cfg, and OVMF measures the result into RTMR[1]. QEMU commit a7542a38f399
("x86/loader: Don't update kernel header for CoCo VMs", first in 10.2.0) stopped
doing that for confidential guests, so the same image measures differently on
QEMU <= 10.1 vs >= 10.2. See #1183 for the full analysis.

Measured empirically on a real dstack image, QEMU's rewrite touches 4 regions,
11 bytes
:

0x0210..0x0212  orig=[00,01]        patched=[b0,81]                type_of_loader, loadflags
0x0219..0x021f  orig=[00 x6]        patched=[10,54,7f,c0,62,a9]    ramdisk_image, ramdisk_size
0x0224..0x0226  orig=[a0,50]        patched=[00,fe]                heap_end_ptr
0x022a..0x022b  orig=[00]           patched=[02]                   cmd_line_ptr

Proposal

Normalize the setup header on both sides so the measured bytes are
version-independent and still equal the shipped file byte for byte:

  1. Image build — normalize the shipped bzImage in place.
  2. OVMF — apply the same normalization to the kernel blob before it is
    measured and loaded.
  3. dstack-mr — drop patched_kernel_authenticode, patch_kernel(), the
    version branch, the qemu_patches_kernel_header escape hatch, and the
    memory_size == 2 GiB || >= 0xB0000000 restriction (that guard only exists
    because the patched digest depends on guest RAM).

Normalization is idempotent, so it produces the same bytes whether or not QEMU
rewrote anything. Because the shipped file is normalized too, the measured value
remains a plain Authenticode hash of the artifact in sha256sum.txt — no
dstack-specific derived digest.

Normalization rule

Defined against the boot protocol, not against QEMU's behavior, so it does
not go stale when QEMU changes:

Zero every field Documentation/arch/x86/boot.rst types as write
(bootloader writes it; the kernel supplies no value), and clear
CAN_USE_HEAP (0x80) in loadflags. Leave every modify field alone.

That covers type_of_loader (0x210), ramdisk_image/ramdisk_size
(0x218/0x21c), heap_end_ptr (0x224), ext_loader_ver/ext_loader_type
(0x226/0x227), cmd_line_ptr (0x228), hardware_subarch[_data] (0x23c/0x240)
and setup_data (0x250).

Do not widen this to "all bootloader-writable fields". modify fields carry
real kernel-supplied values and zeroing them breaks the kernel:

field type offset build-time value
code32_start modify (optional, reloc) 0x214 0x100000 (protected-mode entry)
setup_move_size modify (obligatory) 0x212 0x8000
vid_mode modify (obligatory) 0x1fa 0xffff
loadflags modify (obligatory) 0x211 0x01
root_flags modify (optional) 0x1f2 0x01

vid_mode is the one modify field QEMU can write, but only when the cmdline
contains vga= — and the cmdline is measured into RTMR[2], so it cannot be
smuggled in. Leaving it alone is safe.

Impact on the shipped kernel: 2 bytes

Every write field is already zero in a freshly built bzImage except
heap_end_ptr (0x224 = 0x50a0). So normalization changes exactly two bytes.

This is safe for non-TDX boot:

  • boot.rst types it write (obligatory): the bootloader writes it, the kernel
    supplies no value.
  • init_heap() reads it only when the bootloader has set CAN_USE_HEAP, and
    that flag is itself bootloader-set (zero at build time). Any bootloader that
    sets the flag is obliged by the spec to set heap_end_ptr too.
  • On the EFI-stub path the real-mode setup code never runs at all.
  • The PE headers occupy 0x40..0x170; every setup-header field is outside them,
    so the PE structure and EFI-stub entry are untouched.

Secure Boot is not planned, so there is no signature to invalidate and no
sign-vs-normalize ordering constraint.

Implementation

1. Image build (shared by Yocto and mkosi)

Both backends already funnel through os/image/assemble.sh
(os/yocto/mkimage.sh execs it; mkosi calls it from mkosi.postoutput), so one
insertion point covers both. Add a helper next to the existing
authenticode_hash.py / kernel-cmdline.sh, e.g.
os/image/normalize-kernel-header.py, and call it right after the kernel is
staged:

  verbose cp "$KERNEL_IMAGE" "${OUTPUT_DIR}/bzImage"
+ verbose "$(dirname "${BASH_SOURCE[0]}")/normalize-kernel-header.py" "${OUTPUT_DIR}/bzImage"

assemble.sh:413 runs before tdx-measurement-cbor (:469) and before
CHECKSUM_FILES (:590), so the measurement document and sha256sum.txt
automatically cover the normalized kernel.

The script should be idempotent and should fail loudly if the setup header does
not look like a supported boot protocol version.

Open question: the optional UKI image embeds the kernel in a .linux PE
section. Check whether it needs the same treatment or is unaffected because it
is not loaded via -kernel.

2. OVMF

OvmfPkg/QemuKernelLoaderFsDxe builds the virtual kernel file from
QemuFwCfgItemKernelSetupData ++ QemuFwCfgItemKernelData (see
mKernelBlobItems). That blob is what gBS->LoadImage measures and runs on
the EFI-stub path, so normalizing it there keeps measured bytes == loaded bytes
== file bytes — no "attested something other than what ran" gap.

Note X86QemuLoadImageLib keeps a separate SetupBuf for the legacy
LoadLinux* path; dstack does not use it and it should not be touched.

Worth attempting upstream: this is the missing half of Gerd Hoffmann's 2024
series (f2594d92844, "x86/loader: expose unpatched kernel", whose commit
message ends "Needs OVMF changes too to be actually useful") and it fixes the
same class of Secure Boot breakage that motivated that series. Note that the
etc/boot/kernel fw_cfg file it added only exists in QEMU >= 10.1, which is why
normalization is preferable for us: it works on 8.2.2 too.

3. dstack-mr / dstack-types

Only after images with the new OVMF are the floor. Until then #1183's
patch_kernel_header switch is still needed to verify CVMs running older
firmware.

Documentation

  • os/image/README.md — what the script does and why, so a bit-for-bit
    reproducibility check does not trip over bzImage differing from the raw
    kernel build output.
  • docs/security/security-model.md — replace the "A host-declared QEMU version
    selects between digests" section: once this lands the host no longer has that
    choice at all, which is a strictly stronger property.
  • os/yocto/repro-build/check.sh — make sure the normalization step is part of
    the reproducibility story.

Ordering

Land after #1183. That PR is the fix for the currently broken production path
and it provides the patch_kernel_header switch this transition depends on.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions