Fix precompiled image publish silently dropping non-native-arch manifests - #873
Fix precompiled image publish silently dropping non-native-arch manifests#873shivakunv wants to merge 3 commits into
Conversation
ffc8d93 to
dc84491
Compare
…stead of oci-archive Signed-off-by: Shiva Kumar (SW-CLOUD) <shivaku@nvidia.com>
47818c3 to
df08d06
Compare
|
@shivakunv Can you update the PR title and description to better reflect what you're doing in this PR? |
updated |
tariq1890
changed the title
c857c08 to
e89d63f
Compare
abrarshivani
changed the title
|
Verification:
|
sylvesterkaczmarek
left a comment
sylvesterkaczmarek
left a comment
There was a problem hiding this comment.
The index comparison keeps only architecture and digest, so platform metadata such as os and variant is ignored. Those fields live on the index descriptor rather than the child manifest digest, so a published index can carry the right digest under the wrong platform descriptor and still pass this check. Compare the full relevant platform tuple plus digest and add an ARM variant regression.
skopeo copy oci-archive:file.tar docker://... without --all doesn't error on a multi-platform (amd64+arm64) archive, it silently collapses to a single current-arch manifest, so the published image only had one architecture even though the build produced both. --all on the existing oci-archive: source is sufficient; no need to extract the archive to an oci: directory first. skopeo copy exiting 0 isn't proof the full index was published, so also compare the local artifact's manifest against what's actually live in the registry after the push. verify-manifest-match.sh handles both shapes this repo's build matrix produces: an index (multi-arch legs) compares platform tuple plus digest per entry, filtering out attestation manifests; a plain manifest (the amd64-only legs) compares config and layer digests. Each side determines its own shape independently and rejects empty or degenerate input, so a shape mismatch between local and remote reports as a clean failure instead of a jq crash, and garbage input can't pass by comparing two empty results to each other. Signed-off-by: Abrar Shivani <ashivani@nvidia.com>
A real skopeo copy only ever exercises the happy path (a healthy copy always matches its source), so the failure-detection branch needs deliberately mismatched fixtures to prove it actually fires. Covers matching and mismatched indexes, a dropped architecture, a crossed architecture/variant/digest pairing that an arch-only comparison key couldn't have caught, matching and mismatched single-arch manifests, a local/remote shape mismatch, an index that resolves to no real platforms, and empty input. precompiled.yaml only triggers on schedule/workflow_dispatch, so wire this into image.yaml instead: the one workflow in this repo that actually runs on pull requests, as its own lightweight job rather than inside the existing build matrix. Signed-off-by: Abrar Shivani <ashivani@nvidia.com>
9557886 to
b266c3e
Compare
|
@sylvesterkaczmarek Thanks for catching this. Updated the comparison to include the full platform tuple ( |
rajathagasthya
left a comment
rajathagasthya
left a comment
There was a problem hiding this comment.
Thanks for identifying the root cause @abrarshivani!
sylvesterkaczmarek
left a comment
sylvesterkaczmarek
left a comment
There was a problem hiding this comment.
Re-reviewed the platform-identity finding. Index summaries now compare OS, architecture, variant and digest, and the added ARM-variant regression covers the crossed-variant case I raised. My previous blocker is resolved.
4 participants
The publish step for precompiled driver images used
skopeo copy oci-archive:file.tar docker://...without--all. For a multi-platform (amd64+arm64) OCI archive, skopeo doesn't error on that, it silently collapses to a single current-arch manifest, so the pushed image only had one architecture even though the build produced both.Fix: add
--allto the existingoci-archive:copy. I verified locally against the exact skopeo version CI installs (1.13.3+ds1-2ubuntu0.24.04.3, pulled from a real scheduled run's logs) and against a real push toghcr.iothat--allalone is enough, no need to extract the archive to anoci:directory first like the original fix did.Also kept and adjusted the post-publish verification, since a
skopeo copythat exits 0 isn't proof the full index was published (that's exactly how this bug shipped unnoticed in the first place). It compares the local artifact's manifest against what's actually live in the registry, and now handles both shapes correctly: an index (the multi-arch legs) compares per-platform digests, a plain manifest (ubuntu22.04, ubuntu26.04, and ubuntu24.04 azure-fde, which build amd64-only permulti-arch.mk) compares config/layer digests instead of silently passing on[] == [].One correction on the linked skopeo issue: that issue is a different, still-open failure (a hard crash on a flat index with no attestation manifests), not the silent-drop behavior described above. Related context, but not the exact bug we're hitting here, that's just the missing
--allflag.