Skip to content

fix(download): stop corrupting BuildKit layers when a cache download resumes - #316

Merged
LouisHaftmann merged 5 commits into
devfrom
fix/download-x-ms-range
Nov 13, 2026
Merged

LouisHaftmann merged 5 commits into
devfrom
fix/download-x-ms-range

Conversation

@LouisHaftmann

@LouisHaftmann LouisHaftmann commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

📝 Summary

BuildKit builds that used cache-from: type=gha sometimes failed with corrupted layers (#315). BuildKit reads the cache through the Azure blob SDK. When a download drops, BuildKit retries it, and the SDK asks for the rest of the blob with x-ms-range: bytes=N- only. The server read only Range, so it answered with a 200 and the whole blob from byte 0. The SDK accepts any 200, and BuildKit wrote those bytes at offset N.

Now x-ms-range wins over Range, as in Azure Get Blob. A ranged read never gets the whole entry with a 200:

 GET /download/{id}
-  range = Range
+  range = x-ms-range ?? Range
   merged or single Part     -> 206 from the object (unchanged)
-  unmerged, several Parts   -> 200 + whole entry, range ignored
+  unmerged, several Parts   -> 206: one LIST gives the Part sizes, the read starts
+                                in the Part where the range begins, then stops at its end

A ranged read of an unmerged Storage Location no longer starts a Merge. The next full download still starts one, and BuildKit's first read of a blob is always a full download.

✅ Verification

Real BuildKit v0.34.1, end to end. A fresh buildkitd exports the build cache to the server (filesystem storage, SQLite). A second fresh buildkitd imports it. A proxy cuts one 30 MB layer download after 5 MB.

  • Before: BuildKit retries with only x-ms-range and gets 200 with the whole blob each time. The build fails:
    GET /download/7bc2…  x-ms-range="bytes=5000000-" -> 200 Content-Length=30009430
    GET /download/7bc2…  x-ms-range="bytes=6015808-" -> 200 Content-Length=30009430   (×24)
    ERROR: failed to compute cache key: failed commit on ref "layer-sha256:…" failed size validation: 30165824 != 30009430
    
  • After: the same run passes with lazy merge (26 × 206) and with EAGER_MERGE=true (24 × 206). No ranged read got a 200.
    #8 0.155 /big: OK
    

Tests, written first and failing on dev:

x_ms_range_wins_over_range                        x-ms-range alone; x-ms-range + Range
serves_ranges_of_merged_entries                   range table via Range
serves_ranges_of_unmerged_entries_and_pins_...    range table via Range and x-ms-range on 3 unmerged Parts:
                                                  across a Part boundary, starting on one, suffix, clamped,
                                                  416 past the end; the entry stays unmerged
  • cargo +nightly fmt --check and cargo clippy --all-targets --locked: clean.
  • cargo test: 83/83 on SQLite with filesystem, S3, GCS and Azure Blob storage.
  • pnpm check && pnpm build in website/: passes.

Benchmark (cargo run --release --example benchmark), with images built from dev and from this branch:

Metric dev pr
Startup to healthy 565.7 ms 501.3 ms
Idle / peak memory 3.3 / 101.5 MiB 3.2 / 89.6 MiB
Upload (8 × 128 MiB, parallel) 2169 MiB/s 2148 MiB/s
First download (merge on read) 7336 MiB/s 6845 MiB/s
Later download (merged) 3319 MiB/s 3337 MiB/s
Lookups per second (64 concurrent) 30544 32555
Lookup p50 / p99 1.8 / 14.7 ms 1.9 / 3.8 ms
Small save+restore per second 1058 1067

First download changes the most between runs. Over 3 runs it measured 6840–7336 MiB/s on dev and 6407–7679 MiB/s on this branch. The ranges overlap, so the difference is noise.

🔍 Reviews

Review Found Fixed
bug-review (2 rounds) 1 1
spec-review 3 2
simplicity-review 2 2
  • The bug review found that skipping Parts with 416 probes cost one storage round trip per skipped Part. BuildKit reopens the download on every read once it has resumed, so this grew quadratically. Fixed: list_folder now returns sizes by name, so the LIST the read already makes gives the start Part.
  • The spec review asked for the docs page, which is updated, and for the hot-path benchmark, which is below. Its third finding, that ranged reads no longer start a Merge, is intended: a partial read cannot feed the Merge that object storage builds from a full download.

🔗 Issues

Closes #315

💥 Merge danger

Door: two-way. No schema or stored data changes.

Blast Radius: downloads

Every download goes through this path. Full downloads keep their old code path. Only ranged reads of unmerged multi-Part entries change their answer, from 200 to 206.

⚠️ Needs attention

  • With ENABLE_DIRECT_DOWNLOADS, S3 and GCS signed URLs ignore x-ms-range: the RustFS and fake-gcs test services answer 200 with the whole object. The server cannot fix this, so configuration.md now tells BuildKit users on S3 or GCS to leave direct downloads off. Azure Blob is not affected.
  • In go-actions-cache, readerAtCloser does not set its offset after it reopens a download. So after one resume, every later read reopens the download. That is an upstream bug in tonistiigi/go-actions-cache.

🤖 Generated with Claude Code

LouisHaftmann and others added 5 commits November 13, 2026 15:55
The Azure SDK, and with it BuildKit's type=gha cache, resumes a dropped download with only `x-ms-range: bytes=N-` and takes any 200 body as the bytes from offset N. The server only read `Range`, so a resume got the whole blob from byte 0 and BuildKit spliced it in at N.

`x-ms-range` now wins over `Range`, like Azure Get Blob. A ranged read of an unmerged entry is served as a 206 from its Parts: Parts wholly before the range cost one ranged read answered 416, and the stream stops at the end of the range. Such a read no longer starts a Merge; the next whole read does.

Closes #315

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…isting

Skipping Parts with a 416 probe each cost one storage round trip per skipped Part, and BuildKit reopens a download on every read once it has resumed. `list_folder` now returns sizes by name, so the Part sizes from the existing listing locate the start Part directly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dKit resumes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

🚀 Deploying Preview to Cloudflare 🚀

Preview URL: https://fix-download-x-ms-range-docs-gha-cache-server.louis-haftmann.workers.dev (commit 40cdd88)

This URL reflects your latest Preview deployment

Preview Deployments by commit

Status Deployment URL Commit Updated (UTC) See this deployment's details
  • Build: Success ✅
  • Deployment: Success ✅

View logs ↗
https://14def892-docs-gha-cache-server.louis-haftmann.workers.dev 40cdd88 2026-11-13T15:32:55.991Z Visit the dashboard ↗
  • Build: Success ✅
  • Deployment: Success ✅

View logs ↗
https://347d9658-docs-gha-cache-server.louis-haftmann.workers.dev f3796d2 2026-11-13T15:38:29.984Z Visit the dashboard ↗

@LouisHaftmann
LouisHaftmann merged commit 9723b3f into dev Nov 13, 2026
14 of 15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BuildKit type=gha restores can get corrupted layers: downloads ignore x-ms-range

1 participant