Attach a static linux/arm64 seid binary to releases - #4004
Conversation
Releases carried linux/amd64 only. The musl link path was arch-split in #3932, which made an arm64 static build possible; this wires it through the build script, the CI gate and goreleaser so the archive actually ships. Vendor the aarch64 Alpine 3.15 gcc 10.3.1 libgcc alongside the x86_64 one, each in its own subdirectory. The gcc>=12 unwind b-tree that crashed the amd64 binary at the genesis wasm store is not architecture specific: ATOMIC_FDE_FAST_PATH is gated on atomics support rather than a target allowlist, and an unpinned arm64 build SIGSEGVs on the first boot under RAYON_NUM_THREADS=1. Both targets need the pin. build-static.sh takes a target architecture and writes build/seid-<arch>, so the two builds no longer overwrite each other on one runner. The libgcc directory and the checksums it verifies are both derived from that argument, so a build cannot verify one architecture's archives while linking another's. It also asserts the ELF machine of the output matches what was asked for. The nm b-tree assertion now materialises the symbol table before grepping it. Reading nm through a pipe reported grep's exit status, so a failing nm would print "pre-b-tree unwinder confirmed" and pass. boot-smoke.sh refuses a binary built for another architecture. Previously it reached seid init, died with an exec-format error and reported "did not reach the ABCI handshake", which reads as a crashing binary rather than the wrong input. The release runner is amd64 and cannot execute the arm64 binary, so the goreleaser hook boots only the amd64 one. The arm64 boot gauntlet runs in the new Linux ARM64 (static) job on native hardware. The static build is reproducible, so the binary gated there is byte-identical to the one that ships. Verified with goreleaser release --snapshot: both archives are produced, both are listed in checksums.txt, sha256sum -c --ignore-missing reports OK for both, and each archive contains a binary of the matching architecture.
PR SummaryMedium Risk Overview
CI updates the existing Linux amd64 static job to the new artifact paths and adds a Linux ARM64 (static) job on native Reviewed by Cursor Bugbot for commit b798983. Bugbot is set up for automated code reviews on this repo. Configure here. |
|
The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).
|
![seidroid[bot]](https://avatars.githubusercontent.com/in/2745135?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cleanly executed arch-split of the static build: the per-arch libgcc pin, checksums, output naming, and the ELF-machine guards in build-static.sh / boot-smoke.sh / goreleaser-shim.sh all line up and fail closed. Two things to address: the new binfmt hook runs an unpinned third-party image with --privileged on the release runner, and the b-tree symbol assertion still has no positive control.
Findings: 1 blocking | 2 non-blocking | 2 posted inline
Blockers
- None at the file/PR level.
- 1 blocking issue(s) flagged inline on specific lines.
Non-blocking
- [suggestion] The release-built
linux/arm64binary is never booted; the safety argument rests on it being byte-identical to the natively-built binary that the newLinux ARM64 (static)job runs the 8-boot gauntlet against. Nothing enforces that.scripts/build-static.shrunsapk add --no-cache build-baseinside the pinned golang:1.25.6-alpine image, which resolves Alpine package versions from the live index at build time, so the CI build and a later release build can pick up different gcc/binutils and diverge. Consider printingsha256sumof the produced binary in both the CI job and the goreleaser hook (or pinning the apk package versions) so a divergence is visible rather than silent — otherwise the gate can be green for a binary that isn't the one shipped. - 1 suggestion(s)/nit(s) flagged inline on specific lines.
| # build-static.sh self-verifies (libwasmvm archives present + output is static). | ||
| - bash scripts/build-static.sh | ||
| # The release runner is amd64, so building linux/arm64 needs binfmt registered. | ||
| - docker run --privileged --rm tonistiigi/binfmt --install arm64 |
There was a problem hiding this comment.
[blocker] This runs a mutable latest tag of a third-party image with --privileged on the release runner — the highest-trust machine in the repo, holding the credentials that publish seid binaries. Whoever controls that Docker Hub tag can execute arbitrary privileged code there, and nothing about the run is verified.
The repo pins everything comparable: the golang build image is pinned by digest three lines down in scripts/build-static.sh, every action is pinned by commit SHA, and docker_build.yml / docker_publish.yml already register binfmt via docker/setup-qemu-action@06116385d9baf250c9f4dcb4858b16962ea869c3. This hook is the one unpinned exception.
Fix is one line — pin the digest:
- docker run --privileged --rm tonistiigi/binfmt:qemu-v<version>@sha256:<digest> --install arm64(Also flagged by Codex.)
| # Materialise the symbol table as its own command so `set -e` aborts when nm fails. | ||
| # Reading nm through a pipe into grep reports the grep status, so a broken nm would | ||
| # otherwise read as "no b-tree symbols found" and pass. | ||
| nm '"$OUT"' > /tmp/seid.syms |
There was a problem hiding this comment.
[suggestion] The nm > file split correctly closes the pipe-status hole, but the assertion is still absence-only: if nm succeeds while emitting a short or empty symbol table (stripped output, a future -s -w in STATIC_EXTRA_LDFLAGS, an nm that skips the C objects), grep -q finds nothing and the build reports "pre-b-tree unwinder confirmed" — the same class of failing-open this change set out to fix.
Add a positive control so an unusable symbol table can't pass, e.g. assert a symbol the pinned libgcc must contribute:
nm '"$OUT"' > /tmp/seid.syms
grep -q __register_frame_info /tmp/seid.syms || {
echo "build-static: ERROR: no unwinder symbols in $OUT; the b-tree assertion below would pass vacuously" >&2
exit 1
}![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4004 +/- ##
==========================================
- Coverage 59.33% 58.25% -1.09%
==========================================
Files 2278 2179 -99
Lines 196008 184438 -11570
==========================================
- Hits 116307 107448 -8859
+ Misses 68949 67178 -1771
+ Partials 10752 9812 -940
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
…b-tree check The binfmt hook ran a mutable tag of a third-party image with --privileged on the release runner, which holds the credentials that publish the binaries. Everything comparable in this repo is pinned: the golang build image by digest, every action by commit SHA, and the docker workflows use a pinned setup-qemu-action. Pin the image by digest so this hook is not the exception. The b-tree assertion was absence-only, so an empty or truncated symbol table satisfied it vacuously: a stripped binary reports zero b-tree symbols and the build prints "pre-b-tree unwinder confirmed". Assert first that the symbol table contains __register_frame, the entry point wasmer calls and one every libgcc provides whether pinned or not, so its absence means the table is unusable rather than the pin being wrong. Verified against a stripped copy of the arm64 binary: the old assertion passes it, the positive control rejects it.
…uild digests The arm64 binary that ships was never booted. Its safety rested on being byte-identical to the natively built binary the Linux ARM64 (static) job boots, and nothing enforced that. build-static.sh runs apk add build-base against a live branch index, so the toolchain can move underneath the pinned image digest and change the output without anything noticing. Boot the arm64 binary in the release hooks, under emulation, so the artefact that ships is the one booted. Emulated timing is not trusted to surface the gcc>=12 unwind b-tree, so the native arm64 job still runs the full gauntlet on real hardware; the release hook is a liveness check on the shipped bytes. boot-smoke.sh now probes whether the host can execute the binary rather than comparing its ELF machine to uname -m. The equality check would have rejected exactly the case the release hook needs, since binfmt makes a foreign binary runnable, and probing execution tests the thing that matters instead of a proxy for it. It still fails closed, with a message naming binfmt. build-static.sh prints the sha256 of each binary it produces, at the one point every caller passes through, so a CI build and a later release build of the same commit can be compared rather than assumed equal. Verified locally: an emulated boot completes well inside boot-smoke's 25s cap (amd64 on arm64, the mirror of the release runner's case), the native path still passes, and a binary the host cannot execute is rejected with exit 1.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit b798983. Configure here.
| # The arm64 binary boots here under emulation, so the artefact that ships is the one | ||
| # booted. Emulated timing is not trusted to surface the gcc>=12 unwind b-tree, which | ||
| # is why the Linux ARM64 (static) job still runs the full gauntlet on real hardware. | ||
| - bash scripts/boot-smoke.sh build/seid-arm64 4 |
There was a problem hiding this comment.
Emulated boot hits native timeout
High Severity
The new goreleaser hook runs boot-smoke on build/seid-arm64 under qemu binfmt, but boot-smoke still uses a fixed 25s timeout sized for native boots. Emulated wasmer JIT at genesis is typically far slower than native, so a healthy arm64 binary can be killed before Completed ABCI Handshake and fail the entire release before: hook.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit b798983. Configure here.
2 participants
Releases carry
linux/amd64only. #3932 arch-split the musl link path, which made an arm64 static build possible; this wires it through the build script, the CI gate and goreleaser so the archive actually ships.Verified end to end
goreleaser release --snapshoton this branch:Both archives are listed in
checksums.txt,sha256sum -c --ignore-missingreports OK for both, and each archive contains a binary of the matching architecture. That last check matters because--ignore-missingexits 0 silently for a file that is present but unlisted, so an out of band upload would give operators a verification step that verifies nothing.The toolchain pin is needed on both architectures
The gcc>=12 unwind b-tree that crashed the amd64 binary at the genesis wasm store is not amd64 specific.
ATOMIC_FDE_FAST_PATHis gated on atomics support rather than a target allowlist, and an unpinned arm64 build SIGSEGVs on the first boot underRAYON_NUM_THREADS=1, reproduced on native arm64 hardware. So the Alpine 3.15 gcc 10.3.1 libgcc is now vendored for both, each in its own subdirectory, with provenance and checksums in the README.With the pin applied, the arm64 binary carries zero b-tree symbols and boots 8/8 clean at 4 CPUs.
Build script
build-static.shtakes a target architecture and writesbuild/seid-<arch>, so the two builds do not overwrite each other on one runner. The libgcc directory and the checksums it verifies are both derived from that one argument, so a build cannot verify one architecture's archives while linking another's. It also asserts the ELF machine of the output matches what was requested.Two guards that were failing open are now closed:
nmb-tree assertion materialises the symbol table before grepping. Readingnmthrough a pipe reported grep's exit status, so a failingnmprinted "pre-b-tree unwinder confirmed" and passed.boot-smoke.shrefuses a binary built for another architecture. It previously reachedseid init, died with an exec-format error, and reported "did not reach the ABCI handshake", which reads as a crashing binary rather than the wrong file being passed in.Where the arm64 binary gets booted
The release runner is amd64 and cannot execute the arm64 binary, so the goreleaser hook boots only the amd64 one. The arm64 8-boot gauntlet runs in the new
Linux ARM64 (static)job on nativeubuntu-24.04-armhardware.The link between them is reproducibility: the static build produces a byte-identical binary for a given commit (measured, identical
sha256across independent runs), so the binary CI boots is the one that ships. The release job builds arm64 under emulation via binfmt, which costs roughly 45 minutes and is why the boot gate lives on native hardware instead.Notes for review
ucichange needed. Docker and privileged runs already work on the release runner, so binfmt registration is just anotherbefore:hook.