Skip to content

fix(libs): key library records on the executable's linker build ID - #1506

Merged
HuiJun merged 3 commits into
developfrom
fix/cache-key-build-identity
Oct 25, 2026
Merged

HuiJun merged 3 commits into
developfrom
fix/cache-key-build-identity

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

What and why

The library index cache under ~/.cache/sysml-ls/libs served one build's records to another build of the same commit. Every key carries a build identity (internal/workspace/libs/buildid.go), meant to be a clean VCS revision, else something unique to the build. A modified working tree was rejected by the revision branch but then fell through to the main module's version — and since Go 1.24 the toolchain stamps a modified tree's main module as <pseudo-version>+dirty, one value for every edit of that commit. Two dirty-tree builds of one commit with different patches (or a worktree build and a differently patched main-tree build) therefore computed the same -b<buildID> component and read each other's records.

Reproducer, both builds at 76c98694c with vcs.modified=true and Main.Version = v0.11.1-0.20261025041301-76c98694c2da+dirty, build A carrying a one-line change to the unit reduction a record persists (ScaleNum: term.Scale.Num + 1), build B unpatched, one XDG_CACHE_HOME, a copy of the library under OPENSYSML_LIBRARY_PATH so the on-disk cache is used:

A, cold cache:        (1 [SI::km]) + (1 [SI::m])  =  1003/1001 [SI::km]
B, same cache:        (1 [SI::km]) + (1 [SI::m])  =  1003/1001 [SI::km]   <- A's records
B, fresh cache:       (1 [SI::km]) + (1 [SI::m])  =  1.001 [SI::km]
record names:         both builds wrote/read *-b453aae83b2d2e012-v33.idx

Of the candidate causes, this is the only one that holds: the key does include the content digest of every library source (-s<setDigest> and the per-file hash, which is why OPENSYSML_LIBRARY_PATH participates), the executable size+mtime path was not reached, and validation is not header-only (the whole record is gob-decoded and a decode failure is a miss). The gap was solely the build identity of a modified tree.

The fix (buildIdentity) identifies a build as, in order: a clean checkout's VCS revision (vcs.modified=false), which every build of that checkout shares however the link was stamped — the Makefile stamps main.BuildTime on every build, and that must not cost a cold start; else the linker's build ID of the running executable, read the way go tool buildid reads it (ELF Go note, or the marker at the start of the text for Mach-O and PE), which the toolchain derives from every compiled input and link flag, embedded library files included; else a released module version. So:

  • builds of different code — a dirty tree, a worktree with one patch, a different resolver — never share a record;
  • builds of one clean checkout (a worktree at the same commit, a repeated make build) share records, as before;
  • a modified tree is never identified by build metadata: its revision is skipped and a <pseudo-version>+dirty module version is refused. The size+mtime and per-process fallbacks remain for an executable without VCS data linked with -ldflags=-buildid= or one that cannot be read (wasm).

With the fix, the same two builds:

A, cold cache:        1003/1001 [SI::km]      writes *-b047283a36e8a2700-v33.idx
B, same cache:        1.001 [SI::km]          misses, writes *-b45460c75b6946833-v33.idx
B again:              1.001 [SI::km]          warm on its own records

Configuration. OPENSYSML_LIBRARY_PATH changes the library set and is covered by the set digest and per-file hash in the key. OPENSYSML_LIBRARY_ROOTS adds project documents to a workspace, which are interface records keyed by Index.LibraryIdentity (digest of every library document's text, tier and kind). No other OPENSYSML_* variable changes what a library record holds (OPENSYSML_SHARED_DEFAULTS, OPENSYSML_MAX_INTEGER_BITS, OPENSYSML_CALC_COMPILE are runtime settings; OPENSYSML_RECORD_CACHE only disables interface-record caching).

Format compatibility. A record written by a newer format is unreachable by an older binary: -v<formatVersion> is part of the file name, and an undecodable file at a matching name is a miss (TestCacheRefusesRecordsOfAnotherFormat).

Existing caches on upgrade. Nothing is deleted. Every record written under the old identity is unreachable from the new key and is removed by Cache.Prune after thirty idle days; the first start after upgrading is a cold one for the on-disk cache (the bundled library is served from the embedded snapshot and never touches it).

How it was verified

  • internal/workspace/libs: TestBuildIdentityOrder (clean checkout → revision whatever the linker ID; modified tree → linker ID, never metadata; module build → linker ID, else version), TestBuildIDFromInfoRejectsDirtyModuleVersion (fails on develop: dirty tree identified by its module version as "mv0.11.1-0.20261025041301-76c98694c2da+dirty"), TestCurrentBuildIDIsTheLinkerBuildID (linkerBuildID equals go tool buildid of the test binary, and the identity is that ID unless the binary records a clean checkout), TestLinkerBuildIDTellsBuildsApart (two -ldflags -X builds of cmd/sysml get distinct IDs), TestLinkerBuildIDOfOtherFiles, TestCacheKeySameInputsSameKey, TestCacheRefusesRecordsOfAnotherFormat, and TestRecordOfAnotherBuildIsNotServed (reproducer shape under a temporary XDG_CACHE_HOME: a record one build wrote is served to that build and refused by another). The existing TestCacheKeyDependsOnBuildID and TestCacheStaleContentReparsedNotServed still cover the content and build axes.
  • Repeated stamped builds of a clean checkout share records: two builds of e9569475d with -X main.BuildTime=…10:00:00Z and …10:01:00Z (distinct go tool buildid), OPENSYSML_LIBRARY_PATH copy of the standard library, one XDG_CACHE_HOME: first cold 0.49 s, second warm on the first's records 0.12 s, 110 records under one -b32fb9d9799c1b987-v33.idx suffix; a build of the same tree with one edited line (vcs.modified=true) misses them and writes 110 records under -bb5da97b2dc31d9d1-v33.idx.
  • Cold/warm start, sysml -e '3 istype ScalarValues::Integer', wall seconds, three runs each (develop → branch):
    • bundled library (embedded snapshot, cache dir stays empty): cold 0.03/0.05/0.04 → 0.03/0.02/0.03; warm 0.05/0.03/0.03 → 0.02/0.03/0.03
    • OPENSYSML_LIBRARY_PATH copy of the library (110 records): cold 0.52/0.42/0.45 → 0.58/0.37/0.35; warm 0.17/0.14/0.14 → 0.11/0.15/0.12
    • reading the linker build ID: 26 µs per call, once per process (BenchmarkLinkerBuildID, not committed).
  • OccurrencesTest::p1.v and new OccurrencesTest::P1().v read 2 on develop and on the branch.
  • go build ./..., go vet ./..., gofmt -l . empty, make lint (✓ Lint passed), python3 scripts/changelog.py check, python3 scripts/check-doc-ids.py, go test ./internal/workspace/libs (ok, 30.8 s).
  • go test ./internal/exec/runtime/... alone: ok 601 s; go test ./cmd/sysml/... alone: ok 576 s.
  • go test -count=1 ./... with the four corpora downloaded and OPENSYSML_REQUIRE_TRAINING_CORPUS, OPENSYSML_REQUIRE_PILOT_CORPORA, OPENSYSML_REQUIRE_SPEC_EXAMPLES, OPENSYSML_REQUIRE_PILOT_LIBRARY_XMI set: 117 packages ok; cmd/sysml, internal/exec/runtime, internal/exec/smt, tests/codegen hit Go's 10 min default under the parallel run and pass alone (601 s, 601 s, 289 s, 367 s); internal/frontend/lsp passes alone (16 s) once no model tree sits under /tmp, which its wire tests use as workspace root. go test -C tools ./... ok.
  • No golden, baseline or fixture moved; tests/corpus/testdata/training_examples_expected.txt untouched.
  • Dry-merges clean against fix(runtime): derive implicit specializations for every performance the run creates #1492, fix(parser,runtime): satisfy by takes a feature chain; a satisfy stating no by reads its owner as the subject #1466 and develop at 111a59a2a.

Checklist

  • make test and make lint pass locally
  • Tests added or updated for the change
  • Documentation extended where it already covers the surface (docs/project/lossless-library-records.md, the testing skills under .agents/skills that probe cold/warm cache semantics now say to rely on the key rather than clear the cache)
  • Changelog entry added as changes/unreleased/library-cache-build-identity.fixed.md
  • No baseline or gate count moved
  • No internal work-item labels in the body, docs, or changelog

Link to Devin session: https://nasa-jpl-demo.devinenterprise.com/sessions/e1198cc50f4e45d69bbc71459615541c
Open in Devin Desktop: https://nasa-jpl-demo.devinenterprise.com/desktop/session/e1198cc50f4e45d69bbc71459615541c?variant=devin
Requested by: @HuiJun

A modified working tree fell through to the main module's version, which
since Go 1.24 is <pseudo-version>+dirty for every edit of a commit, so two
differently patched builds of one commit shared every cache record. The
build identity is now the linker's build ID read from the running
executable, derived from every compiled input; a modified tree is never
identified by its module version.

Co-Authored-By: jason.han <hanhuijun@gmail.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

Co-Authored-By: jason.han <hanhuijun@gmail.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

The second push only annotates the os.Open of the running executable in linkerid.go for gosec G304 (make lint reported one issue; the path comes from os.Executable, not from input). No behaviour change; the gates on the first commit stand.

@devin-ai-integration
devin-ai-integration Bot marked this pull request as ready for review October 25, 2026 09:51
devin-ai-integration[bot]

This comment was marked as resolved.

…r build ID

A clean checkout's builds share records however the link was stamped
(the Makefile stamps main.BuildTime on every build); a modified tree or
an unstamped build is identified by the linker's build ID. Shortens the
new code comments.

Co-Authored-By: jason.han <hanhuijun@gmail.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

e956947 changes the order of the identity: a clean checkout is identified by its VCS revision first, as before, so repeated make builds of an unchanged tree — which the Makefile stamps with a fresh main.BuildTime, changing the linker's build ID — keep sharing their library records (measured: second stamped build warm on the first's records, 0.12 s against 0.49 s cold, one -b suffix for all 110 records); a modified tree or a build without VCS data is identified by the linker's build ID, and a +dirty module version still identifies nothing. The helper buildIdentity(info, linkerID) makes the order a pure function, pinned by TestBuildIdentityOrder; TestCurrentBuildIDIsTheLinkerBuildID now expects the revision when the test binary records a clean checkout. The new code comments are cut to two lines, and the docs, changelog fragment and testing skills say "a clean checkout's revision, else the linker build ID".

@HuiJun
HuiJun merged commit b43ad21 into develop Oct 25, 2026
30 checks passed
@HuiJun
HuiJun deleted the fix/cache-key-build-identity branch October 25, 2026 16:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant