Repository navigation
fix(libs): key library records on the executable's linker build ID - #1506
Conversation
A modified working tree fell through to the main module's version, which since Go 1.24 is <pseudo-version>+dirty for every edit of a commit, so two differently patched builds of one commit shared every cache record. The build identity is now the linker's build ID read from the running executable, derived from every compiled input; a modified tree is never identified by its module version. Co-Authored-By: jason.han <hanhuijun@gmail.com>
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
Co-Authored-By: jason.han <hanhuijun@gmail.com>
|
The second push only annotates the |
…r build ID A clean checkout's builds share records however the link was stamped (the Makefile stamps main.BuildTime on every build); a modified tree or an unstamped build is identified by the linker's build ID. Shortens the new code comments. Co-Authored-By: jason.han <hanhuijun@gmail.com>
|
e956947 changes the order of the identity: a clean checkout is identified by its VCS revision first, as before, so repeated |
What and why
The library index cache under
~/.cache/sysml-ls/libsserved one build's records to another build of the same commit. Every key carries a build identity (internal/workspace/libs/buildid.go), meant to be a clean VCS revision, else something unique to the build. A modified working tree was rejected by the revision branch but then fell through to the main module's version — and since Go 1.24 the toolchain stamps a modified tree's main module as<pseudo-version>+dirty, one value for every edit of that commit. Two dirty-tree builds of one commit with different patches (or a worktree build and a differently patched main-tree build) therefore computed the same-b<buildID>component and read each other's records.Reproducer, both builds at
76c98694cwithvcs.modified=trueandMain.Version = v0.11.1-0.20261025041301-76c98694c2da+dirty, build A carrying a one-line change to the unit reduction a record persists (ScaleNum: term.Scale.Num + 1), build B unpatched, oneXDG_CACHE_HOME, a copy of the library underOPENSYSML_LIBRARY_PATHso the on-disk cache is used:Of the candidate causes, this is the only one that holds: the key does include the content digest of every library source (
-s<setDigest>and the per-file hash, which is whyOPENSYSML_LIBRARY_PATHparticipates), the executable size+mtime path was not reached, and validation is not header-only (the whole record is gob-decoded and a decode failure is a miss). The gap was solely the build identity of a modified tree.The fix (
buildIdentity) identifies a build as, in order: a clean checkout's VCS revision (vcs.modified=false), which every build of that checkout shares however the link was stamped — the Makefile stampsmain.BuildTimeon every build, and that must not cost a cold start; else the linker's build ID of the running executable, read the waygo tool buildidreads it (ELF Go note, or the marker at the start of the text for Mach-O and PE), which the toolchain derives from every compiled input and link flag, embedded library files included; else a released module version. So:make build) share records, as before;<pseudo-version>+dirtymodule version is refused. The size+mtime and per-process fallbacks remain for an executable without VCS data linked with-ldflags=-buildid=or one that cannot be read (wasm).With the fix, the same two builds:
Configuration.
OPENSYSML_LIBRARY_PATHchanges the library set and is covered by the set digest and per-file hash in the key.OPENSYSML_LIBRARY_ROOTSadds project documents to a workspace, which are interface records keyed byIndex.LibraryIdentity(digest of every library document's text, tier and kind). No otherOPENSYSML_*variable changes what a library record holds (OPENSYSML_SHARED_DEFAULTS,OPENSYSML_MAX_INTEGER_BITS,OPENSYSML_CALC_COMPILEare runtime settings;OPENSYSML_RECORD_CACHEonly disables interface-record caching).Format compatibility. A record written by a newer format is unreachable by an older binary:
-v<formatVersion>is part of the file name, and an undecodable file at a matching name is a miss (TestCacheRefusesRecordsOfAnotherFormat).Existing caches on upgrade. Nothing is deleted. Every record written under the old identity is unreachable from the new key and is removed by
Cache.Pruneafter thirty idle days; the first start after upgrading is a cold one for the on-disk cache (the bundled library is served from the embedded snapshot and never touches it).How it was verified
internal/workspace/libs:TestBuildIdentityOrder(clean checkout → revision whatever the linker ID; modified tree → linker ID, never metadata; module build → linker ID, else version),TestBuildIDFromInfoRejectsDirtyModuleVersion(fails ondevelop:dirty tree identified by its module version as "mv0.11.1-0.20261025041301-76c98694c2da+dirty"),TestCurrentBuildIDIsTheLinkerBuildID(linkerBuildIDequalsgo tool buildidof the test binary, and the identity is that ID unless the binary records a clean checkout),TestLinkerBuildIDTellsBuildsApart(two-ldflags -Xbuilds ofcmd/sysmlget distinct IDs),TestLinkerBuildIDOfOtherFiles,TestCacheKeySameInputsSameKey,TestCacheRefusesRecordsOfAnotherFormat, andTestRecordOfAnotherBuildIsNotServed(reproducer shape under a temporaryXDG_CACHE_HOME: a record one build wrote is served to that build and refused by another). The existingTestCacheKeyDependsOnBuildIDandTestCacheStaleContentReparsedNotServedstill cover the content and build axes.e9569475dwith-X main.BuildTime=…10:00:00Zand…10:01:00Z(distinctgo tool buildid),OPENSYSML_LIBRARY_PATHcopy of the standard library, oneXDG_CACHE_HOME: first cold 0.49 s, second warm on the first's records 0.12 s, 110 records under one-b32fb9d9799c1b987-v33.idxsuffix; a build of the same tree with one edited line (vcs.modified=true) misses them and writes 110 records under-bb5da97b2dc31d9d1-v33.idx.sysml -e '3 istype ScalarValues::Integer', wall seconds, three runs each (develop → branch):OPENSYSML_LIBRARY_PATHcopy of the library (110 records): cold 0.52/0.42/0.45 → 0.58/0.37/0.35; warm 0.17/0.14/0.14 → 0.11/0.15/0.12BenchmarkLinkerBuildID, not committed).OccurrencesTest::p1.vandnew OccurrencesTest::P1().vread2on develop and on the branch.go build ./...,go vet ./...,gofmt -l .empty,make lint(✓ Lint passed),python3 scripts/changelog.py check,python3 scripts/check-doc-ids.py,go test ./internal/workspace/libs(ok, 30.8 s).go test ./internal/exec/runtime/...alone: ok 601 s;go test ./cmd/sysml/...alone: ok 576 s.go test -count=1 ./...with the four corpora downloaded andOPENSYSML_REQUIRE_TRAINING_CORPUS,OPENSYSML_REQUIRE_PILOT_CORPORA,OPENSYSML_REQUIRE_SPEC_EXAMPLES,OPENSYSML_REQUIRE_PILOT_LIBRARY_XMIset: 117 packages ok;cmd/sysml,internal/exec/runtime,internal/exec/smt,tests/codegenhit Go's 10 min default under the parallel run and pass alone (601 s, 601 s, 289 s, 367 s);internal/frontend/lsppasses alone (16 s) once no model tree sits under/tmp, which its wire tests use as workspace root.go test -C tools ./...ok.tests/corpus/testdata/training_examples_expected.txtuntouched.bytakes a feature chain; a satisfy stating nobyreads its owner as the subject #1466 anddevelopat111a59a2a.Checklist
make testandmake lintpass locallydocs/project/lossless-library-records.md, the testing skills under.agents/skillsthat probe cold/warm cache semantics now say to rely on the key rather than clear the cache)changes/unreleased/library-cache-build-identity.fixed.mdLink to Devin session: https://nasa-jpl-demo.devinenterprise.com/sessions/e1198cc50f4e45d69bbc71459615541c
Open in Devin Desktop: https://nasa-jpl-demo.devinenterprise.com/desktop/session/e1198cc50f4e45d69bbc71459615541c?variant=devin
Requested by: @HuiJun