Repository navigation
docs(perf): measure the 0.12.0 candidate against 0.11.0 - #1740
Merged
Merged
Conversation
Co-Authored-By: jason.han <hanhuijun@gmail.com>
Contributor
Author
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
Contributor
Author
There was a problem hiding this comment.
🔍 Devin Review: 3 flags
Not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)
…ation count does not bend Co-Authored-By: jason.han <hanhuijun@gmail.com>
…leaving them blank Co-Authored-By: jason.han <hanhuijun@gmail.com>
Co-Authored-By: jason.han <hanhuijun@gmail.com>
Co-Authored-By: jason.han <hanhuijun@gmail.com>
Merged
5 of 6 tasks
Co-Authored-By: jason.han <hanhuijun@gmail.com>
6 tasks done
Co-Authored-By: jason.han <hanhuijun@gmail.com>
Co-Authored-By: jason.han <hanhuijun@gmail.com>
Co-Authored-By: jason.han <hanhuijun@gmail.com>
…he quiet host Co-Authored-By: jason.han <hanhuijun@gmail.com>
5 of 6 tasks
…ort rows The record gains a second pass: the release candidate cc1d477 (develop ab88ea9 + #1730) measured with the full method against the two 0.11.0 columns, with 1c6bb19 kept as the pre-fix measurement so the seven fixes' recovery is visible row by row; the export rows re-probed on 5e0322e after #1760 landed on them; every merge up to develop b4d0a35 accounted for by diff or by its own benchstat; the quiet re-run's steal-noise stated rather than read; the record-cache cold-load effect found by the whole-binary sequences named. Co-Authored-By: jason.han <hanhuijun@gmail.com>
This was referenced Oct 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
The release-gate performance record for 0.12.0:
docs/project/performance-release-0.12.0-vs-0.11.0.md, following the method of the 0.11.0 record (every benchmark package,benchstatover six interleaved rounds, whole-binary timings on generated and real models, process floor, thetests/stressmodelconstellation), wired intomkdocs.yml(not-in-nav, after the 0.11.0 record),docs/project/README.md, the "latest is" link ofdocs/internals/performance.md, and achanges/unreleased/perf-0.12.0-record.added.mdfragment.The record measures three columns, because the toolchain moved from
go1.25.11togo1.27.3between the releases and dominates a naive comparison: 0.11.0 as released, the 0.11.0 source rebuilt withgo1.27.3, and the candidate — firstdevelopat1c6bb1972(which holds #1699), the pre-fix measurement that found the regressions below, then the release candidate itself,cc1d477ba(developab88ea950after the seven fixes + #1730), measured with the full method in "The release candidate" with the pre-fix column kept so the recovery is visible row by row. Every statement says which columns it rests on; source regressions are column 2 against 3 (and 2 against the release candidate).What it finds, in short: the candidate is faster on nearly every row a user notices (loads −8…−13 %,
WriteAPIJSON−10…−37 %, action chains −35…−59 %,LookupQualified150 → 43 ns), and allocates more on a handful of paths, each bisected to its merge — #1462 (LoadModel/0), #1478 (every load row, +5 % bytes), #1514/#1461 (AssignmentLoopStep), #1619 (DerivedReadWriteRead), #1508/#1512 (the binding walks), #1626 (FeaturesOf,Instantiate,BatchSatisfy) — and the +31 MiB resident set on the largest generated model is #1441 holding a submission's parse trees through its analysis. The fix for each is a separateperf/<slug>PR; the record's Fixes section is filled in as they open, and its Re-runs section with the quiet interleaved re-run of the rows the loaded machine left in doubt.The fixes (#1741–#1747) are listed as in the candidate with their merge SHAs; the export rows were re-probed on
5e0322e0cwhen #1760 landed oninternal/translate/export; "After the measured heads" accounts for every merge up todevelopb4d0a35c4— bygit diff --stat … -- '*.go' go.mod go.sumfor the client/pom/workflow merges and by #1771's own six-round benchstat for the one runtime refactor. The record also says what it does not decide: the afternoon's quiet re-run ran under 3–8 % hypervisor steal, so the first pass's time verdicts stand, and one open finding (a-run-queryrun leaves the next-validateof the same binary with a cold library record cache) is named and not attributed.How it was verified
python3 scripts/check-doc-links.pyreports one broken link,docs/project/third-party-notices.md→../assets/landing/libavoid-js.LICENSE.txt, which is pre-existing ondevelop(same result on a clean checkout of54e590985) and untouched here; the new record's links all resolve. The raw per-round benchmark outputs (six rounds × three configurations, the release-candidate rounds and re-probes, whole-binary runs, bisect logs) are kept outside the repository and attached to the gate report, as the 0.11.0 record's were.Checklist
make testandmake lintpass locally — documentation only; no Go source changeschanges/unreleased/<slug>.<section>.md, not as an edit toCHANGELOG.mdmake docs-countsrun if a gate count moved — no gate count movedF4,K5) in the body, docs, or changelogLink to Devin session: https://nasa-jpl-demo.devinenterprise.com/sessions/15408827f8104c7fbbdbd1b2c852ed2c
Open in Devin Desktop: https://nasa-jpl-demo.devinenterprise.com/desktop/session/15408827f8104c7fbbdbd1b2c852ed2c?variant=devin
Requested by: @HuiJun