Repository navigation
Stop commit file bursts from stalling the API, and share opened history archives - #83
Merged
Merged
Conversation
…istory archives A burst of per-file requests for one commit (history restore, sprite texts) stalled the API: every request reopened the archive, walked the history and decompressed every blob in the tree to find one path. Requests now share an open archive with decoded commits, trees, blob sizes and reachability, look a file up along its path, and run at most half the cores at a time. The result cache never wrote anything because its file names were over 255 bytes; names are now hashed and kept in their own directory so they cannot evict workspace copies. The workspace cache is capped at 1 GiB.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why the API stalled
On 2026-11-21 around 06:03 the server received 217 parallel
GET /v1/projects/:id/commits/:sha/filerequests for one commit. That is the editor's history restore, which fetches every file of a commit with onePromise.all; the commit page's sprite text loader does the same. Each request took 30 to 57 s, and everything else queued behind them: thumbnails took 10 s, notifications 26 s, events 17 s. Once the burst ended, every route went back to answering in under 5 ms.Every one of those requests:
So one restore repeated the full-tree work 217 times on a 4-core machine.
What changed
native/gitinspection/archive_cache.go):Inspectkeeps up to 6 workspace archives open across requests. Each one remembers the commits, trees, blob sizes, tree listings and reachable sets it has already decoded. An entry is dropped when its file changes (inode, size or mtime), when the file goes missing, or after 5 minutes idle; a janitor runs eviction every minute so deleted archives free their disk space. Concurrent first requests share one open.fileoperation reads only the trees along the requested path and then the one blob.InspectionErrorhelper reads only the status fields.data/tmp/git-inspection-results/, so results can't evict workspace copies. Only results up to 1 MB are cached, failed writes are cleaned up, andfileresults aren't cached because they are cheap now.last_commits.goreuses the shared tree parser, so it gets the tree memo too.Measurements
Synthetic project shaped like the real one (180 text files, 40 binary assets of 64 to 576 KB, 300 commits), on a Ryzen 5 3600:
Commit summaries for a history list also share one open archive now, where before every SHA reopened it.
Checks
go test -racein native/gitinspection and native/storagefs. New tests cover: nested paths, directories and missing paths, a missing blob, a replaced archive, a removed archive, 64 concurrent requests agreeing, the archive limit, reachability with and without the memo,InspectionError, and shared-archive output being byte-identical to a plain archive for every operation across 12 commits.osl lsp check(no new warnings: 644, same as main),osl test(63 passed),osl compile,scripts/check_conventions.py, all with OSL aeb83e39 as pinned by CI.scripts/e2e_smoke.pygains tree, file, inline commit, last-commits and a parallel burst of 192 file requests. Comparing an origin/main binary with this branch gives identical responses over 351 steps.No visible surface: this is server-only, so there are no screenshots.
Version: 1.4.0 -> 1.4.1 (patch: a fix and performance work).