Skip to content

Stop commit file bursts from stalling the API, and share opened history archives - #83

Merged
Mistium merged 2 commits into
mainfrom
api-commit-file-speed
Nov 21, 2026
Merged

Mistium merged 2 commits into
mainfrom
api-commit-file-speed

Conversation

@Mistium

@Mistium Mistium commented Nov 21, 2026

Copy link
Copy Markdown
Member

Why the API stalled

On 2026-11-21 around 06:03 the server received 217 parallel GET /v1/projects/:id/commits/:sha/file requests for one commit. That is the editor's history restore, which fetches every file of a commit with one Promise.all; the commit page's sprite text loader does the same. Each request took 30 to 57 s, and everything else queued behind them: thumbnails took 10 s, notifications 26 s, events 17 s. Once the burst ended, every route went back to answering in under 5 ms.

Every one of those requests:

  1. reopened the project's workspace archive,
  2. walked up to 10,000 commits to check that the SHA is reachable,
  3. decompressed and SHA-1 checked every blob in the tree to find the one path it wanted.

So one restore repeated the full-tree work 217 times on a 4-core machine.

What changed

  • Shared archives (native/gitinspection/archive_cache.go): Inspect keeps up to 6 workspace archives open across requests. Each one remembers the commits, trees, blob sizes, tree listings and reachable sets it has already decoded. An entry is dropped when its file changes (inode, size or mtime), when the file goes missing, or after 5 minutes idle; a janitor runs eviction every minute so deleted archives free their disk space. Concurrent first requests share one open.
  • File lookup by path: the file operation reads only the trees along the requested path and then the one blob.
  • Bounded inspection: at most half the cores (minimum 2) run history inspection at once, so a burst can't take the CPU from the rest of the API.
  • No parse and re-encode: inspection results are sent as the JSON the native code produced. A new InspectionError helper reads only the status fields.
  • Result cache fixed: its file names were about 257 bytes, over the 255-byte filename limit, so the result cache never wrote anything. Names are now hashed and kept in data/tmp/git-inspection-results/, so results can't evict workspace copies. Only results up to 1 MB are cached, failed writes are cleaned up, and file results aren't cached because they are cheap now.
  • Disk: the workspace cache cap drops from 2 GiB to 1 GiB. The production disk has 1.7 GB free.
  • last_commits.go reuses the shared tree parser, so it gets the tree memo too.

Measurements

Synthetic project shaped like the real one (180 text files, 40 binary assets of 64 to 576 KB, 300 commits), on a Ryzen 5 3600:

Case origin/main This branch
220 parallel file requests, cold 2.3 to 4.6 s, 16.4 GB allocated 41 to 62 ms, 150 MB
Repeated tree request 47 ms, 75 MB 0.34 ms, 0.38 MB

Commit summaries for a history list also share one open archive now, where before every SHA reopened it.

Checks

  • go test -race in native/gitinspection and native/storagefs. New tests cover: nested paths, directories and missing paths, a missing blob, a replaced archive, a removed archive, 64 concurrent requests agreeing, the archive limit, reachability with and without the memo, InspectionError, and shared-archive output being byte-identical to a plain archive for every operation across 12 commits.
  • osl lsp check (no new warnings: 644, same as main), osl test (63 passed), osl compile, scripts/check_conventions.py, all with OSL aeb83e39 as pinned by CI.
  • scripts/e2e_smoke.py gains tree, file, inline commit, last-commits and a parallel burst of 192 file requests. Comparing an origin/main binary with this branch gives identical responses over 351 steps.
  • An independent review found nothing must-fix. Its should-fix items (temp file cleanup, result size bound, freeing deleted archives) are fixed in the second commit.

No visible surface: this is server-only, so there are no screenshots.

Version: 1.4.0 -> 1.4.1 (patch: a fix and performance work).

…istory archives

A burst of per-file requests for one commit (history restore, sprite texts)
stalled the API: every request reopened the archive, walked the history and
decompressed every blob in the tree to find one path. Requests now share an
open archive with decoded commits, trees, blob sizes and reachability, look a
file up along its path, and run at most half the cores at a time.

The result cache never wrote anything because its file names were over 255
bytes; names are now hashed and kept in their own directory so they cannot
evict workspace copies. The workspace cache is capped at 1 GiB.
@Mistium
Mistium merged commit ecea1c7 into main Nov 21, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant