Skip to content

perf: separate-process benchmark and leaner receipt sealing - #53

Merged
shreyanshjain7174 merged 7 commits into
mainfrom
perf/tail-latency
Oct 18, 2026
Merged

shreyanshjain7174 merged 7 commits into
mainfrom
perf/tail-latency

Conversation

@shreyanshjain7174

Copy link
Copy Markdown
Contributor

Plan

Follow-up to #50: investigate the gateway's p99 overhead with a better method, and fix only what is safe. Receipt durability and all security checks are unchanged.

Gate status: still not met at any fixed-rate load on this host. This PR improves the method, removes one avoidable cost, and records where the tail goes.

What changed

  • Separate-process benchmark (make bench-proc). The gateway, mock upstream and open-loop load generator run as 3 processes, so the generator no longer competes with the gateway's 4 Ps.
    • Latency is timed from the scheduled send time (no coordinated omission).
    • Upstream time is the mock's own handler time, subtracted per request.
    • Failed requests are excluded from percentiles, and a run with any failure is marked invalid (p99: null).
  • d15cf71, allocation fix on the receipt path. It drops a redundant 3 KB preflight buffer and right-sizes the seal preimage. That saves about 6.2 KB per request, a 24.2% allocation reduction. Receipt bytes, hashes and signatures are unchanged; golden vectors and chain tests pass.
  • 8632eb0, trace regions in ledger.go. These are production code, but they do nothing when tracing is off.
  • bench/RESULTS.md. Separate-process numbers, a tail breakdown, and a verdict per load level.

What the data says

  • Receipt-heavy tail requests: queueing behind group commit is about 55% of the time, and SQLite COMMIT is about 13%. Other stalls are transport-side, including one 1.7s outlier.
  • The p99 movement across runs is host noise, not an effect of d15cf71 (50.7, then 2.73, then 8.19 ms at 5K/s). Two "before" runs of the same case already differed 2.5x.
  • Next steps:
    • an isolated Linux 4-vCPU host;
    • then decide on group-commit tuning or COMMIT cost, without weakening sync-before-response.

Validation

  • go build ./...: passed
  • go vet ./...: passed
  • go test -race ./... -count=2: passed
  • semgrep --config auto --error --quiet ./internal ./cmd ./pkg: passed

Review

Reviewed by a second model (Claude Opus 5.5). Approved. It confirmed:

  • the production changes keep identical receipts;
  • the allocation saving is real (predicted 6,208 B, measured 6,273 B);
  • there are no pooled buffers or races;
  • the method is sound.

Its 3 minor notes (caveats in RESULTS.md, attribution of the trace commit, failed samples) are fixed here.

shreyanshjain7174 and others added 7 commits October 18, 2026 06:52
Measure scheduled arrivals outside the gateway CPU budget. Add opt-in receipt trace regions without changing commit acknowledgment.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Shreyansh Sancheti <43677304+shreyanshjain7174@users.noreply.github.com>
Keep transport queueing in scheduled latency. Limit client connections to avoid loopback accept bursts. Preserve failed-case evidence and correlate request tails with receipt trace phases.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Shreyansh Sancheti <43677304+shreyanshjain7174@users.noreply.github.com>
Capture GC counters outside the request timer. Stop tracing before writing profiles and skip stack lines during trace analysis.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Shreyansh Sancheti <43677304+shreyanshjain7174@users.noreply.github.com>
Validate preflight drafts without building an unused preimage. Size the signed preimage from validated fields instead of maximum field sizes. Keep canonical bytes and synchronous commit acknowledgment unchanged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Shreyansh Sancheti <43677304+shreyanshjain7174@users.noreply.github.com>
Correlate receipt phases with the same tail requests. Record client acquisition separately and profile upstream reuse. Add macOS foreground-policy controls and analyzer regression tests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Shreyansh Sancheti <43677304+shreyanshjain7174@users.noreply.github.com>
Keep the failed gate explicit, including the 1.7-second transport tail. Record correlated receipt, GC, lock, and connection evidence. Qualify shared-host variance and pending Linux validation.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Shreyansh Sancheti <43677304+shreyanshjain7174@users.noreply.github.com>
Exclude failed samples from latency statistics and suppress p99 on invalid rows. Keep successful-only diagnostic percentiles and test partial and total failures. Attribute p99 movement to host noise and clarify production trace instrumentation.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Shreyansh Sancheti <43677304+shreyanshjain7174@users.noreply.github.com>
@shreyanshjain7174
shreyanshjain7174 merged commit b131f16 into main Oct 18, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant