Skip to content

fix: keep the last good GitHub data when the rate limit runs out - #106

Merged
tschm merged 1 commit into
mainfrom
fix/keep-board-on-rate-limit
Nov 14, 2026
Merged

tschm merged 1 commit into
mainfrom
fix/keep-board-on-rate-limit

Conversation

@tschm

@tschm tschm commented Nov 13, 2026

Copy link
Copy Markdown
Member

Problem

When the GitHub quota ran out, the board showed No data on every GitHub panel until the hourly window reset.

GitHub answers an exhausted quota with a 403, and GitHub._json treated every 403 as "this repo is not readable": it logged the call and skipped the repo. Once the quota was gone that happened to every repo, but the pass still counted as a success. Store.update(remote=...) then replaced the last good snapshot with an empty one.

This showed up on a live board. A hand-started container polling every 300s spent about 600 calls per pass, roughly 7,200 an hour against a 5,000 limit, so the board went blank about 40 minutes into every hour.

Fix

  • github_api._rate_limited tells a rate limit from a real permission 403. It checks for x-ratelimit-remaining: 0 (the primary limit) or retry-after (the secondary limit) on a 403 or 429. _get raises the new forge.RateLimited in that case. A 403 with quota left, such as Dependabot alerts being switched off, is still answered as None.
  • fan_out re-raises RateLimited and cancels the queued repos instead of dropping repos one at a time. Without that, a quota that runs out partway through a pass would publish a fleet cut off at an arbitrary repo.
  • github.collect closes its client when it raises, because the caller never gets the client back to close it.
  • _refresh_remote keeps the previous pass's GitHub repos and their exclusions, sets rate_limit_remaining to 0 along with the reset time, and re-raises so jq_collector_errors_total{source="github"} goes up and the data is visibly stale.

Scope: this covers GitHub's rate limit only. A general GitLab outage still drops the GitLab repos, as test_a_gitlab_outage_leaves_the_github_repos_standing expects. During a GitHub rate limit the GitLab pass gets no template tags for that pass, so GitLab drift reads unknown until the next good pass.

Tests

  • HTTP layer: a 403 or 429 with either rate-limit signal raises from both _json and _bytes, the reset time travels with the exception, and a 403 with quota left still returns None.
  • fan_out: RateLimited propagates, and an ordinary per-repo error is still logged and the repo left out.
  • _refresh_remote: GitHub repos and exclusions carry over (GitLab's don't), and the failure and rate-limit figures are recorded.
  • collect closes its client when the limit is hit.

Local results: pytest --cov passes 489 tests at 100% coverage, ruff check and format, mypy, and prek run --all-files (including interrogate) all pass.

🤖 Generated with Claude Code

A 403 from an exhausted quota was handled like a 403 on one repo: logged,
the repo skipped. Once the quota ran out every repo was skipped, the pass
still counted as a success, and its empty snapshot replaced the last good
one - so every GitHub panel read No data until the window reset.

The client now tells the two apart (x-ratelimit-remaining: 0, or a
retry-after for the secondary limit) and raises RateLimited. fan_out lets
it through rather than dropping repos one by one, collect() closes its
client on the way out, and _refresh_remote carries the previous pass's
GitHub repos over while recording the failure against github, so stale
data is visible as stale rather than shown as nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tschm
tschm merged commit d5e303b into main Nov 14, 2026
6 checks passed
@tschm
tschm deleted the fix/keep-board-on-rate-limit branch November 14, 2026 05:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant