Repository navigation
fix: keep the last good GitHub data when the rate limit runs out - #106
Merged
Merged
Conversation
A 403 from an exhausted quota was handled like a 403 on one repo: logged, the repo skipped. Once the quota ran out every repo was skipped, the pass still counted as a success, and its empty snapshot replaced the last good one - so every GitHub panel read No data until the window reset. The client now tells the two apart (x-ratelimit-remaining: 0, or a retry-after for the secondary limit) and raises RateLimited. fan_out lets it through rather than dropping repos one by one, collect() closes its client on the way out, and _refresh_remote carries the previous pass's GitHub repos over while recording the failure against github, so stale data is visible as stale rather than shown as nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When the GitHub quota ran out, the board showed No data on every GitHub panel until the hourly window reset.
GitHub answers an exhausted quota with a 403, and
GitHub._jsontreated every 403 as "this repo is not readable": it logged the call and skipped the repo. Once the quota was gone that happened to every repo, but the pass still counted as a success.Store.update(remote=...)then replaced the last good snapshot with an empty one.This showed up on a live board. A hand-started container polling every 300s spent about 600 calls per pass, roughly 7,200 an hour against a 5,000 limit, so the board went blank about 40 minutes into every hour.
Fix
github_api._rate_limitedtells a rate limit from a real permission 403. It checks forx-ratelimit-remaining: 0(the primary limit) orretry-after(the secondary limit) on a 403 or 429._getraises the newforge.RateLimitedin that case. A 403 with quota left, such as Dependabot alerts being switched off, is still answered asNone.fan_outre-raisesRateLimitedand cancels the queued repos instead of dropping repos one at a time. Without that, a quota that runs out partway through a pass would publish a fleet cut off at an arbitrary repo.github.collectcloses its client when it raises, because the caller never gets the client back to close it._refresh_remotekeeps the previous pass's GitHub repos and their exclusions, setsrate_limit_remainingto 0 along with the reset time, and re-raises sojq_collector_errors_total{source="github"}goes up and the data is visibly stale.Scope: this covers GitHub's rate limit only. A general GitLab outage still drops the GitLab repos, as
test_a_gitlab_outage_leaves_the_github_repos_standingexpects. During a GitHub rate limit the GitLab pass gets no template tags for that pass, so GitLab drift reads unknown until the next good pass.Tests
_jsonand_bytes, the reset time travels with the exception, and a 403 with quota left still returnsNone.fan_out:RateLimitedpropagates, and an ordinary per-repo error is still logged and the repo left out._refresh_remote: GitHub repos and exclusions carry over (GitLab's don't), and the failure and rate-limit figures are recorded.collectcloses its client when the limit is hit.Local results:
pytest --covpasses 489 tests at 100% coverage, ruff check and format, mypy, andprek run --all-files(including interrogate) all pass.🤖 Generated with Claude Code