fix(agent): retry upstream clones so a GitHub 429 stops failing the image build - #67
Merged
Merged
Conversation
…mage build
`docker compose build` failed on 2 of the last 15 CI runs, always in the
deepsql-agent image, always the same way:
Cloning into 'runtime'...
remote: This request was rate-limited due to too many requests.
fatal: unable to access 'https://github.com/NousResearch/hermes-agent.git/':
The requested URL returned error: 429
exit code: 128
Both clones in agent/Dockerfile are unauthenticated requests to github.com,
issued once with no retry. GitHub rate-limits unauthenticated traffic per
source IP, and Actions runners share pooled egress addresses, so under load
the clone is refused and the whole image build dies with it. Nothing about the
pull request under test is involved, which is why the failure looked random
and landed on unrelated branches.
Both clones now retry up to five times with escalating backoff (15/30/45/60s).
The condition is genuinely transient, so retry is the correct layer rather
than a mask: on the final attempt the loop still prints FATAL and exits 1, so
an upstream that is actually gone fails the build loudly instead of producing
an image with no runtime in it.
This is not CI-only defensiveness. `scripts/self-host/install.sh` runs the
same build, so a self-host user behind a shared NAT hits the identical refusal.
Note the webui clone's existing `||` fallback only ever handled a *moved ref*;
against a rate-limit it re-issued the same refused request, so it needed the
outer retry to be of any use.
Verified: both RUN blocks pass `sh -n` as the shell receives them; the wrapper
clones hermes-agent for real (exit 0), and against an unreachable repo it
retries 5x and exits 1.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The fix in this branch is only half the value; the diagnosis is the other half. An intermittent CI failure landing on branches whose diff has nothing to do with the agent reads as flaky infrastructure, and the instinct is to re-run or bisect. It was neither — it was a per-IP rate limit on an unauthenticated clone, visible in one line of the build log. Filed under Agent Runtime Rules rather than Verification Anti-Patterns: it is a property of how the agent image is assembled, next to the MCP SDK pin that guards the same build against the same class of upstream surprise. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
4 tasks
venkateshsakamuri-lab
pushed a commit
that referenced
this pull request
Aug 19, 2026
Follow-up to #67, which fixed the *transient* half of the agent build's fragility. This fixes the *silent* half. ## Problem Both install paths tracked **moving branches**: | path | before | behaviour | |---|---|---| | `agent/Dockerfile` | `AGENT_RUNTIME_REF=main`, `AGENT_API_REF=master` | whatever the branch points at today | | `scripts/self-host/setup-agent.sh` | `git clone --depth 1 "$repo"` — **no ref at all** | default-branch HEAD | An upstream commit can therefore change what the agent image contains, and break it, **with no change on our side**. That is precisely the failure mode the `mcp>=1.0,<2` pin four lines below in the same file exists to prevent — left wide open one layer up. `CLAUDE.md` already states the rule (*"PINNED for reproducibility ... Bump deliberately and re-validate"*); this repo just wasn't following it. ## Change Pinned to the current release pair: ``` hermes-agent v2026.8.18 -> e624e9f hermes-webui v0.52.76 -> 3c9304a ``` **This is a behaviour change, not a no-op.** Both branches had already moved past their latest tag — `main` was at `1f234a1`, `master` at `63a562f` — so this rolls the runtime back from branch HEAD to the tagged release. Reviewers should weigh that deliberately rather than read "pinning" as cosmetic. **Both files change together on purpose.** Pinning only the Dockerfile would leave host installs on branch HEAD and container installs on a tag, so a bug would reproduce on one path and not the other. The Dockerfile comment already asserted the two tracked each other — that assertion was false, and this makes it true. `ensure_clone` still returns early when a checkout already exists, so an install predating this keeps its current ref. Silently deleting a user's agent directory to change a version isn't this script's call — but the message now says the ref was left alone instead of implying it was applied. ## Verification - [x] Both tags clone at the **exact expected SHAs**, with `pyproject.toml` and `requirements.txt` present - [x] `setup-agent.sh` passes `bash -n` - [ ] CI `docker compose build` green — proves the pinned refs *build* - [ ] **Agent smoke test — not covered by CI, and required before merge** That last box is the important one. Per `CLAUDE.md`'s own anti-patterns, *"a dashboard that is HTML and long proves nothing"* and presence ≠ compatibility: a green `docker compose build` proves these refs compile and install, **not** that the agent answers. Since this moves the runtime to a different commit than what's been running, someone should confirm the Agent tab returns a real answer (not "I'm blocked") against a pinned image before this lands. I can't run that from here. ## Ordering note Independent of #67 — different lines of `agent/Dockerfile`, no conflict expected — but #67 should land first so the retry protects these clones too. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
venkateshsakamuri-lab
approved these changes
Aug 19, 2026
Contributor
|
@geekypunk update the branch from main and merge it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Symptom
docker compose buildhas been failing intermittently on unrelated PRs — most recently twice today oncursor/admin-profile-switch-c497. Every other job passes; only the agent image fails.Root cause
Not a code defect. Both upstream clones in
agent/Dockerfileare unauthenticated requests to github.com, issued once, with no retry:GitHub rate-limits unauthenticated traffic per source IP, and Actions runners share pooled egress addresses. Under load the clone is refused and the whole image build dies with it. Nothing about the PR under test is involved — which is exactly why it looked random and kept landing on innocent branches.
Failure rate: 2 of the last 15
docker compose buildruns. That flakiness pattern is itself the diagnosis; a real build break would fail 15 of 15.Fix
Both clones retry up to 5 times with escalating backoff (15/30/45/60s). Retry is the right layer because the condition is transient — but the loop still prints
FATALand exits 1 on the last attempt, so an upstream that is genuinely gone fails the build loudly rather than yielding an image with no runtime in it.This is not CI-only defensiveness:
scripts/self-host/install.shruns the same build, so a self-host user behind a shared NAT hits the identical refusal.Also noted in a comment: the webui clone's existing
|| git clonefallback only ever handled a moved ref. Against a rate-limit it just re-issued the same refused request, so it needed the outer retry to be of any use at all.Verification
RUNblocks passsh -nas the shell actually receives them (continuations joined, in-RUNcomments stripped). Baseline check caught a bug in my own extractor first — all 6 blocks including 3 untouched ones flagged, which is how I knew the harness was wrong rather than the Dockerfile.git cloneof hermes-agent — exit 0, 76 entries,pyproject.tomlpresent.FATAL, exits 1 (not 0).docker compose buildgreen on this PR — the authoritative test.Follow-up, deliberately not bundled here
AGENT_RUNTIME_REF=mainandAGENT_API_REF=masterare unpinned branches, which contradicts the pinning rule inCLAUDE.md("PINNED for reproducibility ... Bump deliberately and re-validate"). That's a second, independent failure mode — an upstream commit can break this build with no change on our side, which is precisely the class of bug themcp>=1.0,<2pin two lines below exists to prevent. Current upstream tags arev2026.8.18(hermes-agent) andv0.52.76(hermes-webui). Worth a separate PR that pins and re-validates.🤖 Generated with Claude Code