From e0efe9dcb7e85d9a0aaa15883e23feba300c4c6d Mon Sep 17 00:00:00 2001 From: geekypunk Date: Tue, 18 Aug 2026 19:54:56 -0500 Subject: [PATCH 1/2] fix(agent): retry upstream clones so a GitHub 429 stops failing the image build `docker compose build` failed on 2 of the last 15 CI runs, always in the deepsql-agent image, always the same way: Cloning into 'runtime'... remote: This request was rate-limited due to too many requests. fatal: unable to access 'https://github.com/NousResearch/hermes-agent.git/': The requested URL returned error: 429 exit code: 128 Both clones in agent/Dockerfile are unauthenticated requests to github.com, issued once with no retry. GitHub rate-limits unauthenticated traffic per source IP, and Actions runners share pooled egress addresses, so under load the clone is refused and the whole image build dies with it. Nothing about the pull request under test is involved, which is why the failure looked random and landed on unrelated branches. Both clones now retry up to five times with escalating backoff (15/30/45/60s). The condition is genuinely transient, so retry is the correct layer rather than a mask: on the final attempt the loop still prints FATAL and exits 1, so an upstream that is actually gone fails the build loudly instead of producing an image with no runtime in it. This is not CI-only defensiveness. `scripts/self-host/install.sh` runs the same build, so a self-host user behind a shared NAT hits the identical refusal. Note the webui clone's existing `||` fallback only ever handled a *moved ref*; against a rate-limit it re-issued the same refused request, so it needed the outer retry to be of any use. Verified: both RUN blocks pass `sh -n` as the shell receives them; the wrapper clones hermes-agent for real (exit 0), and against an unreachable repo it retries 5x and exits 1. Co-Authored-By: Claude Opus 5 (1M context) --- agent/Dockerfile | 34 ++++++++++++++++++++++++++++++---- 1 file changed, 30 insertions(+), 4 deletions(-) diff --git a/agent/Dockerfile b/agent/Dockerfile index c86ba45..55db137 100644 --- a/agent/Dockerfile +++ b/agent/Dockerfile @@ -44,8 +44,24 @@ ARG AGENT_RUNTIME_REF=main ARG AGENT_API_REPO=https://github.com/nesquena/hermes-webui.git ARG AGENT_API_REF=master -# Runtime engine (Python agent) -RUN git clone --depth 1 --branch "${AGENT_RUNTIME_REF}" "${AGENT_RUNTIME_REPO}" runtime \ +# Runtime engine (Python agent). +# +# The clone retries with backoff. github.com rate-limits *unauthenticated* +# requests per source IP, and CI runners share pooled egress addresses, so a +# cold clone here intermittently returns HTTP 429 and took the whole image +# build down with it (`fatal: unable to access ...: error: 429`, exit 128 -- +# 2 of 15 `docker compose build` runs). This is not CI-only defensiveness: a +# self-host user behind a shared NAT hits the identical refusal. Retry is the +# right layer precisely because the condition is transient; the loop still +# exits non-zero and loudly on the last attempt rather than continuing with no +# runtime cloned. +RUN n=0; rm -rf runtime; \ + until git clone --depth 1 --branch "${AGENT_RUNTIME_REF}" "${AGENT_RUNTIME_REPO}" runtime; do \ + n=$((n+1)); \ + if [ "$n" -ge 5 ]; then echo "FATAL: clone of ${AGENT_RUNTIME_REPO} failed after 5 attempts" >&2; exit 1; fi; \ + echo "clone attempt $n failed (HTTP 429 / transient network); retrying in $((n*15))s" >&2; \ + rm -rf runtime; sleep $((n*15)); \ + done \ && cd runtime \ && UV_NO_CONFIG=1 uv sync \ && if [ -d .venv ] && [ ! -d venv ]; then ln -sfn .venv venv; fi \ @@ -56,8 +72,18 @@ RUN git clone --depth 1 --branch "${AGENT_RUNTIME_REF}" "${AGENT_RUNTIME_REPO}" # HTTP API surface the frontend / backend talk to (:8787). # Default branch is master (not main). Fall back to HEAD if the ref moves. -RUN (git clone --depth 1 --branch "${AGENT_API_REF}" "${AGENT_API_REPO}" api \ - || git clone --depth 1 "${AGENT_API_REPO}" api) \ +# +# Same 429 retry as the runtime clone above. Note the inner `||` fallback +# handles only a *moved ref*; on a rate-limit it just re-issues the same +# refused request, so it needed the outer retry to be any use at all. +RUN n=0; rm -rf api; \ + until (git clone --depth 1 --branch "${AGENT_API_REF}" "${AGENT_API_REPO}" api \ + || git clone --depth 1 "${AGENT_API_REPO}" api); do \ + n=$((n+1)); \ + if [ "$n" -ge 5 ]; then echo "FATAL: clone of ${AGENT_API_REPO} failed after 5 attempts" >&2; exit 1; fi; \ + echo "clone attempt $n failed (HTTP 429 / transient network); retrying in $((n*15))s" >&2; \ + rm -rf api; sleep $((n*15)); \ + done \ && cd api \ && if [ -f requirements.txt ]; then \ UV_NO_CONFIG=1 uv pip install --python /opt/deepsql-agent/runtime/venv/bin/python -r requirements.txt; \ From 704a3712ff73147658665e2e157fd4ffbe3f8663 Mon Sep 17 00:00:00 2001 From: geekypunk Date: Tue, 18 Aug 2026 20:20:05 -0500 Subject: [PATCH 2/2] docs: record the unauthenticated-clone 429 as an agent runtime rule MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The fix in this branch is only half the value; the diagnosis is the other half. An intermittent CI failure landing on branches whose diff has nothing to do with the agent reads as flaky infrastructure, and the instinct is to re-run or bisect. It was neither — it was a per-IP rate limit on an unauthenticated clone, visible in one line of the build log. Filed under Agent Runtime Rules rather than Verification Anti-Patterns: it is a property of how the agent image is assembled, next to the MCP SDK pin that guards the same build against the same class of upstream surprise. Co-Authored-By: Claude Opus 5 (1M context) --- CLAUDE.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/CLAUDE.md b/CLAUDE.md index 2cf64a0..4a48aa8 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -235,6 +235,21 @@ returns a number). `hermes_requires <0.20.0` on a "verified" 401 that came from a hand-rolled `hermes serve` run rather than `hermes webui`. 0.20.0 works. Verify against the real start path before writing a version constraint. +5. **The agent image build clones two third-party repos over the public internet, + unauthenticated.** `agent/Dockerfile` fetches `NousResearch/hermes-agent` and + `nesquena/hermes-webui` at build time. GitHub rate-limits unauthenticated + requests *per source IP*, and Actions runners share pooled egress addresses, so + `docker compose build` intermittently died on `fatal: unable to access ...: The + requested URL returned error: 429` (exit 128) — 2 of 15 runs, always on branches + whose diff had nothing to do with the agent. Both clones now retry 5x with + backoff, and still print FATAL and exit 1 on exhaustion so a genuinely dead + upstream cannot yield an image with no runtime in it. Two lessons worth keeping: + a CI failure that is *intermittent and unrelated to the diff* is a network or + rate-limit signature, not a code defect — read the log before bisecting the + branch; and the webui clone's pre-existing `|| git clone` fallback looked like + resilience but only ever handled a *moved ref*, re-issuing the identical refused + request against a 429. A fallback that fails the same way as the thing it backs + up is not a fallback. ### Verification Anti-Patterns (do not repeat)