Repository navigation
feat(harness): read an agent without building it - #245
Merged
khushalsonawat merged 28 commits intoNov 10, 2026
Merged
khushalsonawat merged 28 commits into
khushalsonawat merged 28 commits into
Conversation
khushalsonawat
force-pushed
the
feat/TH-8804-source-subfolder
branch
from
November 6, 2026 04:49
3e4e3ec to
9c187fe
Compare
- Add --read-agent-only to the hosted authoring command: run only the agent reading and leave contract.json, agent.json, cost.json and any failure.json in --output, then stop. - Share one helper, reading_entry, with the agent-finding mode: job check, a soft limit and a hard timer, masking of secrets files, and a cost file that starts incomplete and is rewritten after each turn. - Count Claude's running totals once per turn, keep a contract the gate saved over any later fault, and refuse browser agents by the same rule a build uses. - Treat a turn limit as final, never retried: an error outcome, or ADK's whole call-limit message. A status-less provider failure is retried and marks the cost incomplete. TH-8951 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
khushalsonawat
force-pushed
the
feat/TH-8951-read-without-build
branch
from
November 6, 2026 04:51
5b1a975 to
519f818
Compare
cdileep23
requested changes
Nov 6, 2026
Adds --find-agents to the authoring entrypoint, run through the shared reading helper. It writes /work/reading/agents.json in the shape the platform's AgentScanSerializer reads, plus cost.json on every return. - A plain code walk finds candidates (framework builds, langgraph.json graphs, agent.py, provider clients) and joins files into agents by their imports, without running anything. - Each agent's folder gets TH-8805's framework answer, with a build's limits, in one child process the scan's clock can stop. - A model is asked only when the walk finds nothing usable (mostly TypeScript and JavaScript repos). Its read/glob/grep tools stay inside the source, and only the paths it names, once checked, are kept. - Names, evidence and paths are built to fit the platform's rules, and the file is checked against a copy of them before it is written. - An agent at a path the platform's rule can't carry (a Next.js app/(chat)/ route, say) is left out with truncated set, and is reported as agent_path_unsupported when nothing else is found. TH-8950 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
5 of 10 tasks
Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
…at it copied - A job that fails validation names only where and how, never its values, so a secret in the job can't reach failure.json. - Swallowed errors and crashes log a warning with their trace, masked. - The reading's words, the refusal exit, the folder choice and the cost writer are shared with the build instead of copied. - A stage takes its own idle retries; the time limits are worked out when a reading starts; the timer stops with the work. - Classifying a reading's result moves to reading_result.py. TH-8951 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
- One helper, understand.reading_settings, chooses the stage's turns, thinking and setup-contract gate for both the build and the reading. - A generic job's reading now refuses what the build's gate refuses. TH-8951 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
…s logs - Tests pin the time-limit and crash codes, and that cost.json ends incomplete after a timeout or a crash the work outlived. - After the work returns, the limit still ends a process a leftover thread keeps alive, with the work's own code and no new writes. - A job error's location drops mapping keys; the mask also covers a secret's lines, repr and escaped forms; secrets are read per reading. - The reading's opening shares the build's guidance; names used across modules are public; a dead attribute is gone. TH-8951 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
TH-8950 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
#245 replaced reading_entry._secret_values with a functools.cache reader keyed by the secrets files. The scan tests already point those files at a fresh temporary path, so each test gets its own cache entry and there is nothing left to reset. TH-8950 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
cdileep23
requested changes
Nov 10, 2026
…ts crash - Any model error now keeps the walk's rows with truncated set and exits 0; only a scan with no rows writes scan_model_error, carrying the model's own retryable flag. A retry would pay for the model again. - A fault outside any one file in the walk (run or result) now reaches the scan's crash path (scan_crashed, retryable) instead of being reported as scan_limit_reached. - Catches are narrowed to what each step expects. A file's reader faults still leave that file unread. A folder check that fails in the child sends its trace back, and the row is marked unfinished. Faults that are still absorbed (after the paid step, closing the session, tool calls) log their trace through the reading helper's MaskSecrets filter instead of the class name only. TH-8950 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
- agent_scan imports EXIT_DONE, EXIT_RETRYABLE, limit_seconds and reading_failure from reading_entry, and EXIT_REFUSED from agent_type_check. FIND_AGENTS now sets limit_env and limit_margin_seconds, so the helper and the scan read ALK_SCAN_MAX_SECONDS at the same moment. - The find-agents stage passes idle_retries=0 itself instead of setting session.STAGE_IDLE_RETRIES for the whole process. - The walk uses framework_detection's _Work, which takes an optional limit, instead of its own copy, and bundle_author_v2's ignore list instead of a copy held equal by a test. - The walk and an AI row's description share one folder loop (_source_files) and one shadowed-name lookup (_ShadowedNames). TH-8950 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
- An agent's subfolder counts only the agents of its own project. Adding an unrelated project with its own agent deeper in the repo no longer moves the first agent's folder, which the platform keys on. - The walk reads files outside test, example and other side folders first, then the side folders. A large tests folder sorted ahead of the agent can no longer use up the 20,000-file cap before the agent is read. TH-8950 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
… tests plainly - The shared-folder evidence sentence no longer asks a build to name an entry, which a build job has no field for. - Test names say "the platform format", "the platform rules" and "the path rule" instead of "his". - _result's docstring describes what it does instead of pointing at plan step numbers. TH-8950 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
- At the hard limit a saved contract becomes agent.json and exit 0, instead of a timeout written over it; the close after a stop is bounded. - A cancel is a crash with its failure before the save, and keeps the saved contract after it. - The build and the reading share one agent-document helper, so a folder mismatch reports the build's code. - A failed document logs a masked trace; provider-import and connect-only jobs are refused; a soft-limit test uses a fake clock. TH-8951 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
The model step also runs when the walk found agents only in test or example folders, so the test is no longer named "only when nothing is found". TH-8950 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
The catch still keeps a paid answer from becoming a retry; it now logs the masked trace like the other absorbed faults, so a fault in the cost writer can be found. TH-8950 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
… check faults From the review round on the fixes for the first review of this PR: - One unreadable or unparsable file no longer stops the description of a model-found agent. It marks the row short (partial) and the reading goes on, as the walk goes on past an unread file. - detect_framework takes raise_faults (off for builds). The scan's folder check turns it on, so a fault inside the framework check comes back as a trace and the row is unfinished, instead of a finished unknown. Both the child's and the parent's halves are tested. - A model tool's fault keeps the worker's exception as the cause, so its logged trace shows the real error. - Tests pin the run-time read of ALK_SCAN_MAX_SECONDS and the secret mask on every scan logger. The log comment says only the platform's secret values are masked. TH-8950 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
…es with own code From the review round on the fixes for the first review of this PR: - The file cap is 200,000, up from 20,000. Measured: 100,000 tiny source files walk in 13 s at 266 MB, so the byte cap and the clock are what bound the walk. A repo whose own code passes 20,000 files no longer stops before its agent is read. - A folder's settings template (.env.example and its kin) is read in its folder's pass, not with the side files, so a big tests folder can't keep it from being read. - A reader fault of an unforeseen kind logs its trace once; the rest are counted (reader_faults in the walk's summary line). - ALK_SCAN_AI_MAX_SECONDS is read when the scan starts, as the clock is. - Tests: every file is read in exactly one pass; tests no longer depend on an exported ALK_SCAN_MAX_SECONDS; the process-wide idle-retry setting is set through monkeypatch. TH-8950 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
…ond document build - The hook that built agent.json from the timer's thread is gone: two document builds could starve each other past the limit. The close after a stop is bounded at 20 s, so the reading decides from its files well before the hard limit. - The contract is read once and checked from the same bytes; a mismatch names subfolder_mismatch; a validation error logs no trace; a cancel during the stop's interrupt stays a timeout. TH-8951 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
From the last review round on the fixes for the first review of this PR: - A file over 1 MiB in a model-found agent's folder now marks the row short (partial) instead of being skipped silently, so a paid row with a missing file is never reported complete. - The description logs one trace for a reader fault of an unforeseen kind, then counts the rest, as the walk does. - Comments say traces are logged with the platform's secret values masked, not that they are fully masked. TH-8950 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
… sandbox From the last review round on the fixes for the first review of this PR: - The file cap is 40,000, not 200,000. What the walk holds grows with the names each file imports, not its bytes: about 33 KB a file for files that each import 200 names, a 1,389 MB peak at 40,000 such files. At 200,000 such files the sandbox would kill the scan. 40,000 still covers a repo whose own code passes 20,000 files. - The partial-row sentence says files couldn't be read, not that they weren't read in time, since a file that fails to parse also sets it. - A test pins that ALK_SCAN_AI_MAX_SECONDS is read when the model step is asked, and the comment says so. TH-8950 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
The step that joins the walk's files into agents (the import search from each candidate, and the per-agent loops) now checks a deadline: the walk's own deadline plus 120 s, well inside the helper's 600 s margin before the hard stop. Past it the walk ends as stopped by the clock, so the scan writes its own file (scan_timed_out, exit 0) instead of being ended by the hard stop. The two-pass test now pins the order it claims: a settings template in a folder that sorts after tests/ is read before tests/test_a.py. TH-8950 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
…p a password out of a failed redaction - The hard limit is soft + 300 s, so the document after the longest stop has room on a slow CPU; a test pins the stop's bounds inside it. - The interrupt runs as its own task: one that swallows its cancel can't hold the stop, and Ctrl-C passes through. - A URL the parser refuses is redacted whole instead of raising with its credentials in the message. - The folder choice uses the contract text already checked. TH-8951 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
#224's rule now gives every browser reading one second look first. TH-8951 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
TH-8950 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
Candidates that reach each other hold one entry per pair, which grows with the square of the candidates. The join now counts them as it builds them and, past 5,000,000 (about 90 MB), ends as the walk's other caps do (stopped="limit", scan_limit_reached), instead of growing until the clock stops it. TH-8950 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
feat(harness): list the agents in a repo without building it
cdileep23
approved these changes
Nov 10, 2026
- A Ctrl-C after the contract is saved now passes through; a cancel leaking from the session still keeps the saved contract. - A BaseExceptionGroup (or any non-interrupt, non-exit base exception) from the work ends as the crash failure with exit 1. - _redact logs only the error class when it redacts a URL whole. - The stop-budget test pins exactly 90 s. - Drop a person's name from a scan_walk.py comment. Refs: TH-8951 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
7 of 10 tasks
khushalsonawat
merged commit Nov 10, 2026
22bcbb8
into
feat/TH-8804-source-subfolder
11 checks passed
khushalsonawat
added a commit
that referenced
this pull request
Nov 10, 2026
This was referenced Nov 10, 2026
khushalsonawat
added a commit
that referenced
this pull request
Nov 11, 2026
[TH-9454] fix: follow-ups from #245's last review of the agent reading
5 of 10 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #225 (TH-8804), which is stacked on #224 (TH-8806). This PR's base is
feat/TH-8804-source-subfolder, so its diff shows only TH-8951. It retargets down the stack as those merge.Summary
The platform's Agents tab creates an agent and its first version from a repo before any environment is built, and it needs the agent's description to show. This PR adds a
--read-agent-onlymode to the hosted authoring command. It runs only the agent reading a build already does, leaves the result in--output(/work/reading/on the hosted route), and stops. Nothing is built, and there is no new callback.A run leaves:
contract.json: the raw reading;agent.json: the agent document;cost.json: the build's spend keys pluscomplete, whenever the reading reaches its work. A run refused before that leaves none, and none means nothing was spent. The refusals are: a bad job, a job check that crashes, a provider-defined agent, a wrong source kind, a non-empty output folder, or a failed first cost write;failure.json: on failure only, in theHarnessFailureshape.Exit 0 means read, 78 not retryable, 1 retryable. When
failure.jsonis present, itsretryabledecides.Ticket: TH-8951.
How it works
reading_entry.py, a helper shared with TH-8950's--find-agents(contractreading-entryv0.17). It handles:ALK_READING_MAX_DURATION_SECONDS) and a hard timer 300 s after it;cost.jsonthat startscomplete: falseand is rewritten after each turn.agent_reading.pyandreading_result.py, the reading. It opens its stage exactly ascli._understanddoes, through one shared helper (understand.reading_settings): 70 or 140 turns, thinking on, and the setup-contract gate for generic jobs. It uses the same model and gate as a build. Browser agents are refused by TH-8806's rule (agent_type_unsupported, 78).completeis false after the soft limit, after a stalled stream, after any unpriced turn, and after any status-less provider error, because a cut request may still be billed.max_turnsorerror_max_turnsoutcome, or ADK's whole "Max number of llm calls limit ofnexceeded" message. If the last turn hit the limit, that wins over an earlier status-less failure.Release notes
complete: false. The platform chargestotal_usdeither way and keeps the flag.Known limits
Stageor a backend.connect_onlyjob with anhttporautoconnector is refused, though the build loads no provider definition for it. The refusal fails closed, with a clear 78 before any spend.Verification
tests/harness/test_reading_entry.py: 61 passed.tests/harness/test_agent_reading.py: 105 passed.tests/harness/test_scenario_source_provision_agent.py: 24 passed (atd1f968cc). Run on a locked venv (claude-agent-sdk 0.2.156, google-adk 2.7.1, Python 3.12.14), each whole file in one process. Both are added tosdk-smoke.yml.complete: truecomplete: trueagent_type_unsupportedagent_reading_timed_out,complete: falseThe interrupted Claude turn reported its cost. The gateway keeps no request log, so
cost.jsonwasn't checked against the gateway's own meter. Gemini wasn't run against a real model; it's covered by tests that drive the real ADK runner with an in-memory model.Dileep's review (16 comments) was fixed in
159c5807andbd198d59. One cold review round cleared those two commits, with no High. Its findings went in afterwards asee163b04, unreviewed: tests that pin the timeout and crash paths, a backstop exit after the work returns, and tighter masking of the logs. Each of those fixes has a test that fails without it. Comment 5 became TH-9057.Three cold reviews of the change (Lane N). The third cleared with no High. The fixes after it are unreviewed:
Dileep's second review (7 comments) was fixed in
4617eecf,ef177b36and49c1f4c6. Two review rounds ran.Dileep's third review (5 P3s, with his approval) was fixed in
d1f968cc, with no review round.76fcb60dwas merged in. The browser test now expects feat(harness): refuse browser, computer-use and crawler agents #224's one second look before the refusal (edb69263).Checklist
agent_learning.*,agent-learn, or@future-agi/agent-learning-kitCloses TH-8951
🤖 Generated with Claude Code