Skip to content

feat(harness): read an agent without building it - #245

Merged
khushalsonawat merged 28 commits into
feat/TH-8804-source-subfolderfrom
feat/TH-8951-read-without-build
Nov 10, 2026
Merged

khushalsonawat merged 28 commits into
feat/TH-8804-source-subfolderfrom
feat/TH-8951-read-without-build

Conversation

@khushalsonawat

@khushalsonawat khushalsonawat commented Nov 5, 2026 •

Copy link
Copy Markdown
Member

Stacked on #225 (TH-8804), which is stacked on #224 (TH-8806). This PR's base is feat/TH-8804-source-subfolder, so its diff shows only TH-8951. It retargets down the stack as those merge.

Summary

The platform's Agents tab creates an agent and its first version from a repo before any environment is built, and it needs the agent's description to show. This PR adds a --read-agent-only mode to the hosted authoring command. It runs only the agent reading a build already does, leaves the result in --output (/work/reading/ on the hosted route), and stops. Nothing is built, and there is no new callback.

A run leaves:

  • contract.json: the raw reading;
  • agent.json: the agent document;
  • cost.json: the build's spend keys plus complete, whenever the reading reaches its work. A run refused before that leaves none, and none means nothing was spent. The refusals are: a bad job, a job check that crashes, a provider-defined agent, a wrong source kind, a non-empty output folder, or a failed first cost write;
  • failure.json: on failure only, in the HarnessFailure shape.

Exit 0 means read, 78 not retryable, 1 retryable. When failure.json is present, its retryable decides.

Ticket: TH-8951.

How it works

  • reading_entry.py, a helper shared with TH-8950's --find-agents (contract reading-entry v0.17). It handles:
    • the job check;
    • masking secrets files;
    • a soft limit of 1,800 s (ALK_READING_MAX_DURATION_SECONDS) and a hard timer 300 s after it;
    • a cost.json that starts complete: false and is rewritten after each turn.
  • agent_reading.py and reading_result.py, the reading. It opens its stage exactly as cli._understand does, through one shared helper (understand.reading_settings): 70 or 140 turns, thinking on, and the setup-contract gate for generic jobs. It uses the same model and gate as a build. Browser agents are refused by TH-8806's rule (agent_type_unsupported, 78).
  • Cost.
    • Claude reports running totals within one session, so each turn records only its difference from the last.
    • complete is false after the soft limit, after a stalled stream, after any unpriced turn, and after any status-less provider error, because a cut request may still be billed.
  • Turn limits are final and never retried. A turn limit is an max_turns or error_max_turns outcome, or ADK's whole "Max number of llm calls limit of n exceeded" message. If the last turn hit the limit, that wins over an earlier status-less failure.
  • What is retried: status-less failures, 408, 409, 429 and 5xx. Other 4xx responses and a forbidden billed model are not.
  • A contract the gate saved is kept through a later fault or a timeout, so the platform doesn't pay for the reading twice.

Release notes

  • The sandbox needs at least 2,400 s, which is unchanged. Start-up must fit in 240 s. The hard limit is now 2,100 s (soft 1,800 s + 300 s), so the sandbox must outlast start-up + 2,100 s + 60 s. The margin grew from 180 s on 10 Nov, so a saved reading's document always has room after the longest stop.
  • A status-less error that ends with a saved contract now gives exit 0 with complete: false. The platform charges total_usd either way and keeps the flag.

Known limits

  • A reading that reaches the provider and then dies before its first request reports "the model provider failed" rather than "crashed". The exit is the same; only the code differs. Telling them apart needs a change in Stage or a backend.
  • A connect_only job with an http or auto connector is refused, though the build loads no provider definition for it. The refusal fails closed, with a clear 78 before any spend.
  • When a turn limit on the last turn is followed by an exception, the reading ends as a turn limit, not as a crash. No test covers that order.
  • A browser agent in a large repo is read in full before the refusal, because TH-8806's check runs at save time. On browser-use that cost $6.06 and took 7.7 minutes.

Verification

  • tests/harness/test_reading_entry.py: 61 passed. tests/harness/test_agent_reading.py: 105 passed. tests/harness/test_scenario_source_provision_agent.py: 24 passed (at d1f968cc). Run on a locked venv (claude-agent-sdk 0.2.156, google-adk 2.7.1, Python 3.12.14), each whole file in one process. Both are added to sdk-smoke.yml.
  • 61 planted bugs from the plan, plus each review's and each fixer's revert checks: all caught except two plants with no site in the code.
  • Real run on Claude Sonnet, through a local gateway under a Gemini-named alias, priced at that alias's rate:
Agent Time Cost Result
Small chat agent (LangChain), after Dileep's review 80 s $0.87 exit 0, chat, complete: true
Small chat agent (LangChain), before it 113 s $1.11 exit 0, chat, complete: true
Chat agent in a subfolder (Next.js) 102 s $1.00 exit 0, chat, folder recorded
browser-use 459 s $6.06 exit 78, agent_type_unsupported
Small agent, gateway stopped 358 s $0.00 exit 1, retryable model error
Small agent, 40 s soft limit 42 s $0.60 exit 78, agent_reading_timed_out, complete: false

The interrupted Claude turn reported its cost. The gateway keeps no request log, so cost.json wasn't checked against the gateway's own meter. Gemini wasn't run against a real model; it's covered by tests that drive the real ADK runner with an in-memory model.

  • Dileep's review (16 comments) was fixed in 159c5807 and bd198d59. One cold review round cleared those two commits, with no High. Its findings went in afterwards as ee163b04, unreviewed: tests that pin the timeout and crash paths, a backstop exit after the work returns, and tighter masking of the logs. Each of those fixes has a test that fails without it. Comment 5 became TH-9057.

  • Three cold reviews of the change (Lane N). The third cleared with no High. The fixes after it are unreviewed:

    • the turn-limit rule's count signal removed and the message matched whole;
    • the last-turn precedence;
    • three new tests;
    • three test fakes corrected.
  • Dileep's second review (7 comments) was fixed in 4617eecf, ef177b36 and 49c1f4c6. Two review rounds ran.

  • Dileep's third review (5 P3s, with his approval) was fixed in d1f968cc, with no review round.

Checklist

  • Public API names stay under agent_learning.*, agent-learn, or @future-agi/agent-learning-kit
  • New user-facing workflow has docs or an example: the platform route isn't built yet (TH-8824)
  • New release claim has executable gate coverage
  • No hosted-service dependency was added to release-gated examples
  • Security-sensitive output is redacted or justified

Closes TH-8951

🤖 Generated with Claude Code

@khushalsonawat khushalsonawat self-assigned this Nov 5, 2026
@khushalsonawat
khushalsonawat force-pushed the feat/TH-8804-source-subfolder branch from 3e4e3ec to 9c187fe Compare November 6, 2026 04:49
- Add --read-agent-only to the hosted authoring command: run only the
  agent reading and leave contract.json, agent.json, cost.json and any
  failure.json in --output, then stop.
- Share one helper, reading_entry, with the agent-finding mode: job
  check, a soft limit and a hard timer, masking of secrets files, and
  a cost file that starts incomplete and is rewritten after each turn.
- Count Claude's running totals once per turn, keep a contract the gate
  saved over any later fault, and refuse browser agents by the same
  rule a build uses.
- Treat a turn limit as final, never retried: an error outcome, or
  ADK's whole call-limit message. A status-less provider failure is
  retried and marks the cost incomplete.

TH-8951

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
@khushalsonawat
khushalsonawat force-pushed the feat/TH-8951-read-without-build branch from 5b1a975 to 519f818 Compare November 6, 2026 04:51

@cdileep23 cdileep23 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please check the comments.

Comment thread src/fi/alk/harness/reading_entry.py Resolved
Comment thread src/fi/alk/harness/agent_reading.py Resolved
Comment thread src/fi/alk/harness/reading_entry.py Resolved
Comment thread src/fi/alk/harness/reading_entry.py Resolved
Comment thread src/fi/alk/harness/agent_reading.py Resolved
Comment thread src/fi/alk/harness/reading_entry.py Resolved
Comment thread src/fi/alk/harness/agent_reading.py Resolved
Comment thread tests/harness/test_agent_reading.py Resolved
Comment thread src/fi/alk/harness/agent_reading.py Resolved
Adds --find-agents to the authoring entrypoint, run through the shared
reading helper. It writes /work/reading/agents.json in the shape the
platform's AgentScanSerializer reads, plus cost.json on every return.

- A plain code walk finds candidates (framework builds, langgraph.json
  graphs, agent.py, provider clients) and joins files into agents by
  their imports, without running anything.
- Each agent's folder gets TH-8805's framework answer, with a build's
  limits, in one child process the scan's clock can stop.
- A model is asked only when the walk finds nothing usable (mostly
  TypeScript and JavaScript repos). Its read/glob/grep tools stay inside
  the source, and only the paths it names, once checked, are kept.
- Names, evidence and paths are built to fit the platform's rules, and
  the file is checked against a copy of them before it is written.
- An agent at a path the platform's rule can't carry (a Next.js
  app/(chat)/ route, say) is left out with truncated set, and is
  reported as agent_path_unsupported when nothing else is found.

TH-8950

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
…at it copied

- A job that fails validation names only where and how, never its
  values, so a secret in the job can't reach failure.json.
- Swallowed errors and crashes log a warning with their trace, masked.
- The reading's words, the refusal exit, the folder choice and the
  cost writer are shared with the build instead of copied.
- A stage takes its own idle retries; the time limits are worked out
  when a reading starts; the timer stops with the work.
- Classifying a reading's result moves to reading_result.py.

TH-8951

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
- One helper, understand.reading_settings, chooses the stage's turns,
  thinking and setup-contract gate for both the build and the reading.
- A generic job's reading now refuses what the build's gate refuses.

TH-8951

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
…s logs

- Tests pin the time-limit and crash codes, and that cost.json ends
  incomplete after a timeout or a crash the work outlived.
- After the work returns, the limit still ends a process a leftover
  thread keeps alive, with the work's own code and no new writes.
- A job error's location drops mapping keys; the mask also covers a
  secret's lines, repr and escaped forms; secrets are read per reading.
- The reading's opening shares the build's guidance; names used across
  modules are public; a dead attribute is gone.

TH-8951

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
TH-8950

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
#245 replaced reading_entry._secret_values with a functools.cache
reader keyed by the secrets files. The scan tests already point those
files at a fresh temporary path, so each test gets its own cache entry
and there is nothing left to reset.

TH-8950

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>

@cdileep23 cdileep23 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please check the comments.

Comment thread src/fi/alk/harness/reading_entry.py Resolved
Comment thread src/fi/alk/harness/reading_entry.py Resolved
Comment thread src/fi/alk/harness/reading_result.py Resolved
Comment thread src/fi/alk/harness/reading_result.py Resolved
Comment thread src/fi/alk/harness/agent_reading.py Resolved
Comment thread tests/harness/test_agent_reading.py Resolved
Comment thread src/fi/alk/harness/reading_entry.py Resolved
…ts crash

- Any model error now keeps the walk's rows with truncated set and
  exits 0; only a scan with no rows writes scan_model_error, carrying
  the model's own retryable flag. A retry would pay for the model again.
- A fault outside any one file in the walk (run or result) now reaches
  the scan's crash path (scan_crashed, retryable) instead of being
  reported as scan_limit_reached.
- Catches are narrowed to what each step expects. A file's reader
  faults still leave that file unread. A folder check that fails in the
  child sends its trace back, and the row is marked unfinished. Faults
  that are still absorbed (after the paid step, closing the session,
  tool calls) log their trace through the reading helper's MaskSecrets
  filter instead of the class name only.

TH-8950

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
- agent_scan imports EXIT_DONE, EXIT_RETRYABLE, limit_seconds and
  reading_failure from reading_entry, and EXIT_REFUSED from
  agent_type_check. FIND_AGENTS now sets limit_env and
  limit_margin_seconds, so the helper and the scan read
  ALK_SCAN_MAX_SECONDS at the same moment.
- The find-agents stage passes idle_retries=0 itself instead of setting
  session.STAGE_IDLE_RETRIES for the whole process.
- The walk uses framework_detection's _Work, which takes an optional
  limit, instead of its own copy, and bundle_author_v2's ignore list
  instead of a copy held equal by a test.
- The walk and an AI row's description share one folder loop
  (_source_files) and one shadowed-name lookup (_ShadowedNames).

TH-8950

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
- An agent's subfolder counts only the agents of its own project.
  Adding an unrelated project with its own agent deeper in the repo no
  longer moves the first agent's folder, which the platform keys on.
- The walk reads files outside test, example and other side folders
  first, then the side folders. A large tests folder sorted ahead of
  the agent can no longer use up the 20,000-file cap before the agent
  is read.

TH-8950

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
… tests plainly

- The shared-folder evidence sentence no longer asks a build to name an
  entry, which a build job has no field for.
- Test names say "the platform format", "the platform rules" and "the
  path rule" instead of "his".
- _result's docstring describes what it does instead of pointing at
  plan step numbers.

TH-8950

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
- At the hard limit a saved contract becomes agent.json and exit 0,
  instead of a timeout written over it; the close after a stop is
  bounded.
- A cancel is a crash with its failure before the save, and keeps the
  saved contract after it.
- The build and the reading share one agent-document helper, so a
  folder mismatch reports the build's code.
- A failed document logs a masked trace; provider-import and
  connect-only jobs are refused; a soft-limit test uses a fake clock.

TH-8951

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
The model step also runs when the walk found agents only in test or
example folders, so the test is no longer named "only when nothing is
found".

TH-8950

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
The catch still keeps a paid answer from becoming a retry; it now logs
the masked trace like the other absorbed faults, so a fault in the cost
writer can be found.

TH-8950

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
… check faults

From the review round on the fixes for the first review of this PR:
- One unreadable or unparsable file no longer stops the description of
  a model-found agent. It marks the row short (partial) and the reading
  goes on, as the walk goes on past an unread file.
- detect_framework takes raise_faults (off for builds). The scan's
  folder check turns it on, so a fault inside the framework check comes
  back as a trace and the row is unfinished, instead of a finished
  unknown. Both the child's and the parent's halves are tested.
- A model tool's fault keeps the worker's exception as the cause, so its
  logged trace shows the real error.
- Tests pin the run-time read of ALK_SCAN_MAX_SECONDS and the secret
  mask on every scan logger. The log comment says only the platform's
  secret values are masked.

TH-8950

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
…es with own code

From the review round on the fixes for the first review of this PR:
- The file cap is 200,000, up from 20,000. Measured: 100,000 tiny
  source files walk in 13 s at 266 MB, so the byte cap and the clock
  are what bound the walk. A repo whose own code passes 20,000 files no
  longer stops before its agent is read.
- A folder's settings template (.env.example and its kin) is read in its
  folder's pass, not with the side files, so a big tests folder can't
  keep it from being read.
- A reader fault of an unforeseen kind logs its trace once; the rest
  are counted (reader_faults in the walk's summary line).
- ALK_SCAN_AI_MAX_SECONDS is read when the scan starts, as the clock is.
- Tests: every file is read in exactly one pass; tests no longer depend
  on an exported ALK_SCAN_MAX_SECONDS; the process-wide idle-retry
  setting is set through monkeypatch.

TH-8950

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
…ond document build

- The hook that built agent.json from the timer's thread is gone: two
  document builds could starve each other past the limit. The close
  after a stop is bounded at 20 s, so the reading decides from its
  files well before the hard limit.
- The contract is read once and checked from the same bytes; a
  mismatch names subfolder_mismatch; a validation error logs no trace;
  a cancel during the stop's interrupt stays a timeout.

TH-8951

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
From the last review round on the fixes for the first review of this PR:
- A file over 1 MiB in a model-found agent's folder now marks the row
  short (partial) instead of being skipped silently, so a paid row with
  a missing file is never reported complete.
- The description logs one trace for a reader fault of an unforeseen
  kind, then counts the rest, as the walk does.
- Comments say traces are logged with the platform's secret values
  masked, not that they are fully masked.

TH-8950

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
… sandbox

From the last review round on the fixes for the first review of this PR:
- The file cap is 40,000, not 200,000. What the walk holds grows with
  the names each file imports, not its bytes: about 33 KB a file for
  files that each import 200 names, a 1,389 MB peak at 40,000 such
  files. At 200,000 such files the sandbox would kill the scan. 40,000
  still covers a repo whose own code passes 20,000 files.
- The partial-row sentence says files couldn't be read, not that they
  weren't read in time, since a file that fails to parse also sets it.
- A test pins that ALK_SCAN_AI_MAX_SECONDS is read when the model step
  is asked, and the comment says so.

TH-8950

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
The step that joins the walk's files into agents (the import search
from each candidate, and the per-agent loops) now checks a deadline:
the walk's own deadline plus 120 s, well inside the helper's 600 s
margin before the hard stop. Past it the walk ends as stopped by the
clock, so the scan writes its own file (scan_timed_out, exit 0) instead
of being ended by the hard stop.

The two-pass test now pins the order it claims: a settings template in
a folder that sorts after tests/ is read before tests/test_a.py.

TH-8950

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
…p a password out of a failed redaction

- The hard limit is soft + 300 s, so the document after the longest
  stop has room on a slow CPU; a test pins the stop's bounds inside it.
- The interrupt runs as its own task: one that swallows its cancel
  can't hold the stop, and Ctrl-C passes through.
- A URL the parser refuses is redacted whole instead of raising with
  its credentials in the message.
- The folder choice uses the contract text already checked.

TH-8951

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
#224's rule now gives every browser reading one second look first.

TH-8951

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
khushalsonawat and others added 3 commits November 10, 2026 15:30
TH-8950

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
Candidates that reach each other hold one entry per pair, which grows
with the square of the candidates. The join now counts them as it
builds them and, past 5,000,000 (about 90 MB), ends as the walk's other
caps do (stopped="limit", scan_limit_reached), instead of growing
until the clock stops it.

TH-8950

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
feat(harness): list the agents in a repo without building it

@cdileep23 cdileep23 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please check the comments.

Comment thread src/fi/alk/harness/agent_reading.py Resolved
Comment thread src/fi/alk/harness/reading_entry.py Resolved
Comment thread src/fi/alk/providers/agent.py Resolved
Comment thread tests/harness/test_agent_reading.py Resolved
Comment thread src/fi/alk/harness/scan_walk.py Resolved
- A Ctrl-C after the contract is saved now passes through; a cancel leaking
  from the session still keeps the saved contract.
- A BaseExceptionGroup (or any non-interrupt, non-exit base exception) from
  the work ends as the crash failure with exit 1.
- _redact logs only the error class when it redacts a URL whole.
- The stop-budget test pins exactly 90 s.
- Drop a person's name from a scan_walk.py comment.

Refs: TH-8951
Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
@khushalsonawat
khushalsonawat merged commit 22bcbb8 into feat/TH-8804-source-subfolder Nov 10, 2026
11 checks passed
@khushalsonawat
khushalsonawat deleted the feat/TH-8951-read-without-build branch November 10, 2026 11:26
@khushalsonawat
khushalsonawat restored the feat/TH-8951-read-without-build branch November 10, 2026 11:27
khushalsonawat added a commit that referenced this pull request Nov 10, 2026
… for Dileep

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
khushalsonawat added a commit that referenced this pull request Nov 11, 2026
[TH-9454] fix: follow-ups from #245's last review of the agent reading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants