Repository navigation
feat(harness): let a build's first run reuse the agent reading (TH-8952) - #269
Merged
khushalsonawat merged 6 commits intoNov 11, 2026
Merged
Conversation
A build given --seed-contract (the reading's raw contract.json at /work/seed/contract.json) skips reading the agent again and opens at the world stage, when the reading was made from the same commit or upload, folder, eval list and generic flag. Otherwise it reads as before. - The reading records what it was made from in its contract.json. - The build command refuses a seed that is in the wrong place, unreadable, not JSON or over 5 MiB, as failure.json with exit 78, and marks one it accepts. - The builder installs an accepted seed only when its first stage opens, so a reading that is already open is never replaced. TH-8952 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
TH-8952 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
Base automatically changed from
feat/TH-8951-read-without-build
to
feat/TH-8804-source-subfolder
November 10, 2026 11:26
cdileep23
requested changes
Nov 10, 2026
5 of 10 tasks
…latform Answers the review of #269 (TH-8952): - The builder decides at its first opening, consumes the mark whatever it decided, and writes generic-harness/seed-result.json, whole. A later opening never installs: in a process the decision is kept in memory, and across a restart the file stops it. A restart that finds the seed installed byte for byte, with no file, writes "used". - The file sits under generic-harness/, so the outcome never moves the environment's identity hash on the platform. - Unexpected errors keep their traceback; nothing is printed when a ValidationError is anywhere in the chain, since its text repeats values. - The reading's record is spliced into the bytes document() already read, and written only if the file still holds them. - The build command clears a stale mark before its seed checks. Refs: TH-8952 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
khushalsonawat
added a commit
that referenced
this pull request
Nov 10, 2026
From the review of 078bf7e (no Critical or High): - use_seed leaves generic-harness/.seed-installed after a whole install, and takes the install back if it can't. A restart with no outcome file says "used" from that record even after a contract edit has rewritten contract.json. - holds_values walks exception groups, so a ValidationError inside one never prints its values. - New tests pin a same-length late save, temp files beside their target, a source model surviving a take-back, holds_seed never raising, the traceback on a lost write and a context cycle. Refs: TH-8952 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
khushalsonawat
added a commit
that referenced
this pull request
Nov 10, 2026
…eneric Only a generic-harness job's own step makes generic-harness/, so for a job whose generic_harness_v1 is false or absent the seed's install record had no folder to go in. The install was taken back as an error and the agent was read again. _write_whole now makes the parent folder. New tests cover a non-generic job through use_seed and through the builder, and a reading with no flag seeding a build whose flag is false. Found by a local end-to-end dry run; reviewed in cold round ngfix r1. Refs: TH-8952 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
khushalsonawat
added a commit
that referenced
this pull request
Nov 10, 2026
cdileep23
approved these changes
Nov 11, 2026
khushalsonawat
merged commit Nov 11, 2026
828afea
into
feat/TH-8804-source-subfolder
11 checks passed
khushalsonawat
added a commit
that referenced
this pull request
Nov 11, 2026
…288) into TH-9491 - tests/test_harness_sandbox_server.py: both sides appended tests; kept both. - test_contract_writer's allow-list gains #269's two contract.json copies: reading_result.document's replace_if_unchanged (the saved contract's own bytes plus the reading record) and seed_contract.use_seed's _write_whole (the platform's seed, as written). - sandbox_server.py imports SecretResolutionError, which #284's resolver raises but the base never imports: seven sandbox-server tests fail with NameError on the base itself. Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
khushalsonawat
added a commit
that referenced
this pull request
Nov 11, 2026
…total Brings in #269, #276 (through #283), #283 and #288. Three hunks against #288 (TH-9448) were resolved by hand, keeping both intents: - a skipped file logs its type once at WARNING, and its path and trace at DEBUG (#288); it is also counted as the scan's `skipped` cut, and that count is in the cut line, so #288's separate "skipped N files" line is not repeated (TH-9512); - #288's skip tests and its NUL-byte test now expect the cut line, which holds names and counts only. Tests on 3.12.3: test_agent_type_refusal 720, test_browser_scan_cut 59, test_contract_tools_subfolder 180, all passed. Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
This was referenced Nov 11, 2026
5 of 10 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on
feat/TH-8804-source-subfolder. #245 (TH-8951), which this PR was cut from, merged into it, so GitHub moved this PR's base there; the diff still shows only TH-8952. It retargets down the stack as the PRs under it merge.Summary
Today every new agent version is read twice: once when it is added (the reading from #245, which the customer pays for), and again by its first build. With this PR, a build can be handed that reading and skip the second one.
The platform puts the reading's raw
contract.jsonat/work/seed/contract.jsonand adds--seed-contract /work/seed/contract.jsontohosted_authoring_entrypoint. The build then opens at the world stage with that contract, byte for byte, so the world is built from the reading's ownbase_environmentrows.It reuses the reading only when it was made from the same:
Otherwise the build reads the agent as it does today. A file that is in the wrong place, unreadable, not JSON or over 5 MiB fails the build in the usual
failure.jsonshape, with exit 78.Ticket: TH-8952.
Review after Dileep's comments
078bf7e1, together with the Coordinator's two calls: the outcome file undergeneric-harness/, and a restart that finds the seed installed saysused.078bf7e1was reviewed by two more cold reviewers (round "pushr" r1: no Critical or High). It covered every change since the last cleared round, including the contract check before a seed is used, the install lock, the log line, the--find-agentsmerge resolution and the CI smoke line. Its three Mediums and ten Lows are fixed inf5082a8f:use_seedwritesgeneric-harness/.seed-installedafter a whole install. If it can't be written, the install is taken back and the outcome iserror. A restart with no outcome file then saysusedeven after a contract edit has rewrittencontract.json.usedmeans. The build started from the seed. Contract edits later dropreading_inputs, so the platform keeps its stored reading onused.not_acceptedfollowed by a restart that installs, which is a correct build.seed-result.jsonexists or the build is over.ValidationErrorinside one prints nothing.holds_seednever raising;generic_harness_v1false or absent) failed to record its install, because only the generic step makesgeneric-harness/. It was taken back aserror, and the agent was read again._write_wholenow makes the folder. This was found by a local end-to-end dry run onfeat/TH-8952-e2e-all, reviewed in cold round "ngfix" r1 (two reviewers, no Critical, High or Medium), and is fixed in1b658e69. Production doesn't reach it yet: no platform code passes--seed-contract. Once it does, a hosted job whose metadata saysgeneric_harness_v1: falsewould, because the gateway only defaults the flag to true.How it works
The reading records what it was made from.
reading_result.document()adds one top-level entry,reading_inputs, to the reading'scontract.json. It holds:The entry is spliced in, so every other byte stays as the reading saved it. It is skipped when it would push the file past 5 MiB. The agent document never sees it.
The build command checks the file. With
--await-harness(the hosted fresh build), it checks the flag's path and reads the file, never through a link or a pipe. It refuses withseed_contract_invalidorseed_contract_too_large({domainplatform_sync, stageunderstanding_agent, retryablefalse}). On success it leaves a mark,/work/authoring/.futureagi-seed-accepted. With--adjustmentsthe seed is ignored, with one log line, and a builder that opens there with the seed placed recordsnot_accepted. The flag can't be combined with--certify-only,--read-agent-onlyor--find-agents.The builder installs it when it first opens. The builder conversation opens every reading and every world stage in one place,
_open_builder. When the mark is there and nothing is saved yet, it:/work/authoring/contract.json;It then opens the world stage. Anything that doesn't match, doesn't load, or fails that step leaves nothing behind, and the build reads as today. The builder decides once, at its first opening, consumes the mark whatever it decided, and writes the outcome for the platform (below). It never fails a build over a seed. It also never waits, so it keeps checking in with the platform.
Builds without a seed: when the builder opens with a saved contract and no world, it now writes the source model first, as it already did after the reading's turn. It opens the world stage even if that step fails, logging the step's traceback.
What the platform reads:
seed-result.jsonPath:
/work/authoring/generic-harness/seed-result.json, written whole (a temp file, then a rename), so the platform never reads half a file. Undergeneric-harness/it is stored with the version but never hashed.Shape:
{"seed": <outcome>, "differs": [<field names>]}.differsis empty unless the outcome ismismatch.Outcomes:
used: the build started from the seed, installed as its contract. Contract tools later in the build rewritecontract.jsonwithoutreading_inputs, so the build's final contract may differ from the seed and can't seed a later build. Onused, keep the stored reading; never replace it with the build's contract.mismatch: the reading was made from other inputs;differsnames them.rejected: it doesn't load, or fails the reading's own check.unreadable: it changed after it was accepted.setup_failed: the source-model step failed, and the seed was taken back.error: anything else.not_accepted: the builder opened before the build command accepted it.Anything but
usedmeans the build read the agent itself.When it is written: by the builder, at the first opening of a fresh seeded build, after the install or the decision not to install. If the write fails, the next opening in that process writes it again. Each whole install also leaves a record,
generic-harness/.seed-installed; this is written for every job, generic or not, and the builder makesgeneric-harness/when it isn't there; if that can't be written, the install is taken back and the outcome iserror. A new process that finds no file, a contract, and either that record or the seed's exact bytes ascontract.json, writesused.Once it exists, the builder never installs a seed in that workspace, on any later opening.
No file, once the build is over:
failure.jsonsays why);usedcouldn't be written, and the builder restarted after its own reading had saved;While the seed stays in place, no build that leaves no file ran on the seed.
Limits, both after a failed write:
used, followed by a restart before the builder's own reading saved, is recorded asnot_accepted: "not used" is still right, but the reason is lost;not_accepted, followed by the mark landing and a restart before the builder's own reading saved, lets the restart install the seed and recordused. The build is right; the reading it had begun is thrown away.The platform must keep the seed in place until
seed-result.jsonexists or the build is over, and read the file only from that seeded attempt's own workspace, checkpoint or version: it names neither the seed nor the attempt.Verification
ruff checkon the new files, and no new findings in the edited files against the base.Focused pytest, one file at a time, after merging feat(harness): read an agent without building it #245's tip
d1f968cc:test_seed_contracttest_hosted_authoring_entrypointtest_authoring_entrypoint_find_agentstest_agent_readingtest_agent_type_refusaltest_reading_entrytest_builder_chattest_hosted_conversationtest_agent_scan86 deliberate bugs, planted one at a time (dropping each compared field, following a link, installing outside the builder's first opening, removing the lock, printing a validation error's values, writing the outcome in place, and so on). Each one made a named test fail.
Full
uv run pytest -q: not run on the laptop. CI's SDK smoke job now includestest_seed_contract.py.TypeScript: not touched.
Known limits
not_accepted. The two processes start independently. The build command reaches its mark about 2 to 3 s after it starts (estimated: a ~2 s import, measured on a laptop, then the job and folder checks). The builder only starts after that command is launched, then waits on a background task and several sandbox calls, plus its own ~2 s import, before it opens. So this needs the build command to stall before its check; I expect it to be rare. Not measured on a hosted sandbox. When it happens, the cost is one reading, never a wrong build.reading_inputsstays in an installed contract, so it counts toward the platform's contract hash and shows in its contract diffs.contract.jsonalready exists, so this command never runs there. TH-9453 covers it.Checklist