Skip to content

feat(harness): let a build's first run reuse the agent reading (TH-8952) - #269

Merged
khushalsonawat merged 6 commits into
feat/TH-8804-source-subfolderfrom
feat/TH-8952-seed-contract
Nov 11, 2026
Merged

khushalsonawat merged 6 commits into
feat/TH-8804-source-subfolderfrom
feat/TH-8952-seed-contract

Conversation

@khushalsonawat

@khushalsonawat khushalsonawat commented Nov 10, 2026 •

Copy link
Copy Markdown
Member

Stacked on feat/TH-8804-source-subfolder. #245 (TH-8951), which this PR was cut from, merged into it, so GitHub moved this PR's base there; the diff still shows only TH-8952. It retargets down the stack as the PRs under it merge.

Summary

Today every new agent version is read twice: once when it is added (the reading from #245, which the customer pays for), and again by its first build. With this PR, a build can be handed that reading and skip the second one.

The platform puts the reading's raw contract.json at /work/seed/contract.json and adds --seed-contract /work/seed/contract.json to hosted_authoring_entrypoint. The build then opens at the world stage with that contract, byte for byte, so the world is built from the reading's own base_environment rows.

It reuses the reading only when it was made from the same:

  • commit (or upload);
  • folder;
  • eval list;
  • generic-harness flag.

Otherwise the build reads the agent as it does today. A file that is in the wrong place, unreadable, not JSON or over 5 MiB fails the build in the usual failure.json shape, with exit 78.

Ticket: TH-8952.

Review after Dileep's comments

  • The fixes for Dileep's review were reviewed by two cold reviewers (round "dfix" r1: no Critical or High). Their Mediums and Lows went into 078bf7e1, together with the Coordinator's two calls: the outcome file under generic-harness/, and a restart that finds the seed installed says used.
  • The whole PR at 078bf7e1 was reviewed by two more cold reviewers (round "pushr" r1: no Critical or High). It covered every change since the last cleared round, including the contract check before a seed is used, the install lock, the log line, the --find-agents merge resolution and the CI smoke line. Its three Mediums and ten Lows are fixed in f5082a8f:
    • An install record. use_seed writes generic-harness/.seed-installed after a whole install. If it can't be written, the install is taken back and the outcome is error. A restart with no outcome file then says used even after a contract edit has rewritten contract.json.
    • What used means. The build started from the seed. Contract edits later drop reading_inputs, so the platform keeps its stored reading on used.
    • The "no file" list and the limits are corrected, including a lost not_accepted followed by a restart that installs, which is a correct build.
    • The seed stays in place until seed-result.json exists or the build is over.
    • Exception groups: a ValidationError inside one prints nothing.
    • New tests pin:
      • a same-length late save;
      • temp files in their target's folder;
      • a source model that was already there surviving a take-back;
      • holds_seed never raising;
      • the traceback on a lost write;
      • a context cycle.
    • Comments and contract wording.
  • A job that isn't generic (generic_harness_v1 false or absent) failed to record its install, because only the generic step makes generic-harness/. It was taken back as error, and the agent was read again. _write_whole now makes the folder. This was found by a local end-to-end dry run on feat/TH-8952-e2e-all, reviewed in cold round "ngfix" r1 (two reviewers, no Critical, High or Medium), and is fixed in 1b658e69. Production doesn't reach it yet: no platform code passes --seed-contract. Once it does, a hosted job whose metadata says generic_harness_v1: false would, because the gateway only defaults the flag to true.

How it works

  • The reading records what it was made from. reading_result.document() adds one top-level entry, reading_inputs, to the reading's contract.json. It holds:

    • the commit or upload id;
    • the folder;
    • a hash of the offered eval list;
    • the generic flag.

    The entry is spliced in, so every other byte stays as the reading saved it. It is skipped when it would push the file past 5 MiB. The agent document never sees it.

  • The build command checks the file. With --await-harness (the hosted fresh build), it checks the flag's path and reads the file, never through a link or a pipe. It refuses with seed_contract_invalid or seed_contract_too_large ({domain platform_sync, stage understanding_agent, retryable false}). On success it leaves a mark, /work/authoring/.futureagi-seed-accepted. With --adjustments the seed is ignored, with one log line, and a builder that opens there with the seed placed records not_accepted. The flag can't be combined with --certify-only, --read-agent-only or --find-agents.

  • The builder installs it when it first opens. The builder conversation opens every reading and every world stage in one place, _open_builder. When the mark is there and nothing is saved yet, it:

    1. compares the record with the job;
    2. loads the contract and runs the reading's own structural check on it;
    3. writes the raw bytes as /work/authoring/contract.json;
    4. runs the step that normally follows a reading (the source model the world stage's tools read).

    It then opens the world stage. Anything that doesn't match, doesn't load, or fails that step leaves nothing behind, and the build reads as today. The builder decides once, at its first opening, consumes the mark whatever it decided, and writes the outcome for the platform (below). It never fails a build over a seed. It also never waits, so it keeps checking in with the platform.

  • Builds without a seed: when the builder opens with a saved contract and no world, it now writes the source model first, as it already did after the reading's turn. It opens the world stage even if that step fails, logging the step's traceback.

What the platform reads: seed-result.json

  • Path: /work/authoring/generic-harness/seed-result.json, written whole (a temp file, then a rename), so the platform never reads half a file. Under generic-harness/ it is stored with the version but never hashed.

  • Shape: {"seed": <outcome>, "differs": [<field names>]}. differs is empty unless the outcome is mismatch.

  • Outcomes:

    • used: the build started from the seed, installed as its contract. Contract tools later in the build rewrite contract.json without reading_inputs, so the build's final contract may differ from the seed and can't seed a later build. On used, keep the stored reading; never replace it with the build's contract.
    • mismatch: the reading was made from other inputs; differs names them.
    • rejected: it doesn't load, or fails the reading's own check.
    • unreadable: it changed after it was accepted.
    • setup_failed: the source-model step failed, and the seed was taken back.
    • error: anything else.
    • not_accepted: the builder opened before the build command accepted it.

    Anything but used means the build read the agent itself.

  • When it is written: by the builder, at the first opening of a fresh seeded build, after the install or the decision not to install. If the write fails, the next opening in that process writes it again. Each whole install also leaves a record, generic-harness/.seed-installed; this is written for every job, generic or not, and the builder makes generic-harness/ when it isn't there; if that can't be written, the install is taken back and the outcome is error. A new process that finds no file, a contract, and either that record or the seed's exact bytes as contract.json, writes used.

  • Once it exists, the builder never installs a seed in that workspace, on any later opening.

  • No file, once the build is over:

    • no seed was placed, or it wasn't there at the builder's first opening and the builder never restarted after it landed;
    • the build command refused it (failure.json says why);
    • a contract, a world or a finished build was already there;
    • the builder stopped before deciding and never opened again, so the build never got past its first opening;
    • an outcome other than used couldn't be written, and the builder restarted after its own reading had saved;
    • the seed was removed, against the rule below, after an install whose outcome couldn't be written, and the builder restarted.

    While the seed stays in place, no build that leaves no file ran on the seed.

  • Limits, both after a failed write:

    • an outcome other than used, followed by a restart before the builder's own reading saved, is recorded as not_accepted: "not used" is still right, but the reason is lost;
    • a not_accepted, followed by the mark landing and a restart before the builder's own reading saved, lets the restart install the seed and record used. The build is right; the reading it had begun is thrown away.
  • The platform must keep the seed in place until seed-result.json exists or the build is over, and read the file only from that seeded attempt's own workspace, checkpoint or version: it names neither the seed nor the attempt.

Verification

  • ruff check on the new files, and no new findings in the edited files against the base.

  • Focused pytest, one file at a time, after merging feat(harness): read an agent without building it #245's tip d1f968cc:

    file passed
    test_seed_contract 108
    test_hosted_authoring_entrypoint 12
    test_authoring_entrypoint_find_agents 5
    test_agent_reading 105
    test_agent_type_refusal 463
    test_reading_entry 61
    test_builder_chat 53
    test_hosted_conversation 21
    test_agent_scan 63
  • 86 deliberate bugs, planted one at a time (dropping each compared field, following a link, installing outside the builder's first opening, removing the lock, printing a validation error's values, writing the outcome in place, and so on). Each one made a named test fail.

  • Full uv run pytest -q: not run on the laptop. CI's SDK smoke job now includes test_seed_contract.py.

  • TypeScript: not touched.

Known limits

  • A builder that opens before the build command has marked the seed reads as today, and records not_accepted. The two processes start independently. The build command reaches its mark about 2 to 3 s after it starts (estimated: a ~2 s import, measured on a laptop, then the job and folder checks). The builder only starts after that command is launched, then waits on a background task and several sandbox calls, plus its own ~2 s import, before it opens. So this needs the build command to stall before its check; I expect it to be rare. Not measured on a hosted sandbox. When it happens, the cost is one reading, never a wrong build.
  • Only the commit or upload, folder, eval list and generic flag are compared (as agreed for reusing a reading). A reading made under different build guidance, target model or environment names can still be reused.
  • The eval list changes whenever eval templates are added, so many builds won't match and will read as today.
  • reading_inputs stays in an installed contract, so it counts toward the platform's contract hash and shows in its contract diffs.
  • feat(harness): add scenario proofs, certify-only sealing and nudge rebuild turns #180's rebuild turn doesn't use the seed yet. A rebuild starts from the stored authoring folder, where contract.json already exists, so this command never runs there. TH-9453 covers it.

Checklist

  • Public API names unchanged
  • New user-facing workflow has docs or an example: not user-facing; the platform passes the flag
  • No new release claim
  • No hosted-service dependency was added to release-gated examples
  • Security-sensitive output is redacted or justified: messages are fixed sentences and logs carry names and counts only, never the seed's content or a validation error's text

A build given --seed-contract (the reading's raw contract.json at
/work/seed/contract.json) skips reading the agent again and opens at the
world stage, when the reading was made from the same commit or upload,
folder, eval list and generic flag. Otherwise it reads as before.

- The reading records what it was made from in its contract.json.
- The build command refuses a seed that is in the wrong place, unreadable,
  not JSON or over 5 MiB, as failure.json with exit 78, and marks one it
  accepts.
- The builder installs an accepted seed only when its first stage opens,
  so a reading that is already open is never replaced.

TH-8952

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
Brings in #251 (TH-8950, --find-agents) and #225 (TH-8804). The seed flag
sits beside the reading modes' exclusive group and now also refuses
--find-agents.

TH-8952

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
TH-8952

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
Base automatically changed from feat/TH-8951-read-without-build to feat/TH-8804-source-subfolder November 10, 2026 11:26

@cdileep23 cdileep23 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please check the comments.

Comment thread src/fi/alk/harness/seed_contract.py Resolved
Comment thread src/fi/alk/harness/seed_contract.py Resolved
Comment thread src/fi/alk/harness/seed_contract.py Resolved
Comment thread src/fi/alk/harness/reading_result.py Resolved
Comment thread tests/harness/test_seed_contract.py Resolved
…latform

Answers the review of #269 (TH-8952):
- The builder decides at its first opening, consumes the mark whatever it
  decided, and writes generic-harness/seed-result.json, whole. A later
  opening never installs: in a process the decision is kept in memory, and
  across a restart the file stops it. A restart that finds the seed
  installed byte for byte, with no file, writes "used".
- The file sits under generic-harness/, so the outcome never moves the
  environment's identity hash on the platform.
- Unexpected errors keep their traceback; nothing is printed when a
  ValidationError is anywhere in the chain, since its text repeats values.
- The reading's record is spliced into the bytes document() already read,
  and written only if the file still holds them.
- The build command clears a stale mark before its seed checks.

Refs: TH-8952
Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
khushalsonawat added a commit that referenced this pull request Nov 10, 2026
Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
From the review of 078bf7e (no Critical or High):
- use_seed leaves generic-harness/.seed-installed after a whole install,
  and takes the install back if it can't. A restart with no outcome file
  says "used" from that record even after a contract edit has rewritten
  contract.json.
- holds_values walks exception groups, so a ValidationError inside one
  never prints its values.
- New tests pin a same-length late save, temp files beside their target,
  a source model surviving a take-back, holds_seed never raising, the
  traceback on a lost write and a context cycle.

Refs: TH-8952
Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
khushalsonawat added a commit that referenced this pull request Nov 10, 2026
Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
…eneric

Only a generic-harness job's own step makes generic-harness/, so for a job
whose generic_harness_v1 is false or absent the seed's install record had
no folder to go in. The install was taken back as an error and the agent
was read again. _write_whole now makes the parent folder. New tests cover
a non-generic job through use_seed and through the builder, and a reading
with no flag seeding a build whose flag is false.

Found by a local end-to-end dry run; reviewed in cold round ngfix r1.

Refs: TH-8952
Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
khushalsonawat added a commit that referenced this pull request Nov 10, 2026
…e combined branch for Dileep

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>

@cdileep23 cdileep23 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please check the comments.

Comment thread src/fi/alk/harness/seed_contract.py Resolved
Comment thread src/fi/alk/harness/seed_contract.py Resolved
@khushalsonawat
khushalsonawat merged commit 828afea into feat/TH-8804-source-subfolder Nov 11, 2026
11 checks passed
@khushalsonawat
khushalsonawat deleted the feat/TH-8952-seed-contract branch November 11, 2026 05:51
@khushalsonawat
khushalsonawat restored the feat/TH-8952-seed-contract branch November 11, 2026 05:51
@khushalsonawat
khushalsonawat deleted the feat/TH-8952-seed-contract branch November 11, 2026 05:51
khushalsonawat added a commit that referenced this pull request Nov 11, 2026
…288) into TH-9491

- tests/test_harness_sandbox_server.py: both sides appended tests; kept both.
- test_contract_writer's allow-list gains #269's two contract.json copies:
  reading_result.document's replace_if_unchanged (the saved contract's own bytes
  plus the reading record) and seed_contract.use_seed's _write_whole (the
  platform's seed, as written).
- sandbox_server.py imports SecretResolutionError, which #284's resolver raises
  but the base never imports: seven sandbox-server tests fail with NameError on
  the base itself.

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
khushalsonawat added a commit that referenced this pull request Nov 11, 2026
…total

Brings in #269, #276 (through #283), #283 and #288. Three hunks against
#288 (TH-9448) were resolved by hand, keeping both intents:
- a skipped file logs its type once at WARNING, and its path and trace
  at DEBUG (#288); it is also counted as the scan's `skipped` cut, and
  that count is in the cut line, so #288's separate "skipped N files"
  line is not repeated (TH-9512);
- #288's skip tests and its NUL-byte test now expect the cut line, which
  holds names and counts only.

Tests on 3.12.3: test_agent_type_refusal 720, test_browser_scan_cut 59,
test_contract_tools_subfolder 180, all passed.

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants