[KarthikAvinashFI]

1. Why

Run readiness needs facts that only the harness has: an agent version's own tools when no build has read them, why a check failed, and which tools actually reached the test world during qualification. Without them the platform cannot see a tool a new version added, has to parse the guest log to tell whose failure a check was, and cannot tell a tool routed to the test world from one pointed somewhere else. A tool trace written by the agent itself proves only that the agent says it called a tool, not that the call arrived at the world.

The PR also fixes certify-only for environments whose saved source model was recorded by another harness version: the seed fingerprint covered the harness's own _alk_ tables, which differ by version, so a valid saved World IR was refused with source_model_fingerprint_mismatch.

Companion PR: future-agi #3784 (run readiness part 2) reads the records. Merge order does not matter: each reader falls back to today's behaviour when a file is absent.

2. What changed

File	Change	Preserved
harness/hosted_agent_reading.py (new)	python -m fi.alk.harness.hosted_agent_reading --source --output: sets up the authoring credentials, runs understand on the source, then writes agent.json ({sdk_version, agent}) built with the existing build_authored_agent. It always exits 0, and a failure leaves no document	The understand stage itself
harness/hosted_authoring_entrypoint.py	Credential setup is moved into configure_authoring_credentials() so the reading reuses it	Same values, same target-secret boundary
harness/failure_record.py (new)	Writes /work/failure.json (domain harness, world or agent; phase, stage, code, process, and one redacted line of the harness's own text) from the typed ProcessRuntimeError and the structured diagnostics the failure carries, never from the agent's output; a bare runtime error the harness raised with its own typed code text is read as the harness's. A later failure replaces it; a passing validation removes it. Diagnostic only: the tested agent shares the sandbox's OS user and can write /work, so nothing may use domain to gate a run or decide who owns a failure until the writer is isolated or the record is authenticated; nothing in ALK reads it
harness/chat_readiness.py	record_tool_routing writes generic-harness/tool-routing-observations.json from the world's side only. A tool is routed: true when the world's call log (_alk_tool_trace) or the harness's own execution of a returned call shows it arrived. A tool the agent reports calling, whose calls the world logs (a service tool behind the tool proxy, or a tool the bundle's MCP toolbox serves) but never received, is routed: false, reason: not_received. A reported tool the world keeps no record of (the agent runs it in its own process) is listed under unobserved as no_world_record, never as routed. request_uncovered_tools then asks the agent once for each tool not yet shown, driving the request like a scenario turn so returned calls run at the world, and reads the result the same way. A tool the contract says leaves the test world is not asked (reaches_outside), and neither is any tool the contract says has effects (has_effects), whether or not the world logs its calls. Each request has one timeout from the check's remaining budget around the whole turn, however many times the agent is called in it	The probe, its receipt and its pass rule
harness/chat_call_runner.py	_HostedChatTarget.received: the calls the harness ran at the world for the agent, exposed on the runner after a conversation	What a conversation records and grades
harness/mcp_toolbox_adapter.py	Each MCP toolbox call is also written to the world's call log, as the tool proxy does. A toolbox client whose Accept header does not name JSON (toolbox-core sends */*) gets application/json added: the JSON-response server refused every such client with 406 on the production template's mcp version, so no toolbox agent could load its tools	The file trace and the tool's response
harness/process_runtime.py	Seeding a saved World IR composes the live schema with the harness's own _alk_ tables as the build recorded them, or leaves them out where it recorded none (_without_later_harness_tables)	Any change to the agent's own tables or columns still refuses
harness/authoring_runtime_validation.py	Both validation loops record or clear the failure; both qualification paths (workflow probe and test message) record routing and then run the tool requests; neither can fail the check	Repair decisions and printed lines

3. Behavior callouts

• A check sandbox gets /work/failure.json only when a validation fails. Certified bundles carry generic-harness/tool-routing-observations.json.
• Routing never counts what the agent reports. An agent that runs its tools in its own process reads "no world record" for them, not routed.
• Environments whose saved source model lacks (or holds a narrower) _alk_tool_trace than the running harness creates now certify instead of failing source_model_fingerprint_mismatch. A saved model recorded for different agent tables still fails.
• The reading runs only when the platform asks for it (on an explicit Re-check). It makes model calls through the understand stage, journalled where a build's are.
• Qualification now makes extra agent turns, on every build, every check, and an ordinary run whose certify misses the bundle cache: after the qualification conversation, the harness sends one fresh message per tool it did not show. Never asked: tools whose contract says they leave the world (reaches_outside), and every tool whose contract names effects (has_effects), logged by the world or not: a logged call is observable, not safe, so a synthetic request only ever reads. At most 8 tools and 5 minutes in all for the whole check (TOOL_REQUEST_LIMIT, TOOL_REQUEST_SECONDS), enforced around each whole request, not per agent call: a retry attempt reuses the earlier answers from generic-harness/tool-requests.json and asks only within what is left. Each request is a model call on the customer's agent. Tools left over are recorded not_asked and read Partial on the platform, never Blocked.
• The failure record is cleared at the start of every validation attempt, so a record left by an earlier attempt never decides whose failure a later one was.
• The actors change (feat(harness): actors, one world door for agent and actors, on top of dev #257) adds the same _without_later_harness_tables helper; whichever merges second resolves one identical hunk.

4. Tests

Mutation checks (8 mutants on routing, 4 on the seed composition, 3 on the failure domain, each test seen failing with its change reverted) were run locally on earlier commits and are not re-run on this head. Test counts below are counted from the code on this head.

Test	Locks
tests/harness/test_tool_routing_observations.py (14 tests, 15 cases)	Routed only from the world's call log or the harness's own execution, with the proxy's path name mapped to the tool; reported but not received is not_received; a tool the agent runs itself is no_world_record; each uncalled tool asked once, a returned call run at the world, a contract-outside tool never asked, a tool with effects never asked (world logs it or not), one request bounded as a whole, an unanswered or ignored request named
tests/harness/test_chat_readiness.py (1 new, 1 updated)	A workflow probe whose callback ran the tool itself records nothing routed; a call returned to the harness and run at the world is routed
tests/harness/test_mcp_toolbox_adapter.py (1 new with 2 cases, 1 updated)	A toolbox call is written to the world's call log; a client sending */* or no Accept lists the tools (real request through the adapter)
tests/harness/test_process_runtime.py::test_canonical_seed_reads_the_harness_tables_as_the_build_recorded_them (5 cases)	Log not recorded by the build, log widened since, same harness: seeds; agent table or column added: refuses
tests/harness/test_failure_record.py (10 tests, 19 cases)	Domain per error from its fields and diagnostics (a message cannot decide it); the agent's output can neither forge a domain nor leak into the message; one redacted line; replace, clear and never raise; both validation loops write and clear
tests/harness/test_hosted_agent_reading.py (7)	The document is built from the contract; credentials first; spend to the named journal; failed understand, partial or unreadable contract leaves no document

Fixed on 10 Nov after an independent review

Path	How tested	Result
A retry attempt of the same check	test_a_later_attempt_of_the_check_reuses_the_answers_and_stays_within_the_limit	No request is sent again; the earlier answers are reused; the limit and time hold for the whole check
A tool that changes state and the world keeps no record of	now test_a_tool_with_effects_is_never_asked_even_when_the_world_logs_its_calls	Recorded has_effects (was not_observable), never asked

Fixed on 12 Nov after review (36eaa36f, on dev merged in at d16c7884)

Review point	Change	Test (each fails with its fix reverted)
1. Probing could ask an effectful tool whenever the world logged it	Every tool whose contract names effects is skipped and recorded has_effects, before any earlier answer is reused	test_a_tool_with_effects_is_never_asked_even_when_the_world_logs_its_calls (world logs it, and no world record)
2. Each of up to eight agent calls in one request got a fresh timeout	One asyncio.timeout from the remaining budget around the whole target.say(); a request that runs out is request_failed and the rest not_asked	test_one_request_stays_within_the_budget_however_many_times_the_agent_is_called (slow agent returning a tool call every time: ends inside the 0.5 s budget)
3. /work/failure.json shares the agent's OS boundary	Stated diagnostic-only in failure_record.py and above; nothing in ALK reads domain	none (no behaviour change)

Fixed on 12 Nov after a second review (88e7204c)

During a readiness tool request only the asked tool runs at the world; any other call the agent returns (for example a tool with effects) is refused, not run, and listed under refused_during_requests in the routing file. Test: test_a_request_runs_only_the_asked_tool_at_the_world, seen failing with the guard removed. A tool whose contract leaves effects empty is treated as read-only, and a callback agent runs its own tools before answering, which the harness cannot stop; the world is reset after the check. The reading module's docstring no longer says the check makes no model call.

Reasons this PR writes, for the platform's reason table (future-agi #3784)

In generic-harness/tool-routing-observations.json: under tools, routed: false with reason: not_received; under unobserved, reason is one of no_world_record, reaches_outside, has_effects, not_asked, request_failed, not_called. not_observable is no longer written; has_effects replaces it. future-agi #3784 maps every one of these reasons to its own words on the tool routing row. Like the tool certification beside it, this file is written inside the check's sandbox, which the tested agent shares; readiness relies on that same boundary for both.

4a. What we tested on

Local platform stack on production sandbox specs (production template, 4 CPU, 8192 MB, 23 GB, restricted egress, svc-control), this branch's 8 Nov commits (up to ca45c5f3) overlaid on the template's source; 47256eab and 7b026f9e (the 9 and 10 Nov bound and record fixes) were not re-run live; no other local change. No run or job ids were recorded for these rows.

Case	Kind	Quality check	Result
Same-behaviour new version of a chat agent with five tools behind its own tools service	HTTP agent, service tools behind the tool proxy	Every tool's call arrives at the world	Qualification's conversation showed three tools at the proxy; the one-shot requests showed the other two (both refused by the world's rules, still received). All five routed: true; the platform read routing Supported and the pairing ready
The same agent with one tool pointed at a stand-in server inside its own sandbox (nothing real)	Same	The misrouted tool is caught, the rest are not	get_deal reported by the agent, routed: false, not_received; the other four routed. The platform blocked every case using it
Certify-only on an environment whose saved source model recorded no _alk_tool_trace (built under the actors change)	Seed of a saved World IR	Seeds without rebuilding or dropping saved state	Certified; before the fix the same check failed source_model_fingerprint_mismatch
Certify-only on an environment built by the released harness (LangGraph, in-process tools)	Seed of a saved World IR	Seeds under this branch	World ready in 15 s, no fingerprint error. The check then failed on the agent's own reply: its model key is rejected on the local stack
Certify-only on a multi-agent returns chat app whose tools run in its own process (Anthropic model)	HTTP agent, in-process tools	Reported calls are never counted as routed	get_order and list_refunds reported by the agent, no world record: both no_world_record; issue_refund asked once, not called (before 12 Nov; it names effects, so it is now has_effects and not asked). The platform read routing Partial naming each reason and the pairing ready. The environment's saved model recorded no _alk_tool_trace; it seeded without error
A minimal catalog agent whose two SQL tools are served by the MCP toolbox (Anthropic model)	MCP toolbox	Every tool's call arrives at the world	Built, then certified: find_items and get_item both in the world's call log through the toolbox; routing Supported, pairing ready. Before the Accept fix, its build failed: the toolbox answered 406 and the agent could not load its tools
The same agent with a JSON reply that reports its tool calls, get_item pointed at a stand-in inside its own sandbox	MCP toolbox	The bypass is caught	get_item reported, routed: false, not_received; find_items routed. The platform read the pairing Blocked, 3 of 3 cases held; the row names production_only, because this build's contract also records a dependency the environment cannot copy for both tools, and that contract finding outranks the routing reading
The OpenAI-style variant with the same bypass	MCP toolbox, openai_chat	Never Supported	get_item reads not_called: that protocol carries no completed tool results, so the agent cannot report a call the world did not receive

Not tested live: voice agents' routing (a voice check places no call, so qualification records no routing for them).
Re-checked 11 Nov on 7b026f9e (image): on the OpenAI-style catalog fixture (get_item sent to a stand-in inside the agent's sandbox), a pairing with no certificate yet was certified on this head. Its routing record has find_items routed (7 calls) and get_item under unobserved as not_called; the platform reads tool routing Partial, "Not yet shown reaching the test world: get_item (asked, but not called)". So this head does not claim the stand-in tool reaches the world. The Supported reading seen earlier comes from the same agent's certificate recorded on 8 Nov, before this head; a Re-check of an already certified pairing does not certify again.
Not tested live: the request budget (test_requests_past_the_limit_are_recorded_as_not_asked, test_no_request_is_sent_once_the_time_is_spent) and clearing the record per attempt (test_a_record_from_an_earlier_attempt_never_speaks_for_a_later_failure): unit-tested only.

Re-checked live 13 Nov with the review fixes (36eaa36f, 88e7204c, ed60aed3) on the same local stack and production sandbox specs: a new candidate was validated (certified) and saved for two chat environments, and the platform read the routing row from the new certificate.

Case	Kind	Quality check	Result
Multi-agent returns app, four tools in the world; three name read or create effects, update_order_status names an update	HTTP agent, world tools	A tool whose contract names effects is never asked	update_order_status under unobserved with has_effects; get_order (2 calls), list_refunds (5) and issue_refund (1) routed from the qualification conversation itself; the platform row reads Partial, "its contract says it changes something, so the check did not call it"
Deals desk app, five tools behind its own tools service; three leave the test world	HTTP agent, service tools	Tools that leave the world are never asked; the rest reach it	apply_discount, post_message, request_discount under unobserved with reaches_outside; get_deal (1 call) and read_channel (5) routed
Agent returns a tool other than the one asked during a request (88e7204c)	either	Refused and listed under refused_during_requests	Not forced live: neither agent answered a request with a different tool, so both records have no refused_during_requests; unit-tested (test_a_request_runs_only_the_asked_tool_at_the_world)

Verified against code on 7b026f9, 10 Nov

5. Commands to run the tests

PYTHONPATH=src pytest tests/harness/test_tool_routing_observations.py tests/harness/test_chat_readiness.py \
  tests/harness/test_mcp_toolbox_adapter.py tests/harness/test_process_runtime.py \
  tests/harness/test_failure_record.py tests/harness/test_hosted_agent_reading.py

Running test_authoring_runtime_validation.py, test_workflow_intent.py, test_chat_readiness.py and test_hosted_authoring_entrypoint.py in one session showed 8 failures and then hung in a local run, as it did on clean dev; each file passed on its own. Not re-checked on this head.

6. Scope

Not in this PR: a world-side record for tools that run inside the agent's own process. The world sees their data but no call, so they stay "no world record".

7. Certificates record what made them (harness version, frameworks, judge)

The platform needs to tell a certificate made by an earlier harness, or with another framework version, from a current one, and to know whether the judge answered during the check, so judged cases are not run when it cannot grade them.

File	Change	Preserved
harness/certification.py	CertificationRuntime.harness_version, .frameworks and CertificationChecks.judge_reachable, serialized only when set	A certificate made before them keeps its fingerprint and format
harness/authoring_runtime_validation.py	_write_generic_certificate(..., source) fills the harness version (fi.alk.__version__, none for a 0.0.0+ dev build) and the agent's frameworks; _with_judge_check runs judge.judge_reachable() only when the catalogue has a judged sub-goal. The job's repository is read into its own name (origin), so the agent's source reaches the framework reader	Every other certificate field
harness/generated_runtime.py	installed_frameworks(source) reads the agent's .venv when the source has one. Otherwise it reads the python3 the agent is installed into, counting only the frameworks the agent's dependency files name (requirements*.txt, pyproject.toml, setup.py, setup.cfg, Pipfile): that interpreter also holds the harness's own dependencies. opentelemetry-api is a span marker, not a framework
harness/judge.py	judge_reachable(): one 5-token request to the judge's model through the gateway, 20 s, passed or failed; any error reads failed	How the judge grades

Behavior callouts:

• Certificates gain runtime.harness_version, runtime.frameworks and, with judged sub-goals only, checks.judge_reachable. One extra 5-token model call per check with judged sub-goals.
• An agent that declares no framework records none, rather than the harness's own (claude-agent-sdk, google-adk), which would make every harness dependency bump read as the agent's framework change.

Tests (each mutation-checked: dropping the harness version, recording unset fields, running the judge check without judged sub-goals, passing without a gateway, counting opentelemetry-api, recording undeclared interpreter frameworks: all fail the test):

Test	Pins
test_certification.py::test_old_certificate_keeps_its_fingerprint	Unset fields are left out, so old fingerprints hold
::test_certificate_records_harness_and_frameworks	The fields round-trip
::test_judge_check_runs_only_with_judged_sub_goals (2)	The judge call runs only when a sub-goal is judged
::test_judge_unreachable_without_a_gateway	No gateway reads failed
::test_frameworks_are_read_from_the_agents_environment	Read from the agent's .venv, opentelemetry-api left out
::test_the_interpreter_counts_only_the_frameworks_the_agent_declares	The interpreter's harness dependencies are not the agent's
test_authoring_runtime_validation.py::test_generic_certificate_combines_runtime_evidence_and_repair_history (2)	The written certificate carries the job's repository, the agent's frameworks and the harness version

Break-it pass (throwaway tests): a broken dist-info, a source that is a file, python3 returning a list or timing out, an old certificate JSON without the fields, an invalid judge_reachable value: all held.

Live, on the production-spec local stack: a Validate of a reviewed change (validation job f5f21e1e) wrote runtime: {"harness_version": "0.3.7", ...} and checks.judge_reachable: "passed"; the platform then read Stale "Checked with harness 0.3.7; runs now use 0.3.8" once a newer version was recorded, and a Re-check certified the pairing again (de67f195). That live run also showed the interpreter fallback recording the harness's own claude-agent-sdk and google-adk for an agent that uses neither, which the dependency-file rule above fixes.

PYTHONPATH=src pytest tests/harness/test_certification.py tests/harness/test_authoring_runtime_validation.py tests/harness/test_certify_only.py

Tickets and plan

• Linear: Part of TH-8874, the harness side; the platform reads these records in future-agi #3784 (stacked on #3755), which completes the ticket.
• Product plan shared on 1 Nov: T06 "Separate start/message, tool binding, reset, cancellation, evidence and grading", for tool binding: each tool's routing observed in the test world, and a failure named by its component so a failure on our side is not the agent's.
• Built for: TH-8874 capability 7 "Controlled tool routing" from a passed check rather than a declaration, and requirement 1 (a framework name alone never makes a capability Supported).
• Considered: routing is read only from what the test world received, never from what the agent reports; a tool the representative scenario did not call gets one request only when its contract names no effects, whatever the world logs; one structured failure record (domain, stage, code, a redacted line) replaces parsing log text; an unread agent version's tools are read only on an explicit Re-check.

Part of TH-8874

[hadarishav]

Requesting changes on three safety/correctness issues:

1. Synthetic readiness probing can invoke effectful tools whenever the world logs them. Logging makes an action observable, not safe. Please skip every effectful tool during synthetic probing and record why it was not asked.

2. The five-minute budget is not enforced around the whole request. target.say() can perform up to eight wrapper calls, each receiving a fresh timeout, so one tool request can greatly exceed the shared deadline. Wrap the entire target.say() operation in one timeout based on the remaining budget and add a regression test.

3. /work/failure.json shares a writable OS boundary with the tested agent, while Platform #3784 uses its domain for run gating. Until the writer is isolated or the record is authenticated, this file must remain diagnostic-only.

Please also coordinate the not_observable reason with Platform #3784's reason table.

The harness-side evidence flow is otherwise coherent, but effectful probes are a release blocker.

[KarthikAvinashFI]

Round 1 fixes, head ed60aed3f:

1. A tool whose contract names effects is never asked in probing and is recorded as has_effects (36eaa36f). Test: test_a_tool_with_effects_is_never_asked_even_when_the_world_logs_its_calls.
2. One timeout from the remaining budget wraps the whole target.say() (36eaa36f). Test: test_one_request_stays_within_the_budget_however_many_times_the_agent_is_called.
3. /work/failure.json is diagnostic only, nothing in ALK reads domain, and the failure_record docstring says so (36eaa36f, ed60aed3).
   The reason strings are listed in the body; #3784 maps has_effects.

[hadarishav]

The round-one fixes are present: effectful tools are skipped, unexpected returned tools are refused, and Platform treats failure.json as diagnostic only. Three correctness issues remain.

1. Service endpoints sharing a final segment produce false routing results.

chat_readiness._tool_names() maps service calls by the endpoint's final path segment and keeps the first mapping. The proxy logs both /users/search and /orders/search as search. I reproduced a real find_orders call credited to find_users, with find_orders marked not_received.

Please preserve a canonical tool identity or full endpoint in proxy records. Treat ambiguous older aliases as unobserved instead of guessing.

2. The shared timeout does not stop synchronous tool handlers.

chat_readiness.py:480 wraps target.say() in asyncio.timeout(), but chat_call_runner.py:413 calls world.handle_tool_call() synchronously on the event loop. A real GeneratedWorld handler taking about 200 ms completed successfully under a 10 ms deadline and recorded routed=true.

Please enforce the remaining deadline at a supervised tool-execution boundary that can terminate blocking work. Add a blocking-handler regression; the existing slow-async-wrapper test does not cover this path.

3. A repaired candidate can reuse the previous candidate's routing proof.

tool-requests.json is keyed only by tool name and survives material repair attempts in the shared authoring directory. I reproduced a first candidate using /lookup, then a repaired candidate using /changed/lookup; the second sent no request and inherited the first candidate's routed=true.

Please bind reusable evidence to the candidate or bundle fingerprint and tool definition. Keep the shared request/time budget separate. If the candidate changes after the budget is exhausted, report the tool unobserved rather than reuse stale proof.

Verification on this head covered 144 focused cases. One PostgreSQL integration case was skipped and one SQL-adaptation test lacked psycopg. GitHub workflows are green, but these three cases need regressions before approval.