Skip to content

feat(harness): worker permissions and deadlines at the world's door (TH-8887, TH-8888) - #265

Open
KarthikAvinashFI wants to merge 57 commits into
feat/TH-8875-actorsfrom
feat/TH-8887-worker-permissions-and-timeouts
Open

KarthikAvinashFI wants to merge 57 commits into
feat/TH-8875-actorsfrom
feat/TH-8887-worker-permissions-and-timeouts

Conversation

@KarthikAvinashFI

@KarthikAvinashFI KarthikAvinashFI commented Nov 9, 2026 •

Copy link
Copy Markdown
Member

Why

The plan's "Show worker actions, permissions and failure outcomes" asks that a refusal, a permission failure and one timed-out worker are visible, with partial evidence and the coordinator's response after the timeout kept. The participants work records which worker did what; this adds what each worker may do and a worker-action deadline, applied at the world door, so a worker's call outside its allow-list or past its deadline is refused and recorded as that worker's. The deadline refuses the worker's later world calls; it does not stop the worker's own execution. Workers are told apart by their channel, which is cooperative attribution, not isolation (see Review round 1).

Built inside the actors' world door (world_door.py), so workers and actors share one refusal path and one action log.

Base: this branch is agent-learning-kit #257 (actors, feat/TH-8875-actors) with #226 (participants, feat/TH-8886-declared-workers) merged in. Merge order: #226 and #257 first, then this. The diff against #257 includes #226's changes until #226 lands; this PR's own changes are the non-merge commits after merge 18f75713.

What changed

Area Files What
The door world_door.py A call carries the worker it came from (ToolCall.worker). For the agent's call on a worker's channel the door refuses, before the world: a channel no worker of this trial owns (identity_conflict), a call past the trial's worker deadline (deadline_passed), an action outside the worker's sealed allow-list (not_permitted). Limits that cannot be read refuse. Every row in the action log carries worker_id
Tool proxy tool_trace_proxy.py Reads /w/<token>/..., strips the token (never logged or forwarded), resolves it to the worker; passes the sealed allow-lists (permissions.json beside the proxy) and the trial's deadline to the door
Trial state call_runner.py, chat_call_runner.py The runner writes the worker deadline beside the trial's faults in the world database (_alk_trial_faults.worker_deadline_at) at the trial's start and clears it after; a budget per worker goes in _alk_trial_faults.worker_deadlines. worker_deadline() / worker_deadlines() are start plus the scenario's worker_budget_seconds, the only worker setting a scenario carries
Sealing bundle_author_v2.py, contract.py, process_runtime.py, process_preflight.py, tools.py, skills Each declared worker's allow-list comes from the agent contract, never from a scenario: at build, _worker_permissions seals the worker's allowed_tools (or all its tools when that is empty), each as its route's method and path, into permissions.json. A worker wired through its own code (binding: separate, address_env) gets its own channel token per world ({{WORKER_CHANNEL_<id>}})
Grading scenario_source.py, scenario_tools.py, hosted_scheduler.py, hosted_entrypoint.py, outbound.py, catalogue.py, write-scenarios skill A sub-goal can name one worker as its subject and is graded on that worker's calls; with none recorded it is inconclusive. A worker's verdicts are reported beside the task's, never folded into it. A worker past its action deadline has its own checks inconclusive
State changes call_runner.py Each worker the world told apart is its own entry in the trial's state changes, with its refused attempts and why. A change one of the agent's workers committed is not "no change"
Reference app examples/harness/returns_reference/ The returns coordinator's workers reach the ledger through their own clients, with the duplicate-refund case

Behaviour callouts

  • An agent without declared workers seals and runs exactly as before: no permissions.json, no channels, no deadline.
  • Workers whose tools run through code the agents share (binding: shared) are not told apart at the world, so they cannot be limited there: permitted tools on a shared worker, or a deadline naming one, go back to authoring, and on a bundle that still has them the trial is held with worker_limits_not_enforced rather than graded. Their calls are attributed by the participants record as before.
  • Limits are cooperative attribution, not isolation: every worker runs in the agent's one process, so a separate worker is held to its tools only when its calls use its own channel.
  • A broken or unreadable seal refuses the worker's call; it never widens what a worker may do.
  • Where each limit comes from: what a worker may do is the contract's declared workers (tools / allowed_tools), sealed at build; a scenario sets only when workers must stop (worker_budget_seconds). The scenario writer and the builder chat set it through submit_scenario / edit_scenario, whose field description says how; a budget naming a worker the contract does not declare is refused (76d32d31). The write-scenarios skill says a budget names only separate workers. A budget naming a binding: shared worker goes back to authoring, since that worker has no channel at the world.
  • worker_budget_seconds is a number (one deadline for all workers) or {worker id: seconds} (only those workers, so one worker can pass its deadline while the others finish), counted from the trial's start, not from a hand-off. The tool description, the field comment and the code say the same.
  • Routes and worker allow-lists are sealed by method and path (GET orders.{order_id}): two tools on one path are two actions, and a path that is one of the world's own literal routes (orders.update_status) is that route's action, never a value of a template.
  • Environments sealed before this release keep their sealed proxy and door until they are rebuilt or re-sealed. Those proxies have no worker handling at all, so worker allow-lists and deadlines exist only in bundles sealed by this release and an older environment has no worker limits for this to weaken. A new seal records what its door enforces (generated/tool-proxy/door.json). A scenario that relies on worker limits (a worker budget, or an agent that declares a separate worker or one allowed fewer tools than it holds) on a proxy sealed before then is held, not graded: the trial says so and asks for a rebuild (notice worker_limits_not_enforced), and the scheduler records the scenario as a scenario problem, not the agent's, with every check unmeasured. A chat agent whose tools the harness runs itself (no tool proxy) is held the same way.
  • What the worker route guard covers. A worker's call reaches only routes on its sealed allow-list, matched by method and path; a path with an encoded separator (%2F, %5C, %2E) or a .. segment is refused, since the service would decode it into a different route than the one checked. A templated entry (GET orders.{order_id}) never matches another tool's sealed route, but the proxy knows only the contract's tool routes: a route the service serves that is not one of the agent's tools (for example /orders/admin_dump) at the same depth matches the template. Declare such routes as tools to keep them apart.
  • Methods through the proxy. The tool proxy forwards GET and POST only (unchanged from dev); other methods get 501 before the door, so the DELETE refusal row below is door-unit level.
  • A deadline the runner cannot write (no world database) fails the trial; one the proxy cannot read refuses workers' calls. Neither lets workers act with no deadline.
  • An allow-list entry for a templated route (orders/{order_id}) matches one value per {...} segment; the query is never part of the action name.
  • A worker is known only by the channel its own code calls. A worker that calls the agent's shared tool address instead is the agent's call there, so it is not limited; the readiness of that wiring comes from the contract (binding, address_env).

Tests

  • New tests in this PR's commits: 63. The door rules, the proxy channel and the deadline each have a test that fails with the rule removed (checked by removing each rule in turn).
  • The proxy tests run against a real PostgreSQL (Docker): a worker's allowed call reaches the world and names the worker with the token stripped; outside its allow-list it is refused before the world; an unknown channel is refused; past the deadline written by the runner every worker call is refused and the agent's own call is not.
  • Wider suites on 7b2846aa (before the 10 Nov fixes): 1,958 passed. One failure, test_grade.py::test_the_cap_holds_for_the_session_that_saves_too (NameError: st), fails the same way on both base branches.
uv run --frozen python -m pytest -q tests/harness/test_world_door.py tests/harness/test_tool_trace_proxy.py tests/harness/test_call_runner.py tests/harness/test_hosted_scheduler.py tests/harness/test_bundle_author_v2.py tests/harness/test_process_runtime.py

Tested on

Situation How tested Result
Worker's allowed action, through its channel proxy + real PostgreSQL passed
Worker's action outside its allow-list door unit + proxy + real PostgreSQL refused not_permitted, world not reached
Channel no worker owns door unit + proxy refused identity_conflict
Past the worker deadline door unit + proxy, deadline written by the runner refused deadline_passed; the agent's own call still commits
Unreadable limits door unit refused
Templated route /orders/{order_id} and a queried route allow-list + door unit real calls permitted; other paths refused
A templated allowance against another tool's literal route or another method door unit + sealing unit, mutation-checked POST orders.update_status and GET orders.cancel refused for a worker allowed GET orders.{order_id}; DELETE orders.1043 refused; GET orders.1043 permitted
A budget for one worker only runner + proxy over real PostgreSQL the worker past its own deadline is refused deadline_passed while another worker's call commits
Worker budgets on a proxy sealed before worker limits runner unit + sealing unit, mutation-checked notice worker_limits_not_enforced for one number or per worker; a new seal writes door.json
Deadline cannot be written / read runner + proxy unit trial fails / worker calls refused
Worker subject with a space contract test against the platform's pattern accepted on both sides
Worker's refused attempt in state changes runner unit own entry with code and reason
Worker permission refusal (returns reference, lookup_tries_to_write) live chat, local stack, production sandbox specs passed: the order-lookup worker's update_order_status was refused not_permitted before the world; World changes shows it under worker:order_lookup; the worker-subject sub-goal held and the agent told the caller the update was not permitted. Run 7a7b9742, call 11fd9fa7
Duplicate refund (duplicate_refund) live chat passed: no second refund; the coordinator reported the existing one; graded under the agent's own tool names. Run 7a7b9742, call d0322113
Worker deadline (refund_worker_runs_out_of_time, budget 3 s) live chat the world refused the worker's calls deadline_passed, kept as its refused attempts; the worker ended past its deadline (deadline_passed); the agent said truthfully the refund did not go through. One builder-written check failed because it required a successful get_order, which the deadline refused. Run 7a7b9742, call b3f422a0
User cancel during a call live chat, four attempts the first three cancels landed after the call had ended (calls ecffadb1, 03d74d5c, 7806f4fe), and each kept its participants record. The fourth landed mid-trial (run e1d60c08, call ff3c96e4): the call reads cancelled and its participants record is present with no evidence, so partial worker evidence after a user cancel is not shown by this run (see Known gaps)
Worker permission refusal, re-run on the 10 Nov heads live chat, local stack, production sandbox specs (this PR at 96eaa89, future-agi #3791 at 103680d2b) passed: update_order_status refused not_permitted before the world; World changes lists it under "Worker · order_lookup" with why, and the worker's two reads as list_refunds, get_order; the worker check order_lookup_worker_no_writes (order_lookup) held. Run cde8ba8c, call 4f73a9a9
Duplicate refund, refund inside the window, lookup only, worker deadline (10 Nov heads) live chat, same stack duplicate refund passed (call 6d909408), refund inside the window passed (call 44e461e5), lookup only passed (call 5fbc87ae); the deadline trial refused both worker reads deadline_passed (call 550b67ac). Runs cde8ba8c, 93caf76f, 433cd6f2
Failed worker check on a passed scenario, live live chat on this branch at 96eaa89 with future-agi #3791, production sandbox specs The receipt reads passed; order_lookup_worker_marks_return_requested (worker:order_lookup) fails with "not done: update_order_status was not called for order 1043". Run 53eef8ef, call d98af5bc. The platform's call outcome was fixed to leave the worker check out (future-agi #3791)
Worker id with a space, live builder chat on the returns environment A subject naming the worker by its display name (worker:Order-lookup worker) is refused: subjects must name a declared worker id. Declared ids here are slugs, so a spaced id does not arise from a build
Templated route, live Run ee5dfcd3, call 0829bbf8, at 6999294: the refund worker allowed /refunds/{order_id}/issue called /refunds/1043/issue; the world committed it, World changes lists it as issue_refund, and the refund sub-goal (issues_eligible_refund) and the scenario passed Earlier run 46e27e76 failed the refund check: the call was logged as issue (fixed in dbe4e8ab), then run 0cc6fb75 failed it with order_id missing, since the order id travels in the path (fixed in 6999294b)

Verified against code on 2a0b8abc, 10 Nov.

Fixes from the 10 Nov review

Finding Fix Test (fails without the fix)
A worker allowed a route with a path parameter or a query was refused not_permitted on every call: three copies of "route to action" disagreed One world_action rule used by the proxy, the sealed allow-lists and the service routes; templated routes matched per segment (9a4f4cfe) test_a_worker_allowed_a_templated_or_queried_route_is_permitted_its_real_calls
A worker id the harness accepts (worker:Order lookup) was refused by the platform's receipt The platform now takes the same rule (future-agi #3791); a test here pins that every subject the harness accepts matches it (facd8392) test_every_subject_the_harness_accepts_matches_the_platform_s_receipt_rule
A templated route's real call was logged under its last path segment (issue), so a check on the tool (issue_refund) failed though the world committed it Templated routes are sealed into the tool map and the proxy matches them per segment (dbe4e8ab) test_a_templated_route_s_real_call_is_logged_under_its_tool_s_own_name, test_a_templated_route_s_real_call_is_logged_under_the_agent_s_tool_name
A templated route's path values (order_id in /refunds/1043/issue) were missing from the logged arguments, so a check on them read None The proxy adds the values the matched templated route's path carries to the logged arguments; the body's own keys win and the world still gets the body as sent (6999294b) test_a_templated_route_s_path_values_are_logged_with_the_call_s_arguments, test_a_templated_route_names_the_values_its_path_carries
worker_budget_seconds described as per worker before per-worker budgets existed Described as one deadline from the trial's start (360a91de); per-worker budgets were then built (1ae310d1) and the description now covers both shapes test_a_scenario_writer_and_an_editor_can_set_a_worker_budget (schema takes both shapes)
The understand-agent skill pointed at our own reference manifest Sentence removed (360a91de) wording
The deadline failed open when it could not be written or read Fails closed (c0807c53) test_a_worker_deadline_that_cannot_be_written_fails_the_trial_rather_than_run_without_one, test_a_worker_deadline_that_cannot_be_read_refuses_rather_than_reads_as_none

Not run live: the fail-closed deadline (unit level above). To verify: with the world database stopped, a trial with a worker budget fails with the deadline reason.

Templated route, live (run ee5dfcd3): the refund worker's issue_refund on /refunds/1043/issue committed and the scenario passed.

templated route passed

Fixes found by the live runs

Each has a test that fails without it.

  • 55f7c96a: World changes named a worker's calls by the last part of their route (Committed: get, list, refused update_status), unlike the tool names its checks are graded under. A call through the tool proxy is now named by the agent's own tool (get_order, update_order_status); actors' rows are unchanged. Re-run live: call 4f73a9a9.
  • b65c5fd8: the returns reference answered every turn on a new event loop, so with an OpenAI model its second turn failed with "Event loop is closed". It now runs every turn on one loop. This also affects the reference app in feat(harness): declared workers in the contract and a returns reference app #226.
  • 60c52e85: the world's log named a call by the last part of its route (list, get), so checks written with the agent's tool names failed for an app whose routes differ from its tools. The bundle now seals each route's tool, the log keeps it in agent_tool beside the route, and calls are graded under it. The route stays the row's name for what actors read.
  • 7b2846aa: a chat trial with limited workers and no actors never kept the workers' refused attempts in World changes; it now reads the world's table as it does for actors.
  • 70cba0fc (actors code, found while checking the actors fix 7d55ab53): a chat trial's actors watched only the runner's own action log, never the agent's calls through its tool proxy, so an on_tool trigger could not fire for such an agent. They now read the world's table from the trial's start.
Live check on a new seal How Result
One worker past its action deadline while the other finishes, live, on a newly sealed bundle Run e9ebc3c8 (certified bundle assembled on this branch, 10 Nov), call f17fc9d6, on suite version 3, where the builder chat set worker_budget_seconds: {"refund": 2} on refund_worker_runs_out_of_time with edit_scenario (10 Nov) passed: the order-lookup worker's reads committed, the refund worker's call refused deadline_passed, workers read completed / past their action deadline, no refund in the world, the coordinator said the refund could not be processed; both checks passed
Worker permission refusal, live, on the same new seal Run e9ebc3c8, call 8210f1c5 passed: update_order_status refused "orders.update_status is not permitted for order_lookup"; reads committed; the worker check and the task's check passed

Rows above that name earlier runs ran on bundles sealed before these fixes (a default run reuses its environment's sealed bundle); the two rows just above ran on a bundle sealed by this branch before the final-review fixes below (so the undeclared-worker budget refusal, 76d32d31, was not exercised live); not re-run on 2a0b8abc.

Base update, 10 Nov

Merged the latest actors base (feat/TH-8875-actors at df07acfc) at a53b1c74, with no conflicts. Re-run on the merged head: 425 tests across the world door, actor runtime, scheduler, chat worker, grading, multi-agent references, contract and scenario source. One grading test (test_the_cap_holds_for_the_session_that_saves_too) fails the same way on the actors base and on this branch before the merge, so it is not from this PR. The live rows above ran on the earlier head; the two rows on a new seal re-check it.

Merged the latest actors base again (feat/TH-8875-actors at 0fbeb473) at 4ceeefb2, with no conflicts; 282 + 272 focused tests pass, including test_bundle_compiles_discovered_source_tool_entrypoint, which the actors branch fixed.

Fixes from the final review, 10 Nov

Finding Fix Test (fails without the fix)
A worker's broken check (raised, or returned something not a verdict) faulted the whole call and left the task's later sub-goals ungraded Recorded as unmeasured on that worker, with why; the call is graded (a389e849) test_a_worker_s_broken_check_is_unmeasured_on_that_worker_and_never_faults_the_call
On a proxy sealed before worker limits, a per-worker budget was said to be held to the shortest, though that proxy holds no worker limit, and a single budget went unenforced with nothing said Notice worker_limits_not_enforced for both shapes (127f953a); chat with no tool proxy too (55ed456f) test_worker_budgets_on_a_door_sealed_before_worker_limits_are_said_not_enforced, test_a_chat_agent_without_a_tool_proxy_says_its_workers_budgets_were_not_held
An encoded separator or dots let a templated allowance reach another route; an actor's templated permission could reach another tool's route Refused for workers; actors checked by method and sealed routes (c5784634) test_a_templated_allowance_never_reaches_a_route_hidden_behind_an_encoded_separator_or_dots, test_an_actor_s_templated_permission_never_reaches_another_tool_s_own_route
Any /w/... path was taken as a worker channel, even in a trial with no workers Only when the trial gave workers channels (8febc776) test_an_app_s_own_w_route_runs_as_before_in_a_trial_that_gave_no_worker_channels
A budget naming a worker the contract does not declare silently gave that worker no deadline Refused with the declared workers named (76d32d31) test_a_budget_for_a_worker_the_contract_does_not_declare_is_refused_with_the_declared_ones

Release order

future-agi #3791 is deployed before this PR's template pin: on an older platform a worker's failed check would count toward the scenario's outcome.

Known gaps

  • After a user cancel mid-trial, the cancelled call's participants record was present but showed no worker evidence; whether the workers had acted before the cancel could not be told from that run.

  • Evaluator-only material and workers. At the world, a limited worker reaches only the world actions on its sealed list: a route that is no world action, or a request naming an actor, is refused before the world and recorded as that worker's (test_a_limited_worker_s_request_for_anything_that_is_not_a_world_action_never_reaches_the_world, test_a_worker_naming_an_actor_never_borrows_that_actor_s_permissions, test_a_worker_s_request_for_a_route_that_is_no_world_action_is_refused_before_the_world; each fails with its guard removed). Actors are never given evaluator-only fields or files (actors PR). Not closed here: workers run inside the tested agent's own process, and on the hosted sandbox every process runs as one OS user, so the agent's own code, coordinator or worker, can read files the harness keeps in the sandbox, including the environment's sub_goals.json. Keeping evaluator-only files out of the agent's reach is part of evidence separation (T44 in the product plan), which is planned separately and will come as its own PRs.

  • Per-worker readiness rows. Run readiness does not yet show, per worker, whether its limits are Allowed, Blocked or Not enforced for this agent. Not built; about 1 to 2 days, later.

  • On an environment sealed before this release, a scenario that relies on worker limits is now held rather than graded (Review round 1, point 5), so the run 65403e6a case (no_refund_issued_on_timeout failing although no deadline was enforced) no longer counts against the agent.

Review round 1

Point What changed Commit
1. The "timeout" is only a deadline on later world calls Renamed to a worker-action deadline in the field comment, the scenario tool description and this body; the deadline refuses a worker's later world calls and never claims to stop it. TH-8888's execution timeout is deferred explicitly (Tickets below) c3998cb0
2. Channels are not an identity boundary Described as cooperative attribution everywhere it was claimed (contract field comment, channel wiring docstring, this body): every worker runs in the agent's one process, which can read every channel and the proxy's own address, so per-worker limits hold for cooperating code only c3998cb0
3. A missing permissions.json fails open While the trial gave its workers channels, a missing or unreadable allow-list file is an empty allow-list: every worker call is refused not_permitted before the world. Missing-file regression test_a_worker_s_call_is_refused_when_the_sealed_allow_lists_are_missing_while_channels_were_given c3998cb0
4. Duplicate address_env, ids no placeholder can carry Contract validation returns to authoring: two separate workers on one address variable, a separate worker id outside [A-Za-z0-9_.-], and a separate worker with no address variable. Sealing gives no channel to an ambiguous or unnameable worker, so one worker's calls are never handed to another c3998cb0
5. Old seals run and grade without enforcement A scenario that relies on worker limits (a budget, or a declared separate or narrowed worker, so permission-only scenarios too) on a door sealed before worker limits, or on a chat agent with no tool proxy, is held: the call carries notice worker_limits_not_enforced, and the scheduler records failure worker_limits_not_enforced (scenario-side, not retried) with every check unmeasured instead of grading c3998cb0

Each point has a test that fails with its fix reverted (seen failing for 3, 4 (validation and sealing), 5 (scheduler hold, voice notice carried to the scheduler, permission-only notice on voice and chat)). Base merged first: feat/TH-8875-actors at e521d248 (95512e60, 605c5fd3). Tests: every test file this PR touches, 1,720 passed and 1 failed, test_grade.py::test_the_cap_holds_for_the_session_that_saves_too, which fails the same way on dev (a test body merged into another, using an undefined st). Ruff findings on the changed source files: unchanged from before.

Found by the live run after round 1, fixed in 5ce18a2b: a scenario with a short worker-action deadline failed certification. The readiness probe ran the scenario with its deadline, so every worker call was refused deadline_passed, and the probe reported "Inspect how the agent's tools are bound to the world", so the environment could never be validated or saved. The probe now checks the tools' binding without the scenario's worker-action deadline (a rule of its graded trials), and a worker's call the door refuses for that worker's own limits (not_permitted, deadline_passed) counts as reaching the world, since it came through the worker's bound channel; an agent's own not_permitted and a channel no worker owns (identity_conflict) still fail it. Tests: test_a_readiness_probe_sets_no_worker_deadline_while_a_graded_trial_does and three new cases in test_private_workflow_fails_when_no_tool_call_reached_the_world, each failing with its fix reverted. Live on the local stack with production's sandbox specs: a 2 s worker budget draft now validates (ready) and saves, and its run shows the order-lookup worker as deadline_passed with both calls refused.

Found by the live run after that, fixed in 7d4f8c67: a fault the scenario names by the agent's tool (get_order) was never applied to that call when it came through the tool proxy, a worker's channel included, because the proxy door matched faults by the route's world action only; the trial then said fault_not_applied. The door now also matches the call's sealed tool, and the trial's notice reads the log row's tool. Tests: test_a_fault_naming_the_tool_holds_its_call_through_the_proxy_and_a_worker_s_channel, test_a_fault_the_proxy_applied_under_the_route_s_name_says_nothing, each failing with its fix reverted. Live on the local stack: the order-lookup worker's first get_order was faulted once and its retry committed, with no notice.

Found by the final review, fixed in 84313436: a binding: shared worker's sealed allow-list and budget were held by nothing (no channel), with no notice, so the scenario was graded against the agent. Permitted tools on a shared worker, a deadline naming one, and one deadline for all with no separate worker now go back to authoring, and the trial is held with worker_limits_not_enforced where a bundle still has them. The write-scenarios skill now says a budget names only separate workers and that a deadline scenario's task checks must not require the late worker's calls. Tests fail with each fix reverted.

Not built here: process or OS-user isolation of workers (so evaluator-only material stays unreachable) is planned under T44 and will come as its own PRs; limits for shared-binding and in-process workers stay not enforced until then; TH-8888's execution timeout, cancel evidence per worker and cleanup status move to the follow-up TH-10189 (reduced scope recorded on TH-8888).

Scope

In: per-worker allow-lists and deadlines at the world, worker sub-goals and verdicts, worker entries in state changes, the returns reference.
Out: limiting workers that share their tools' code (they cannot be told apart at the world); workers of agents whose tool calls the harness itself runs in process.

Test cases

The live cases with their outputs (permissions from the contract, a refused write, one worker passing its deadline while the other completes, an environment sealed before worker limits) are in future-agi #3791's Test cases section, runs e9ebc3c8 (fresh seal from this branch) and 65403e6a (older seal).

Tickets and plan

  • Linear: Part of TH-8887 and TH-8888, with future-agi #3791, under TH-8884.
    • TH-8887, delivered with #3791: each separate worker's calls held to its contract's tools at the world door when they come through its own channel (cooperative attribution), an out-of-permission write refused not_permitted before the world and recorded as that worker's attempt (live, call 4f73a9a9); a sub-goal can name one worker and is graded on that worker's calls, reported beside the task's verdict, and reads inconclusive with no evidence for that worker; the duplicate-refund case is in the returns reference and passes live with no second refund (call 6d909408). Left: evaluator-only files can still be read by the agent's own process, coordinator or worker, on the hosted sandbox (evidence separation, T44, not scoped); no run that creates a second refund was shown failing.
    • TH-8888, delivered with #3791: a per-worker deadline inside the trial; a worker past it has its later world calls refused deadline_passed and keeps its earlier calls, while another worker finishes (live, call f17fc9d6). Reduced scope, recorded on TH-8888: the deadline is a worker-action deadline and does not stop the worker's execution; a real execution timeout (which needs process control of the customer's worker), a user cancel recorded per worker with its partial evidence, and a per-worker cleanup status move to the follow-up TH-10189.
  • Product plan shared on 1 Nov: T17 "Test refusal, permission failure and one timed-out worker" and "Keep partial evidence and coordinator response after timeout" (with #3791). T16 "Use an internal test app for CI and acceptance" in part: the returns reference gains its ledger, its workers' own clients and the duplicate-refund case.
  • Built for: TH-8887 "Tool permissions scoped per worker identity in the world's tool binding", "Blocked attempts recorded as attempted violations", "A per-worker check with no evidence for that worker is unmeasured, not passed" and the duplicate-refund scenario; TH-8888, the part delivered: a per-worker action deadline after which the world refuses that worker's calls and keeps its events so far. Not delivered (deferred): ending the worker's execution at the limit.
  • Considered: built inside the actors' one world door rather than a second copy of the rules; a worker's allow-list comes from the agent contract, never from a scenario, and a scenario sets only when workers must stop; limits that cannot be read refuse rather than widen; workers that share one tool client cannot be told apart at the world, so limits on them go back to authoring, and a bundle that still has them holds the trial with a notice.

Part of TH-8887
Part of TH-8888

Head on 12 Nov: 84313436, with the latest actors base merged (feat/TH-8875-actors at 66cc5f44).

…nto feat/TH-8887-worker-permissions-and-timeouts

# Conflicts:
#	src/fi/alk/harness/call_runner.py
#	src/fi/alk/harness/hosted_scheduler.py
…he door, held to a sealed allow-list and the trial's deadline
…side the task's, and chat calls read the world's log where the bundle says
…s checks are inconclusive, and the sealed trace keeps why the world refused a call
@KarthikAvinashFI KarthikAvinashFI self-assigned this Nov 9, 2026
@KarthikAvinashFI
KarthikAvinashFI marked this pull request as ready for review November 9, 2026 12:31
…H-8887-worker-permissions-and-timeouts

# Conflicts:
#	tests/harness/test_actor_plan.py
#	tests/harness/test_chat_call_runner.py
…H-8887-worker-permissions-and-timeouts

# Conflicts:
#	src/fi/alk/harness/call_runner.py
#	src/fi/alk/harness/world/runtime.py
#	tests/harness/test_chat_readiness.py
…nto feat/TH-8887-worker-permissions-and-timeouts
…H-8887-worker-permissions-and-timeouts

# Conflicts:
#	src/fi/alk/harness/call_runner.py
…-lists, templated routes matched per segment
…al's start and drop the reference-manifest line
…H-8887-worker-permissions-and-timeouts

# Conflicts:
#	src/fi/alk/harness/call_runner.py
#	src/fi/alk/harness/chat_call_runner.py
#	src/fi/alk/harness/tool_trace_proxy.py
#	tests/harness/test_actor_runtime.py
@KarthikAvinashFI
KarthikAvinashFI marked this pull request as ready for review November 10, 2026 13:01
@KarthikAvinashFI

Copy link
Copy Markdown
Member Author

f620cd8 brings in the latest actors branch (agent-learning-kit #257: 8870075, 9f8e9a2); nothing else changed in this PR.

@hadarishav hadarishav left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes. The world-door direction is good, but the current implementation does not yet provide the worker isolation or execution timeout claimed by TH-8887 and TH-8888.

  1. The worker "timeout" is only a deadline on later world calls. Nothing stops the worker's execution. A worker can exceed the deadline, make no further call, or catch the refusal and return successfully. Please enforce the deadline around actual worker execution. If that is outside this increment, rename this to a worker-action deadline and defer TH-8888's execution-timeout requirement explicitly.

  2. The channel is not a worker identity boundary. Every worker URL and the unrestricted TOOLS_API_URL are in the same tested process, and worker-channels.json is readable by that process. A worker can reuse another worker's token or call the unrestricted URL. Real enforcement needs process or OS-user isolation. Otherwise this must be described and reported as cooperative attribution / not enforced, not "each worker can only use its tools."

  3. A missing permissions.json fails open. _worker_limits() omits the permissions key when the file disappears, and WorldDoor then skips permission checks. When worker channels exist, a missing or unreadable policy must produce no permissions and refuse every worker call. Please add the missing-file regression.

  4. Separate-worker binding accepts two invalid states. Duplicate address_env values collapse workers onto the last channel, and IDs such as "Order lookup" pass contract validation but generate a WORKER_CHANNEL placeholder the runtime cannot resolve. Reject duplicate separate-worker address_env values and either use opaque placeholder keys or validate the supported ID pattern.

  5. Old seals continue running and grading without worker enforcement. Permission-only scenarios get no warning because the notice is conditional on a worker budget. Budgeted scenarios continue after ALK knows the limit was not enforced, which can turn missing enforcement into an agent failure. Refuse/hold these scenarios until rebuild, or at minimum make dependent checks unmeasured.

Plan gaps also need explicit product-owner alignment in this PR and Linear:

  • TH-8887 requires evaluator-only material to be unreachable, but the tested process can still read it under the shared OS user.
  • Shared-binding workers and in-process tools are not permission-limited.
  • TH-8888's cancellation and cleanup-status requirements are not delivered by this stack.

Focused verification: 68 contract, world-door, and worker-budget tests passed. The larger container-backed command did not complete and was interrupted, not failed.

Please fix 1 through 5 before approval and document any reduced ticket scope explicitly.

…sealed allow-lists, reject ambiguous worker channels, hold scenarios whose worker limits were not enforced
…adline and counts worker-limit refusals as reaching the door
…H-8887-worker-permissions-and-timeouts

# Conflicts:
#	src/fi/alk/harness/chat_call_runner.py
#	tests/harness/test_chat_call_runner.py
…H-8887-worker-permissions-and-timeouts

# Conflicts:
#	src/fi/alk/harness/tool_trace_proxy.py
#	src/fi/alk/harness/world_door.py
@KarthikAvinashFI

Copy link
Copy Markdown
Member Author

On the scope point: the reduced scope is recorded on TH-8888. A real execution timeout, a user cancel shown per worker, and a per-worker cleanup status move to the follow-up TH-10189. Linking LangGraph background workers to their hand-off is TH-9023, coming as a separate PR on top of this one. Neither blocks this review. Isolation of the tested agent from evaluator-only files is planned under T44 and will come as its own PRs; until then the PR states it as not enforced.

@KarthikAvinashFI

Copy link
Copy Markdown
Member Author

Round 1 fixes, head d46d591cc:

  1. Renamed to a worker-action deadline that only refuses later world calls (c3998cb0).
  2. Channels are described as cooperative attribution, not isolation, in code and body (c3998cb0, 84313436).
  3. A missing or unreadable allow-list refuses every worker call (c3998cb0). Test: test_a_worker_s_call_is_refused_when_the_sealed_allow_lists_are_missing_while_channels_were_given.
  4. Duplicate address_env and ids no channel can carry go back to authoring (c3998cb0). Test: test_separate_workers_sharing_an_address_or_with_an_id_no_channel_can_carry_are_returned_to_authoring.
  5. Old seals and limits a shared worker cannot be held to hold the trial as worker_limits_not_enforced, never graded (c3998cb0, 84313436). Tests: test_a_trial_that_could_not_hold_the_worker_limits_its_scenario_relies_on_is_held_not_graded, test_a_permission_only_scenario_on_a_door_sealed_before_worker_limits_is_said_not_enforced.
    Scope: see the comment above.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants