Repository navigation
feat(harness): worker permissions and deadlines at the world's door (TH-8887, TH-8888) - #265
KarthikAvinashFI wants to merge 57 commits into
Conversation
…nto feat/TH-8887-worker-permissions-and-timeouts # Conflicts: # src/fi/alk/harness/call_runner.py # src/fi/alk/harness/hosted_scheduler.py
…he door, held to a sealed allow-list and the trial's deadline
…n at the tool proxy, per world
…rs are sealed with their own allow-lists
…side the task's, and chat calls read the world's log where the bundle says
…in the trial's state changes
…s checks are inconclusive, and the sealed trace keeps why the world refused a call
…H-8887-worker-permissions-and-timeouts
… so an on_tool trigger sees the agent's calls through its tool proxy
…the route, and calls are graded under it
…in its world record
…H-8887-worker-permissions-and-timeouts # Conflicts: # tests/harness/test_actor_plan.py # tests/harness/test_chat_call_runner.py
…H-8887-worker-permissions-and-timeouts # Conflicts: # src/fi/alk/harness/call_runner.py # src/fi/alk/harness/world/runtime.py # tests/harness/test_chat_readiness.py
…nto feat/TH-8887-worker-permissions-and-timeouts
…H-8887-worker-permissions-and-timeouts # Conflicts: # src/fi/alk/harness/call_runner.py
…-lists, templated routes matched per segment
…al's start and drop the reference-manifest line
…latform's receipt rule
…H-8887-worker-permissions-and-timeouts # Conflicts: # src/fi/alk/harness/call_runner.py # src/fi/alk/harness/chat_call_runner.py # src/fi/alk/harness/tool_trace_proxy.py # tests/harness/test_actor_runtime.py
…H-8887-worker-permissions-and-timeouts
…nto feat/TH-8887-worker-permissions-and-timeouts
…its are said not enforced
…rkers' budgets were not held
…efused, and an actor's templated permission never reaches another tool's route
…ve no worker channels
…H-8887-worker-permissions-and-timeouts
…H-8887-worker-permissions-and-timeouts
…H-8887-worker-permissions-and-timeouts
hadarishav
left a comment
There was a problem hiding this comment.
Requesting changes. The world-door direction is good, but the current implementation does not yet provide the worker isolation or execution timeout claimed by TH-8887 and TH-8888.
-
The worker "timeout" is only a deadline on later world calls. Nothing stops the worker's execution. A worker can exceed the deadline, make no further call, or catch the refusal and return successfully. Please enforce the deadline around actual worker execution. If that is outside this increment, rename this to a worker-action deadline and defer TH-8888's execution-timeout requirement explicitly.
-
The channel is not a worker identity boundary. Every worker URL and the unrestricted TOOLS_API_URL are in the same tested process, and worker-channels.json is readable by that process. A worker can reuse another worker's token or call the unrestricted URL. Real enforcement needs process or OS-user isolation. Otherwise this must be described and reported as cooperative attribution / not enforced, not "each worker can only use its tools."
-
A missing permissions.json fails open. _worker_limits() omits the permissions key when the file disappears, and WorldDoor then skips permission checks. When worker channels exist, a missing or unreadable policy must produce no permissions and refuse every worker call. Please add the missing-file regression.
-
Separate-worker binding accepts two invalid states. Duplicate address_env values collapse workers onto the last channel, and IDs such as "Order lookup" pass contract validation but generate a WORKER_CHANNEL placeholder the runtime cannot resolve. Reject duplicate separate-worker address_env values and either use opaque placeholder keys or validate the supported ID pattern.
-
Old seals continue running and grading without worker enforcement. Permission-only scenarios get no warning because the notice is conditional on a worker budget. Budgeted scenarios continue after ALK knows the limit was not enforced, which can turn missing enforcement into an agent failure. Refuse/hold these scenarios until rebuild, or at minimum make dependent checks unmeasured.
Plan gaps also need explicit product-owner alignment in this PR and Linear:
- TH-8887 requires evaluator-only material to be unreachable, but the tested process can still read it under the shared OS user.
- Shared-binding workers and in-process tools are not permission-limited.
- TH-8888's cancellation and cleanup-status requirements are not delivered by this stack.
Focused verification: 68 contract, world-door, and worker-budget tests passed. The larger container-backed command did not complete and was interrupted, not failed.
Please fix 1 through 5 before approval and document any reduced ticket scope explicitly.
…sealed allow-lists, reject ambiguous worker channels, hold scenarios whose worker limits were not enforced
…adline and counts worker-limit refusals as reaching the door
…H-8887-worker-permissions-and-timeouts # Conflicts: # src/fi/alk/harness/chat_call_runner.py # tests/harness/test_chat_call_runner.py
…e proxy and a worker's channel
…H-8887-worker-permissions-and-timeouts # Conflicts: # src/fi/alk/harness/tool_trace_proxy.py # src/fi/alk/harness/world_door.py
…horing and hold the trial
…ived list like the real one
…holds with the readiness records branch
|
On the scope point: the reduced scope is recorded on TH-8888. A real execution timeout, a user cancel shown per worker, and a per-worker cleanup status move to the follow-up TH-10189. Linking LangGraph background workers to their hand-off is TH-9023, coming as a separate PR on top of this one. Neither blocks this review. Isolation of the tested agent from evaluator-only files is planned under T44 and will come as its own PRs; until then the PR states it as not enforced. |
|
Round 1 fixes, head
|
Why
The plan's "Show worker actions, permissions and failure outcomes" asks that a refusal, a permission failure and one timed-out worker are visible, with partial evidence and the coordinator's response after the timeout kept. The participants work records which worker did what; this adds what each worker may do and a worker-action deadline, applied at the world door, so a worker's call outside its allow-list or past its deadline is refused and recorded as that worker's. The deadline refuses the worker's later world calls; it does not stop the worker's own execution. Workers are told apart by their channel, which is cooperative attribution, not isolation (see Review round 1).
Built inside the actors' world door (
world_door.py), so workers and actors share one refusal path and one action log.Base: this branch is agent-learning-kit #257 (actors,
feat/TH-8875-actors) with #226 (participants,feat/TH-8886-declared-workers) merged in. Merge order: #226 and #257 first, then this. The diff against #257 includes #226's changes until #226 lands; this PR's own changes are the non-merge commits after merge18f75713.What changed
world_door.pyToolCall.worker). For the agent's call on a worker's channel the door refuses, before the world: a channel no worker of this trial owns (identity_conflict), a call past the trial's worker deadline (deadline_passed), an action outside the worker's sealed allow-list (not_permitted). Limits that cannot be read refuse. Every row in the action log carriesworker_idtool_trace_proxy.py/w/<token>/..., strips the token (never logged or forwarded), resolves it to the worker; passes the sealed allow-lists (permissions.jsonbeside the proxy) and the trial's deadline to the doorcall_runner.py,chat_call_runner.py_alk_trial_faults.worker_deadline_at) at the trial's start and clears it after; a budget per worker goes in_alk_trial_faults.worker_deadlines.worker_deadline()/worker_deadlines()are start plus the scenario'sworker_budget_seconds, the only worker setting a scenario carriesbundle_author_v2.py,contract.py,process_runtime.py,process_preflight.py,tools.py, skills_worker_permissionsseals the worker'sallowed_tools(or all itstoolswhen that is empty), each as its route's method and path, intopermissions.json. A worker wired through its own code (binding: separate,address_env) gets its own channel token per world ({{WORKER_CHANNEL_<id>}})scenario_source.py,scenario_tools.py,hosted_scheduler.py,hosted_entrypoint.py,outbound.py,catalogue.py, write-scenarios skillcall_runner.pyexamples/harness/returns_reference/Behaviour callouts
permissions.json, no channels, no deadline.binding: shared) are not told apart at the world, so they cannot be limited there: permitted tools on a shared worker, or a deadline naming one, go back to authoring, and on a bundle that still has them the trial is held withworker_limits_not_enforcedrather than graded. Their calls are attributed by the participants record as before.tools/allowed_tools), sealed at build; a scenario sets only when workers must stop (worker_budget_seconds). The scenario writer and the builder chat set it throughsubmit_scenario/edit_scenario, whose field description says how; a budget naming a worker the contract does not declare is refused (76d32d31). The write-scenarios skill says a budget names only separate workers. A budget naming abinding: sharedworker goes back to authoring, since that worker has no channel at the world.worker_budget_secondsis a number (one deadline for all workers) or{worker id: seconds}(only those workers, so one worker can pass its deadline while the others finish), counted from the trial's start, not from a hand-off. The tool description, the field comment and the code say the same.GET orders.{order_id}): two tools on one path are two actions, and a path that is one of the world's own literal routes (orders.update_status) is that route's action, never a value of a template.generated/tool-proxy/door.json). A scenario that relies on worker limits (a worker budget, or an agent that declares a separate worker or one allowed fewer tools than it holds) on a proxy sealed before then is held, not graded: the trial says so and asks for a rebuild (noticeworker_limits_not_enforced), and the scheduler records the scenario as a scenario problem, not the agent's, with every check unmeasured. A chat agent whose tools the harness runs itself (no tool proxy) is held the same way.%2F,%5C,%2E) or a..segment is refused, since the service would decode it into a different route than the one checked. A templated entry (GET orders.{order_id}) never matches another tool's sealed route, but the proxy knows only the contract's tool routes: a route the service serves that is not one of the agent's tools (for example/orders/admin_dump) at the same depth matches the template. Declare such routes as tools to keep them apart.DELETErefusal row below is door-unit level.orders/{order_id}) matches one value per{...}segment; the query is never part of the action name.binding,address_env).Tests
7b2846aa(before the 10 Nov fixes): 1,958 passed. One failure,test_grade.py::test_the_cap_holds_for_the_session_that_saves_too(NameError: st), fails the same way on both base branches.Tested on
not_permitted, world not reachedidentity_conflictdeadline_passed; the agent's own call still commits/orders/{order_id}and a queried routePOST orders.update_statusandGET orders.cancelrefused for a worker allowedGET orders.{order_id};DELETE orders.1043refused;GET orders.1043permitteddeadline_passedwhile another worker's call commitsworker_limits_not_enforcedfor one number or per worker; a new seal writesdoor.jsonupdate_order_statuswas refusednot_permittedbefore the world; World changes shows it underworker:order_lookup; the worker-subject sub-goal held and the agent told the caller the update was not permitted. Run 7a7b9742, call 11fd9fa7deadline_passed, kept as its refused attempts; the worker ended past its deadline (deadline_passed); the agent said truthfully the refund did not go through. One builder-written check failed because it required a successfulget_order, which the deadline refused. Run 7a7b9742, call b3f422a0update_order_statusrefusednot_permittedbefore the world; World changes lists it under "Worker · order_lookup" with why, and the worker's two reads aslist_refunds, get_order; the worker checkorder_lookup_worker_no_writes (order_lookup)held. Run cde8ba8c, call 4f73a9a9deadline_passed(call 550b67ac). Runs cde8ba8c, 93caf76f, 433cd6f2order_lookup_worker_marks_return_requested(worker:order_lookup) fails with "not done: update_order_status was not called for order 1043". Run 53eef8ef, call d98af5bc. The platform's call outcome was fixed to leave the worker check out (future-agi #3791)worker:Order-lookup worker) is refused: subjects must name a declared worker id. Declared ids here are slugs, so a spaced id does not arise from a build/refunds/{order_id}/issuecalled/refunds/1043/issue; the world committed it, World changes lists it asissue_refund, and the refund sub-goal (issues_eligible_refund) and the scenario passedissue(fixed indbe4e8ab), then run 0cc6fb75 failed it withorder_idmissing, since the order id travels in the path (fixed in6999294b)Verified against code on
2a0b8abc, 10 Nov.Fixes from the 10 Nov review
not_permittedon every call: three copies of "route to action" disagreedworld_actionrule used by the proxy, the sealed allow-lists and the service routes; templated routes matched per segment (9a4f4cfe)test_a_worker_allowed_a_templated_or_queried_route_is_permitted_its_real_callsworker:Order lookup) was refused by the platform's receiptfacd8392)test_every_subject_the_harness_accepts_matches_the_platform_s_receipt_ruleissue), so a check on the tool (issue_refund) failed though the world committed itdbe4e8ab)test_a_templated_route_s_real_call_is_logged_under_its_tool_s_own_name,test_a_templated_route_s_real_call_is_logged_under_the_agent_s_tool_nameorder_idin/refunds/1043/issue) were missing from the logged arguments, so a check on them read None6999294b)test_a_templated_route_s_path_values_are_logged_with_the_call_s_arguments,test_a_templated_route_names_the_values_its_path_carriesworker_budget_secondsdescribed as per worker before per-worker budgets existed360a91de); per-worker budgets were then built (1ae310d1) and the description now covers both shapestest_a_scenario_writer_and_an_editor_can_set_a_worker_budget(schema takes both shapes)360a91de)c0807c53)test_a_worker_deadline_that_cannot_be_written_fails_the_trial_rather_than_run_without_one,test_a_worker_deadline_that_cannot_be_read_refuses_rather_than_reads_as_noneNot run live: the fail-closed deadline (unit level above). To verify: with the world database stopped, a trial with a worker budget fails with the deadline reason.
Templated route, live (run ee5dfcd3): the refund worker's
issue_refundon/refunds/1043/issuecommitted and the scenario passed.Fixes found by the live runs
Each has a test that fails without it.
55f7c96a: World changes named a worker's calls by the last part of their route (Committed: get, list, refusedupdate_status), unlike the tool names its checks are graded under. A call through the tool proxy is now named by the agent's own tool (get_order,update_order_status); actors' rows are unchanged. Re-run live: call 4f73a9a9.b65c5fd8: the returns reference answered every turn on a new event loop, so with an OpenAI model its second turn failed with "Event loop is closed". It now runs every turn on one loop. This also affects the reference app in feat(harness): declared workers in the contract and a returns reference app #226.60c52e85: the world's log named a call by the last part of its route (list,get), so checks written with the agent's tool names failed for an app whose routes differ from its tools. The bundle now seals each route's tool, the log keeps it inagent_toolbeside the route, and calls are graded under it. The route stays the row's name for what actors read.7b2846aa: a chat trial with limited workers and no actors never kept the workers' refused attempts in World changes; it now reads the world's table as it does for actors.70cba0fc(actors code, found while checking the actors fix7d55ab53): a chat trial's actors watched only the runner's own action log, never the agent's calls through its tool proxy, so an on_tool trigger could not fire for such an agent. They now read the world's table from the trial's start.worker_budget_seconds: {"refund": 2}on refund_worker_runs_out_of_time withedit_scenario(10 Nov)deadline_passed, workers read completed / past their action deadline, no refund in the world, the coordinator said the refund could not be processed; both checks passedupdate_order_statusrefused "orders.update_status is not permitted for order_lookup"; reads committed; the worker check and the task's check passedRows above that name earlier runs ran on bundles sealed before these fixes (a default run reuses its environment's sealed bundle); the two rows just above ran on a bundle sealed by this branch before the final-review fixes below (so the undeclared-worker budget refusal,
76d32d31, was not exercised live); not re-run on2a0b8abc.Base update, 10 Nov
Merged the latest actors base (
feat/TH-8875-actorsatdf07acfc) ata53b1c74, with no conflicts. Re-run on the merged head: 425 tests across the world door, actor runtime, scheduler, chat worker, grading, multi-agent references, contract and scenario source. One grading test (test_the_cap_holds_for_the_session_that_saves_too) fails the same way on the actors base and on this branch before the merge, so it is not from this PR. The live rows above ran on the earlier head; the two rows on a new seal re-check it.Merged the latest actors base again (
feat/TH-8875-actorsat0fbeb473) at4ceeefb2, with no conflicts; 282 + 272 focused tests pass, includingtest_bundle_compiles_discovered_source_tool_entrypoint, which the actors branch fixed.Fixes from the final review, 10 Nov
a389e849)test_a_worker_s_broken_check_is_unmeasured_on_that_worker_and_never_faults_the_callworker_limits_not_enforcedfor both shapes (127f953a); chat with no tool proxy too (55ed456f)test_worker_budgets_on_a_door_sealed_before_worker_limits_are_said_not_enforced,test_a_chat_agent_without_a_tool_proxy_says_its_workers_budgets_were_not_heldc5784634)test_a_templated_allowance_never_reaches_a_route_hidden_behind_an_encoded_separator_or_dots,test_an_actor_s_templated_permission_never_reaches_another_tool_s_own_route/w/...path was taken as a worker channel, even in a trial with no workers8febc776)test_an_app_s_own_w_route_runs_as_before_in_a_trial_that_gave_no_worker_channels76d32d31)test_a_budget_for_a_worker_the_contract_does_not_declare_is_refused_with_the_declared_onesRelease order
future-agi #3791 is deployed before this PR's template pin: on an older platform a worker's failed check would count toward the scenario's outcome.
Known gaps
After a user cancel mid-trial, the cancelled call's participants record was present but showed no worker evidence; whether the workers had acted before the cancel could not be told from that run.
Evaluator-only material and workers. At the world, a limited worker reaches only the world actions on its sealed list: a route that is no world action, or a request naming an actor, is refused before the world and recorded as that worker's (
test_a_limited_worker_s_request_for_anything_that_is_not_a_world_action_never_reaches_the_world,test_a_worker_naming_an_actor_never_borrows_that_actor_s_permissions,test_a_worker_s_request_for_a_route_that_is_no_world_action_is_refused_before_the_world; each fails with its guard removed). Actors are never given evaluator-only fields or files (actors PR). Not closed here: workers run inside the tested agent's own process, and on the hosted sandbox every process runs as one OS user, so the agent's own code, coordinator or worker, can read files the harness keeps in the sandbox, including the environment'ssub_goals.json. Keeping evaluator-only files out of the agent's reach is part of evidence separation (T44 in the product plan), which is planned separately and will come as its own PRs.Per-worker readiness rows. Run readiness does not yet show, per worker, whether its limits are Allowed, Blocked or Not enforced for this agent. Not built; about 1 to 2 days, later.
On an environment sealed before this release, a scenario that relies on worker limits is now held rather than graded (Review round 1, point 5), so the run 65403e6a case (
no_refund_issued_on_timeoutfailing although no deadline was enforced) no longer counts against the agent.Review round 1
c3998cb0c3998cb0permissions.jsonfails opennot_permittedbefore the world. Missing-file regressiontest_a_worker_s_call_is_refused_when_the_sealed_allow_lists_are_missing_while_channels_were_givenc3998cb0address_env, ids no placeholder can carry[A-Za-z0-9_.-], and a separate worker with no address variable. Sealing gives no channel to an ambiguous or unnameable worker, so one worker's calls are never handed to anotherc3998cb0worker_limits_not_enforced, and the scheduler records failureworker_limits_not_enforced(scenario-side, not retried) with every check unmeasured instead of gradingc3998cb0Each point has a test that fails with its fix reverted (seen failing for 3, 4 (validation and sealing), 5 (scheduler hold, voice notice carried to the scheduler, permission-only notice on voice and chat)). Base merged first:
feat/TH-8875-actorsate521d248(95512e60,605c5fd3). Tests: every test file this PR touches, 1,720 passed and 1 failed,test_grade.py::test_the_cap_holds_for_the_session_that_saves_too, which fails the same way ondev(a test body merged into another, using an undefinedst). Ruff findings on the changed source files: unchanged from before.Found by the live run after round 1, fixed in
5ce18a2b: a scenario with a short worker-action deadline failed certification. The readiness probe ran the scenario with its deadline, so every worker call was refuseddeadline_passed, and the probe reported "Inspect how the agent's tools are bound to the world", so the environment could never be validated or saved. The probe now checks the tools' binding without the scenario's worker-action deadline (a rule of its graded trials), and a worker's call the door refuses for that worker's own limits (not_permitted,deadline_passed) counts as reaching the world, since it came through the worker's bound channel; an agent's ownnot_permittedand a channel no worker owns (identity_conflict) still fail it. Tests:test_a_readiness_probe_sets_no_worker_deadline_while_a_graded_trial_doesand three new cases intest_private_workflow_fails_when_no_tool_call_reached_the_world, each failing with its fix reverted. Live on the local stack with production's sandbox specs: a 2 s worker budget draft now validates (ready) and saves, and its run shows the order-lookup worker asdeadline_passedwith both calls refused.Found by the live run after that, fixed in
7d4f8c67: a fault the scenario names by the agent's tool (get_order) was never applied to that call when it came through the tool proxy, a worker's channel included, because the proxy door matched faults by the route's world action only; the trial then saidfault_not_applied. The door now also matches the call's sealed tool, and the trial's notice reads the log row's tool. Tests:test_a_fault_naming_the_tool_holds_its_call_through_the_proxy_and_a_worker_s_channel,test_a_fault_the_proxy_applied_under_the_route_s_name_says_nothing, each failing with its fix reverted. Live on the local stack: the order-lookup worker's firstget_orderwas faulted once and its retry committed, with no notice.Found by the final review, fixed in
84313436: abinding: sharedworker's sealed allow-list and budget were held by nothing (no channel), with no notice, so the scenario was graded against the agent. Permitted tools on a shared worker, a deadline naming one, and one deadline for all with no separate worker now go back to authoring, and the trial is held withworker_limits_not_enforcedwhere a bundle still has them. The write-scenarios skill now says a budget names only separate workers and that a deadline scenario's task checks must not require the late worker's calls. Tests fail with each fix reverted.Not built here: process or OS-user isolation of workers (so evaluator-only material stays unreachable) is planned under T44 and will come as its own PRs; limits for shared-binding and in-process workers stay not enforced until then; TH-8888's execution timeout, cancel evidence per worker and cleanup status move to the follow-up TH-10189 (reduced scope recorded on TH-8888).
Scope
In: per-worker allow-lists and deadlines at the world, worker sub-goals and verdicts, worker entries in state changes, the returns reference.
Out: limiting workers that share their tools' code (they cannot be told apart at the world); workers of agents whose tool calls the harness itself runs in process.
Test cases
The live cases with their outputs (permissions from the contract, a refused write, one worker passing its deadline while the other completes, an environment sealed before worker limits) are in future-agi #3791's Test cases section, runs e9ebc3c8 (fresh seal from this branch) and 65403e6a (older seal).
Tickets and plan
not_permittedbefore the world and recorded as that worker's attempt (live, call 4f73a9a9); a sub-goal can name one worker and is graded on that worker's calls, reported beside the task's verdict, and reads inconclusive with no evidence for that worker; the duplicate-refund case is in the returns reference and passes live with no second refund (call 6d909408). Left: evaluator-only files can still be read by the agent's own process, coordinator or worker, on the hosted sandbox (evidence separation, T44, not scoped); no run that creates a second refund was shown failing.deadline_passedand keeps its earlier calls, while another worker finishes (live, call f17fc9d6). Reduced scope, recorded on TH-8888: the deadline is a worker-action deadline and does not stop the worker's execution; a real execution timeout (which needs process control of the customer's worker), a user cancel recorded per worker with its partial evidence, and a per-worker cleanup status move to the follow-up TH-10189.Part of TH-8887
Part of TH-8888
Head on 12 Nov:
84313436, with the latest actors base merged (feat/TH-8875-actorsat66cc5f44).