feat(harness): refuse browser, computer-use and crawler agents by khushalsonawat · Pull Request #224 · future-agi/agent-learning-kit

khushalsonawat · 2026-11-02T09:37:36Z

Stack: #225 (TH-8804) stacks on this PR. This one stays based on dev.

Summary

Only voice and chat agents are supported. An agent whose main job is driving a browser, operating a computer or crawling sites used to spend a whole build, then end as a crash that retries or an error blaming us. It now stops at the reading step with a clear "not supported".

No code can say for certain what an agent's main job is, so a check of the agent's code and the model's reading work together:

Model says	Code check	What happens
Browser	tool found	one second look, naming the tool; if it still says browser, blocked
Browser	no tool	one second look; if it still says browser, blocked
Chat or voice	tool found	one second look; if it still says chat or voice, it passes
Chat or voice	no tool	passes, as before

The block

Writes failure.json in the HarnessFailure shape: agent_type_unsupported, domain agent, stage understanding_agent, not retryable. Then exits 78.
Applies everywhere, including runs on a person's own machine, where the message prints and a new read clears an old file.
A blocked agent gets no contract.json, and a resubmission as chat is refused.

The parts

failure_file.py: writes the file whole, in one step, on a disk with hard links; without them it writes in place, and a reader could catch it part-written. The first failure of a build stands. Also used by TH-8804.
agent_type_check.py: the tool list, the check and the rule. No model call, no customer code run, no job or session needed. Test, sample, docs and developer-setup files don't count, and neither do comments.
tools.py: the rule runs first in submit_contract, which every reading passes through. The scan runs off the event loop, once per conversation.

What stops after a block

_await_harness, _understand and _chat exit 78. A timeout, or no contract with no file, stays 1.
The builder stops prompting, and a platform command is answered with no model turn once the job failed or was canceled, or the file exists. A completed job still runs its post-build check. A person's message still gets a reply.
set_modality to browser refuses and changes nothing.

The model's question (skills/understand-agent/SKILL.md, schema text): does a person come to this agent for a conversation, or to have it carry out work on websites or a computer? A coding agent is chat.

Deploy order: future-agi #3661 (the platform reading failure.json) ships first. An older platform reads exit 78 as a failed environment: not retried, but labelled wrongly, as our failure. Don't move the pinned image ahead of it.

Verification

uv run ruff check .: run on the touched Python files only. authoring_entrypoint.py has two F811 errors that are on dev.
Focused pytest targets: tests/harness/test_agent_type_refusal.py, 355 passed, and tests/harness/test_executor_completed_counts.py, 8 passed. Both added to sdk-smoke.yml.
Full uv run pytest -q when touching shared runtime, CLI, SDK, or release gates: not run on the laptop; named files below. CI runs the job list.
TypeScript build/test when touching typescript/: not touched.
agent-learn release-proof when touching release-candidate behavior: not run.

Also run, whole files: test_contract 17, test_storage_decision_admission 32, test_hosted_conversation 19, test_hosted_authoring_entrypoint 9, test_early_readiness_order 6, test_new_kind_needs_no_code 4, test_instrumentation_evidence 33, test_builder_chat 51, test_harness_surface_conformance 17, test_native_bundle_admission 4, test_poc_guest_booking 25, test_release_workflow 18, test_certify_only 39, test_http_setup_failure_rule 21, test_http_diagnostics 17: all pass.

Pre-existing failures, the same tests fail at c95d01e: test_harness 6, test_vertex_gemini_backend 2, test_harness_architecture 11.

70 deliberately planted bugs were each caught by a named test.

Dileep's second review was fixed in 880f9a22, then checked in three review rounds. The fixes from the first two rounds are 4435f0e6 and d2c32290. The last fixes, 134ff1c6, went in after the third round without a further review. Each of them has a test that fails without it.

Dileep's third review, and four crafted slowdowns found in TH-8804's reviews, were fixed in b9bda589. That covers: records read once a scan with a 16 MiB budget and a 4 KiB minimum each, the idna, punycode and undefined codecs not decoded (their raw bytes are still searched for whole browser names), and one second look before every refusal, its state moved into SecondLook. Two fresh reviewers cleared it. Their two one-line fixes went in afterwards, unreviewed, as 35fd0c98. Smaller findings are filed as TH-9448. Dileep's fourth review was fixed in 4060c6e6, checked by two fresh reviewers. Their fixes for folder links and for code one level inside a test folder went in as 7726ded8, each with a test that fails without it.

The reading step ran on 11 agents, through a local gateway on Claude Sonnet, not production's Gemini: 10 read as expected. Blocked: a browser agent, a computer-use agent, a crawler agent and a browser agent with an unlisted tool. Passed: plain chat, chat with a browser helper, a research agent, a chat app with Playwright tests, a coding agent with a browser tool, a voice agent with a browser tool. The second look fired 5 times and changed no reading.

Checklist

Public API names stay under agent_learning.*, agent-learn, or @future-agi/agent-learning-*: no public API added.
New user-facing workflow has docs or an example: HOW-IT-WORKS.md updated; no new workflow.
New release claim has executable gate coverage: no release claim.
No hosted-service dependency was added to release-gated examples
Security-sensitive output is redacted or justified: only names from our own tool list reach the message or details.

Known limits

In the end the model decides. A misread that survives the second look is a wrong block with no retry or override.
crawl4ai was read as chat: a crawling service with no agent of its own. It would build in full and fail at the end. Accepted for now.
A real browser agent using a tool not on the list can be talked into chat by the second look, with the same outcome. A longer list narrows it. Traced in the code, not seen from a model.
The code check reads Python and JavaScript or TypeScript only. In Go, Java or Ruby it finds nothing, so a browser reading gets a second look and the model decides.
A Python file with a single line hundreds of thousands of characters long makes the code check slow on the sandbox's Python. A repo built to do this can stall its own build past its time limit. Follow-up ticket to come.
On someone's own machine, if the output folder is read-only and holds an earlier reading, a refused agent can still exit 0. Hosted builds can't reach this.
On Claude Code, calls in one model message run one after another. So the second call in a first message is decided before the model has read the second look. A browser reading can then be refused before the model has read the look. A real browser agent read as chat twice in one message is saved, builds in full and fails at the end.
A browser answer to a chat or voice look is refused at once, since that look said browser stops the build. Every real browser agent now takes one extra model turn before its refusal.
A chat app whose test suite keeps page objects or helpers inside a test-named folder, such as ui_tests/pages/, now gets one second look, where it got none before.
Rebuilds with a saved contract.json skip the reading, so they aren't blocked.
detect_instrumentation can still raise on a very deep Python file, as it does on dev (TH-8977).
Scanning only the agent's folder is done in feat(harness): build an agent from a repo's subfolder #225, which stacks on this.

Refs: TH-8806

Only voice and chat agents are supported. An agent whose main job is driving a browser, operating a computer or crawling sites used to spend a whole build and end as a crash or an error blaming us. It now stops at the reading step with agent_type_unsupported in failure.json, and exits 78. No code can say for certain what an agent's main job is, so a check of the agent's code and the model's reading work together. Where they disagree the model looks once more, and its answer stands. Refs: TH-8806 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>

Refs: TH-8806 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>

cdileep23

Please check the comments.

cdileep23

Please check the comments.

- Two submissions at once no longer skip the second look. - A disk with no hard links still gets failure.json, and no temporary file is left behind on a failed write. - Only folders named wholly for tests are skipped, so e2e_agent/ is read. - Only the tools' API names count as computer-use, not the bare words. - Packages unpacked beside the agent's code aren't scanned, by each distribution's own record; a browser package imported from there counts. - A lone apostrophe no longer protects a comment in script files. - The scan has an overall size limit and is reused per checkout. - A refusal removes an earlier contract.json from the folder. - The sandbox-server route reports the failure a build left. Refs: TH-8806 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>

cdileep23

Please check the comments.

- Folders whose last part is a test word, such as functional_tests/, are skipped again; e2e_agent/ and testing_agent/ are still read. - A scan is reused only while the files it reads are unchanged, so a nested edit or a removed helper is scanned again. - Files too big to read no longer use up the scan's size limit. Refs: TH-8806 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>

… scan - A second look now gives the model a value to send back, and an answer counts only if it holds one. Calls in one model message run before the model reads any result, on either loop, so none of them can know it. This replaces the entry-time read and the one-second rule. - RECORD files are matched by exact path, read only when they are regular files, and capped at 64 MiB in total. - Python and package.json are read first, in path order. - The computer-use object key needs an environment, so flags and labels don't count. - More test-helper folder names are skipped. - The earlier contract.json is removed before the failure is written, and the failure is written even if the removal fails. - A disk that refuses chmod still gets the whole failure file. - The executor tests run in CI. Refs: TH-8806 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>

cdileep23

Please check the comments.

- Take the call after an unanswered second look as the model's answer, so a model that never sends the look's value back can still be saved or refused instead of looking again until its turns run out. - Decide each call on the value it carries, so a value-less call in the same message can't undo an answer that was just saved. - Skip a file the tokenizer can't read (a NUL byte on Python 3.12) instead of failing every scan. - Keep a JS regex literal like a string, so a `/*` inside one no longer hides the rest of the file. - Apply the RECORD rule to every folder holding a dist-info, so agent code beside bundled packages in a subfolder is still read. - Read Python and package.json saved with a BOM. - Read devDependencies, and add puppeteer-extra, playwright-extra, webdriverio and @crawlee/ packages. TH-8806 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>

An unclosed character class could end either at a "]" or nowhere, so a run of "[" could be split many ways. Make its end a "]", the line's end or the file's end, so each input has one reading. TH-8806 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>

…r answer - One record budget for the whole walk; linked dist-info folders and records are skipped, and rows stay text. - The same answer re-sent without the value stands; a made-up value no longer ends the unanswered look. - Calls in one message share one scan. - A regex literal's closing slash must read as one, so a division that starts a line or a slash in JSX text isn't taken for a literal. - Python markers are read from bytes, as imports are. - Tests pin each of these, the accepted residual, and three scan paths that were unpinned. TH-8806 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>

…nly a lost scan - Record rows stay relative to the folder whose records hold them, so a deep folder no longer multiplies what each row costs. - A scan runs again only when it raised, was cancelled, or belongs to another loop; one that finished after its waiters were cancelled keeps its answer. - A regex literal that meets a stray bracket is kept to its line's end, so its quotes and backticks can't open a string or template. - Folders are walked in sorted order, and the record budget's real cost is stated. - The second look keeps one flag for a chat answer that carried a value; behaviour is unchanged. - Tests pin each of these and the close rule's flags and quotes. TH-8806 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>

…at never changed - A file is checked against the records of the nearest four folders that hold them, kept as a stack the walk unwinds, so a crafted chain of such folders no longer multiplies every file's check. - The second look's same-answer rule reads the answer alone: a value on a browser answer has already refused and stopped every call. - The comment states what the regex literal's exclusions keep linear and the template opener the line-end fallback can take in. - Tests pin the nearest folders, memory against folder depth, the walk's order on any disk, and classes across escaped line ends. TH-8806 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>

Conflicts, resolved keeping both sides: - agent_type_check.py: both constant blocks kept. `_python_code` takes this branch's token list, and Python markers stay with `_python_markers` and the scan's TokenBudget. #224's coding-line and BOM handling moves into `python_source.read_module`, which now decodes as Python does (`tokenize.detect_encoding`), so every scan honours a coding line. - tools.py: both the last-sent counters and #224's shared, shielded scan; the scan runs `_tools_in_agent_folder` with the folder. - test_agent_type_refusal.py: the tokenizer-breaks test keeps both blobs, named `a_blob.py` to be read first; #224's guard test patches `read_module`, which reads a .py on this branch. Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>

…he folder, and answer round 4 (TH-8804) - Refuse a path past twice the system's limit while resolving: links that leave parts behind as they unwind grew it a level at a time, each step a copy. - Read the records of the agent's folder and every folder above it once, under one 16 MiB budget, with rows kept as text, as #224's walk does. - Read a .py in the encoding its coding line names in the import walk too, and find the coding line with lone CR line ends; refuse non-text codecs. - Pin each scan against coding lines Python can't read, and assert the stop message when a refusal arrives during the scan. Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>

cdileep23

Please check the comments.

…and undecodable files cheaply, answering round 6 (TH-8804, TH-9442) Round 6 on 7544883 found two coding-line stalls of the save, a Medium that round 5 had claimed fixed, and TH-9442's shape at 12.3 s. Khushal chose to fix all of it here, TH-9442 and the five Lows included, with one more round. - idna is not read, as punycode and undefined aren't: its decode runs in Python for each label (40 x 128 KB: 3.58 s -> 0.13 s on 3.12.3). - The import walk decodes strictly, then falls back to UTF-8 with replacement. Replacing ran Python's own error handler for each byte in most codecs (32 x 1 MB of cp1252 + 0x8d: 2.06 s -> 0.22 s). - Once the shared record budget is spent, a browser package at the folder or the root counts as installed, not as the customer's own module: unread records can't say otherwise, and no read order keeps both whole. Round 5's "root first" order didn't fix its own shape; records are now read nearest first. - A record hides a file the agent imports only from the nearest four folders holding it, as _agent_files checks. - At the save the walk stops at 100,000 lookups on disk (_WALKED_LOOKUPS): a candidate costs one, and a file found one more for each part of its path. Bundling keeps no such limit. The walk lists the folder's own code with os.walk, not entering ignored folders and with no relative_to, which on 3.12 built every parent of each path. TH-9442's shape: walk 7.82 s -> 1.41 s, save 12.3 s -> 4.9 s; the rest is its 31,000 files, which a whole-repository save on #224 pays too. Honest folders use under 9,000 lookups. Unreviewed: round 7 is scoped to this commit. Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>

… browser reading one look - Each RECORD is read once a scan and none on a reopen; the budget is 16 MiB and every record costs at least 4 KiB. - A .py declared in idna, punycode or undefined is not read. - A browser reading always gets one second look before a refusal; the look's state lives in SecondLook beside the check. - A file the scan can't read is logged; a test-word folder is skipped only when it holds only tests; a scoped package reports its scope. - A refused local understand prints its spend; the docs describe the current rule. TH-8806 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>

… every import once the budget is spent - A non-ASCII byte on the coding line no longer hides an idna codec. - Once the record budget is spent, no top folder shows the agent wrote an import, so a vendored toolkit can't hide its browser tool. - The docs and SecondLook state where a look is not given. TH-8806 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>

#224 (TH-8806) moved on: records read once per scan under 16 MiB with a minimum charge per record, slow codecs skipped, the second look in a SecondLook class with one look for every browser reading, a module logger, test-suffixed folders skipped only when they hold just tests, and scoped packages reported by scope. Merged, not rebased. Resolved: - tools.py: #224's SecondLook replaces #225's copy of the receipts and _second_look; #225 keeps its race counters (sent, saved), the sibling check and the shared reach. - agent_type_check.py: #225's shared reader (python_source.read_module and source_encoding, which already refuses idna, punycode, undefined and non-text codecs, and leaves a file whose coding line won't detect unread) stays the one path, so #224's _unread_codec, now unused, is dropped. #224's records-once and its every-import-counts once the budget is cut move into _scan, where #225's generic _tools_in takes the paths and the local modules. - Tests: #225's folder-scan test answers the look; the exact-budget test charges #224's minimum per record; a race test patches decide where SecondLook calls it. #224's NUL-byte test now checks what holds through the shared reader: the file is left unread and nothing raises. Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>

cdileep23

Please check the comments.

…t test folders hold - A .py in a refused codec (idna, punycode, undefined, unicode-escape) is not decoded, but every browser name and marker in its raw bytes still counts, so it gets a second look. - A test-named folder is skipped only when it holds nothing but tests: package.json counts as code, conftest.py alone doesn't decide it, and a code folder inside it is read. - Lone CRs are normalised before the codec check; scripts are read with replacement characters; files the helpers drop are logged. TH-8806 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>

…code one level inside it - Links inside a test-named folder are not followed to decide it, so a folder of links can't multiply the check. - A package or a nested test-named folder holding code inside a test-named folder keeps the folder read. TH-8806 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>

#224's rule now gives every browser reading one second look first. TH-8951 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>

cdileep23

Please check the comments.

…ole names in unread source - unicode-escape is decoded like any other codec: it is linear, and its escapes can spell an import that the raw bytes don't hold. - In a file left unread, a browser name counts only as a whole name, so "steelers" is no import of steel. - Tests pin a test-named folder that is itself a link, and the toolkit named in an unread file. TH-8806 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>

#224's two new commits (4060c6e, 7726ded) come in as written where they don't meet #225's code: the test-named-folder rules (package.json counts as code, conftest.py alone doesn't decide, a code folder one level in keeps the folder read, a linked folder isn't listed) and the doc line. Where they meet #225's shared reader, their intent is carried over: - a .py in a codec refused by name now counts the tool markers and the toolkit in its raw bytes, as well as the browser packages; the packages go through the agent's own modules, as #224's do; - the tokenizer reads a lone CR as a line end, as Python does, so a file of only CRs that starts with a comment isn't all comment (3.11 missed its markers); - scripts are read with replacement characters, as Node reads them; - Python files the readers drop are logged at debug. #224's unread-codec helper and its own codec set aren't kept: #225's reader already refuses the same four codecs. Two tests added: the toolkit in a refused codec, and a linked test-named folder not listed. Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>

#224's last commit (39e5ae8) and dev's other merges come in. Where #224's reader meets #225's shared reader, #225's stays: it already decodes unicode-escape as Python does (under a per-scan total, which #224 doesn't need) and counts whole names only in a file it can't decode. #224's tests come in, with one copy each of the escaped-import test and the codec rows. Both sides' new hosted-conversation tests are kept. Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>

[TH-9448] fix: follow-ups from #224's last review of the browser-agent scan

khushalsonawat self-assigned this Nov 2, 2026

test(harness): stop depending on MCP internals in the write-failure test

15f74f3

Refs: TH-8806 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>

khushalsonawat mentioned this pull request Nov 2, 2026

feat(harness): build an agent from a repo's subfolder #225

Open

5 of 10 tasks

khushalsonawat requested a review from cdileep23 November 2, 2026 11:24

cdileep23 reviewed Nov 2, 2026

View reviewed changes

Comment thread src/fi/alk/harness/failure_file.py Resolved

cdileep23 self-requested a review November 2, 2026 11:52

cdileep23 requested changes Nov 2, 2026

View reviewed changes

khushalsonawat requested a review from cdileep23 November 2, 2026 12:09

cdileep23 requested changes Nov 2, 2026

View reviewed changes

Comment thread src/fi/alk/harness/agent_type_check.py Resolved

Comment thread src/fi/alk/harness/agent_type_check.py Resolved

Comment thread src/fi/alk/harness/agent_type_check.py Resolved

khushalsonawat requested a review from cdileep23 November 2, 2026 12:39

khushalsonawat mentioned this pull request Nov 2, 2026

feat(harness): report the agent's framework in the agent document #229

Merged

4 of 5 tasks

cdileep23 requested changes Nov 5, 2026

View reviewed changes

khushalsonawat mentioned this pull request Nov 5, 2026

feat(harness): read an agent without building it #245

Merged

4 of 5 tasks

github-advanced-security Bot found potential problems Nov 5, 2026

View reviewed changes

Comment thread src/fi/alk/harness/agent_type_check.py Fixed

khushalsonawat added 3 commits November 5, 2026 15:24

khushalsonawat mentioned this pull request Nov 6, 2026

feat(harness): list the agents in a repo without building it #251

Merged

5 of 10 tasks

khushalsonawat requested a review from cdileep23 November 6, 2026 10:24

cdileep23 requested changes Nov 10, 2026

View reviewed changes

khushalsonawat added 2 commits November 10, 2026 13:35

khushalsonawat requested a review from cdileep23 November 10, 2026 08:42

cdileep23 requested changes Nov 10, 2026

View reviewed changes

khushalsonawat added 2 commits November 10, 2026 14:38

khushalsonawat requested a review from cdileep23 November 10, 2026 09:34

khushalsonawat added a commit that referenced this pull request Nov 10, 2026

test(harness): a browser reading gets its one look before the refusal

edb6926

#224's rule now gives every browser reading one second look first. TH-8951 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>

cdileep23 previously approved these changes Nov 10, 2026

View reviewed changes

Comment thread src/fi/alk/harness/agent_type_check.py Resolved

Comment thread src/fi/alk/harness/agent_type_check.py Resolved

khushalsonawat dismissed cdileep23’s stale review via 39e5ae8 November 10, 2026 10:05

khushalsonawat requested a review from cdileep23 November 10, 2026 10:05

khushalsonawat merged commit dc25cad into dev Nov 10, 2026
15 checks passed

khushalsonawat deleted the feat/TH-8806-refuse-browser-agents branch November 10, 2026 10:08

khushalsonawat mentioned this pull request Nov 11, 2026

[TH-9448] fix: follow-ups from #224's last review of the browser-agent scan #288

Merged

1 task done

khushalsonawat added a commit that referenced this pull request Nov 11, 2026

Merge pull request #288 from future-agi/feat/TH-9448-scan-follow-ups

bc5a363

[TH-9448] fix: follow-ups from #224's last review of the browser-agent scan

Conversation

khushalsonawat commented Nov 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Verification

Checklist

Known limits

Uh oh!

cdileep23 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cdileep23 left a comment

Choose a reason for hiding this comment

Uh oh!

cdileep23 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cdileep23 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cdileep23 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cdileep23 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cdileep23 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

khushalsonawat commented Nov 2, 2026 •

edited

Loading

cdileep23 left a comment •

edited

Loading