[khushalsonawat]

Description

When the kit reads an agent, it writes the agent's prompt, rules, summary and notes in its own words. That text can hold a web address with a password or key in it, such as https://bot:hunter2@hooks.example/v1?key=sk-1. Until now it reached the platform as written: in the provision payload, the suite evals, the judge's instructions, the stored reading and the hosted builder's chat.

After this change, every web address in that text loses its username, password, query and fragment wherever the kit saves or sends it.

Stacked on #271: this branch merges #271's head 53f06233, so that commit shows here until #271 lands.

Type of Change

• Bug fix (security: customer credentials leaving the sandbox)

Changes

• providers/agent.py, _redact: now also cleans addresses inside prose. It is still one cleaner, not a second one.
  • Every :// starts an address, even after a digit, dot or dash (1.https://…, -https://…, ${SCHEME}://…).
  • An address runs to the next whitespace, so no guessed end inside a word can leave a key behind.
  • A field that is only an address is cleaned as before.
• harness/contract.py, one writer for contract.json: every writer goes through it, and it cleans the free text first. That covers the reading, the builder, the rule and tool amendments, and the repair steps.
  • Each field is marked as free text, which is cleaned, or technical, which is kept as written.
  • A test fails if a field is unmarked.
  • A scan test fails if a module writes contract.json without the writer.
• Send-time guards: for readings saved by older versions, the text is cleaned again just before each send. That covers:
  • _chosen_evals_and_prompt, the provision payload;
  • run/grade.py, the suite evals' agent_prompt and the grade calls;
  • hosted_entrypoint._agent_instructions, the judge;
  • the live run view;
  • emit_model_event;
  • cli._render's stdout.
• harness/hosted_chat_entrypoint.py, the hosted builder's chat:
  • Cleaned events: tool_result, the artifact paths, the question prompt and its options, and every assistant_message, on success, interruption and failure.
  • The model's text first: it is cleaned alone, before the save note, health lines and failure detail are added.
  • Streamed deltas: they go through a bounded holdback:
    • text since the last whitespace is held;
    • past 4,096 characters, everything before the last address start is sent;
    • past 16,384 characters, the address becomes [redacted];
    • assistant_message and the end of the turn flush what is held.
  • Also cleaned: amend's rule echo and cli's authoring drain.

Known limits (each has an owner)

• Traces: TH-9504.
• Owned by TH-9500, the secret scanner:
  • bare keys and tokens in an address's path;
  • the agent's own files;
  • a password that contains whitespace;
  • an address broken by a line break or tab inside a sentence.
• The builder's other pipes, which still send raw: the SDK transcript mirror (PlatformSessionStore.append), coordinator_state, the remember notes and request_adjustment. TH-9532.
• The model's files inside the workspace checkpoint archive: .futureagi-adk.db, .claude/, the pending-question file and the kept notes. TH-9521.
• Honest text that changes: described in the design page's "What can go wrong". For example, words joined to an address that has a query are lost, and hosts are lowercased. In every case more is removed, never less.
• A key split across two chat deltas past the 4,096-character hold-back goes out in two halves, each cleaned on its own. No real key is that long without whitespace.

Testing

Each test file was run on its own on the final commit:

test file	passed
test_builder_chat	179
test_contract_writer	43
test_providers_agent_build	120
test_hosted_entrypoint	145
test_scenario_source_provision_agent	28
test_agent_reading	106
test_judge	71
test_harness -k suite_eval	5
test_hosted_conversation	22
test_contract_tools_subfolder	175
test_agent_type_refusal	515
tests/providers	161
test_harness_sandbox_server	36, plus 2 failures

The 2 failures in test_harness_sandbox_server are the base's own github_commit_sha_required failures, and they fail the same way on the base.

After merging the base at cf3ebb44, the files that overlap it passed again:

• test_contract_writer 43;
• test_harness_architecture 55;
• test_agent_type_refusal 651;
• test_contract_tools_subfolder 180;
• test_bundle_author_v2 323;
• test_instrumentation_evidence 48;
• test_builder_chat 179;
• test_harness_credentials 80.

Planted bugs check that the tests catch what they claim. Each was a one-token edit, restored and confirmed by sha256:

• Round 3's fix: 60 plants, 58 caught. Of the two survivors, one is a named limit of the scan, and the other is caught by its own named test.
• The last four tests, added for the fix check's Mediums: 4 plants, all caught.

Two reviewers each re-ran the leak probes, and both found 0 leaks:

• reviewer A over 509,000 adversarial messages, plus 1,200 full turns;
• reviewer B with its own fuzz.

Review history:

• Plan review: three rounds, then a replanned design (v5.1) whose review cleared.
• Change review: three rounds, then a check of the fix only by two fresh reviewers, as ruled after round 3. Both were CLEAR, and their Mediums are fixed in this commit.

Related Issues

Refs: TH-9491

[cdileep23]

Please check the comments.