[khushalsonawat]

Into feat/TH-8804-source-subfolder (#225's branch). One commit on the base as it stands, c0e51b7e.

Summary

TH-9491 (#278) cleans a URL's userinfo, query and fragment. It can't catch a secret that isn't inside a URL, or a password split by a space, a line break, /, ?, # or @. This adds those shapes to the kit's one cleaner (providers/agent.py _redact), in three steps (providers/secret_shapes.py):

1. The raw value:
   • Rule 1: a password broken by whitespace, ?, # or /.
   • Rule 1b: a password or user holding @ before the password's /. This covers Dileep's [TH-9491] fix: keep URL credentials in an agent's description out of what the kit sends and stores #278 inputs (postgres://admin:Xk9/Qp+Zr8@db…, https://user:pa#ss@host…) and Ash's (postgres://admin:P@ss/Xk9@db…, smtp://me@gmail.com:app/pass@smtp…).
   • Rule 2: a query after a break.
   • One-URL values: a value that is one URL is joined first, as the chat stream joins it.
2. TH-9491's URL step: unchanged.
3. Key patterns:
   • provider keys (GitHub, OpenAI, Anthropic, AWS, Slack, Stripe, Google and others);
   • JWTs, PEM key blocks and Telegram bot URLs;
   • named values (password=, token:, "api_key": "…"), with guards for honest text.

Where it runs:

• Chat: the stream's seam (ChatHold, _DeltaCleaner) never cuts inside a match, so every split streams what cleaning the whole reply stores.
  • Before prose and past N, it cuts only at a line break.
  • It holds while a URL's userinfo, or the URL's own ://, is still open.
• Outbound logs and errors run the full cleaner.
• The scenario send builds its persona record from cleaned fields, so a key is never cut short before it's cleaned.
• The session's activity line is cleaned before its cut.

Verification

Python 3.12.3, one run after the last change, locally under a memory cap; test_secret_shapes and chat also ran on 3.11. CI (sdk-smoke.yml) runs test_builder_chat.py, test_scenario_source_provision_agent.py, test_reading_entry.py, test_hosted_entrypoint.py and tests/providers (so test_secret_shapes.py and test_providers_agent_build.py).

Results:

• tests/providers/test_secret_shapes.py: 235 passed. It includes speed budgets on 5 MiB crafted shapes, and rule 1b is held to at most 2× the cleaner without it.
• tests/harness/test_builder_chat.py: 393 passed. The reviewers' line-break sweeps are tests here, and every case streams equal to the stored message, with no part of its secret in any event.
• test_outbound.py: 242 passed, and test_contract_writer.py: 43, locally. Both pass whole on the base (c0e51b7: 235 and 43), and c62d03ca adds them to sdk-smoke.yml. Its fix-only check (one reviewer) is clear, with no findings: on Python 3.12.14 the two files give 285 passed.
• test_scenario_source_provision_agent.py: 48 (in CI). test_scenario_source.py: 115, locally; this PR doesn't change that file.
• test_providers_agent_build.py: 86 passed and 41 skipped locally (no hypothesis). CI runs the property tests.

Failures that are the same on the base:

• tests/test_harness_sandbox_server.py: 2 GitHub tests (test_public_github_branch_url_populates_job_ref, test_private_github_submission_becomes_hosted_job).
• livekit tests in test_hosted_entrypoint.py (livekit isn't installed locally).

Planted bugs: 57 planted, 52 caught. The 5 that survive change nothing, or are constant-factor slowdowns. Every review fix is pinned by a row that fails without it.

Review:

• The design took four rounds, and the rule 1b addendum had two reviewers.
• The change went through five review rounds and a final check of the last fix, each with two reviewers, apart from the narrow later checks, which had one by the coordinator's call.
• Fix-only check after the clear: the four Lows fixed after the final check, and the split fake-key literals, had their own check (one reviewer): clear, no findings. The tree it checked is this PR's head.

Known limits

Passwords and hosts the rules can't read

• Left as TH-9491 leaves it: a password with two or more whitespace runs, a blank line, a digits-first or upper-case-port first half, a part over its bound, %40, or a ? or # after the password's @.
• A password that itself contains :// can stream in chat before prose. The stored message is clean.
• Rule 1b can move an honest host. For example, a URL with a password whose first path segment is text@host (https://u:p@api.example.com/a@b.com → https://b.com), the same for /react@18 and /repo@main, the scp form, and a port rule 1 doesn't read. The password is still removed.
• Rule 1 moves an honest broker list: BROKERS=amqp://rabbitmq,amqp://b@other.example → amqp://other.example.
• A one-URL value joins its next line (TH-9491's join).
• A key or named value after a URL, in text with no whitespace, takes the rest of the run.

Chat before prose, past N (4,096 characters)

• It holds an open match up to 16,384 characters. Past that it sheds as before, and the tail of a longer key can stream.
• These honest lists hold to the end, or stream as [redacted]:
  • a list of URLs with an @ line before each URL;
  • a URL list with mailto: or bare hash lines;
  • registry lists with a port and a /@ path;
  • a line that is only a scheme, then ://…, then more than about 12 K of glued URLs.
• A name, its = and its value on three separate lines can stream lower-cased.
• A structural alternative (hold, then redact, never cut before prose) is recorded as a High follow-up in the coordination register.

Speed (crafted input only, linear, under 5 s)

• 5 MiB of a://u@ takes 1.03 s on the base already.
• _url_around takes 1.6–1.8 s on 5 MiB of repeated named-value hosts.
• With a key block open, each chat delta costs as much as the held text.

Not cleaned here (recorded for follow-up)

• A local run's own provision (platform.py's persona_of callers) sends persona records uncleaned.
• The one-URL join misses a jdbc: prefix and a line break inside ://.

Checklist

• Security-sensitive output is redacted or justified
• No hosted-service dependency was added to release-gated examples