[khushalsonawat]

#278 (TH-9491) has merged into feat/TH-8804-source-subfolder, so this diff is now this PR's own change only. It goes into #225's branch because a stack may only be two deep.

Hand-resolved merge hunks, for review. Merge 41718e66 had a conflict in one hunk of src/fi/alk/harness/tools.py, _save_contract:

• [TH-9491] fix: keep URL credentials in an agent's description out of what the kit sends and stores #278's side replaced the inline atomic write (temp file, then os.replace) with a call to write_contract(contract, destination) and path = destination / CONTRACT_FILE.
• The base's side built the reply as said = (...) instead of returning _ok(...) directly.
• I kept both: [TH-9491] fix: keep URL credentials in an agent's description out of what the kit sends and stores #278's write_contract lines and the base's said reply. That's the same resolution Ash made on [TH-9491] fix: keep URL credentials in an agent's description out of what the kit sends and stores #278 (e8499e78). The base now holds it as merged, so tools.py is no longer in this PR's diff against feat/TH-8804-source-subfolder.

One other merge was resolved by hand. In 50552115, both sides added tests at the end of tests/test_harness_sandbox_server.py, and both sets are kept unchanged. That file isn't in this PR's diff either. The rest (071abf6d, f1d2c49c, e7e0764f, 796cb120) merged cleanly.

Description

When a hosted job carries FI keys, the kit's traces go to the platform's own Observe project. Those spans carry the customer's text: tool input and output (up to 5,000 characters), prompts, tool error text, and recorded exceptions with their stack traces. No cleaner saw any of it, so a URL with a password or key in the agent's prompt reached Observe as written.

This change puts a cleaning exporter in front of fi-instrumentation's HTTP exporter, as the provider's only processor. Every string in every span goes through the kit's one cleaner (_redact, TH-9491) before it's sent.

Type of Change

• Bug fix

Changes

• A cleaning exporter (_CleaningExporter in observability.py). For each batch it gathers every string into one flat list: the span name, attribute values (strings inside tuples too), event names, event attributes, link attributes and the status description. It cleans them in one _redact call, then sends a copy of each span.

  • The copy keeps the span's ids, parent, kind, times, scope, resource, links and dropped counts.
  • Numbers and booleans pass unchanged. The cleaner's key rule is applied to each string attribute by its own key (an http.request.header.authorization or api_key value is masked whole), never to numbers, so token counts such as gen_ai.usage.input_tokens stay.
  • JSON values stay JSON: a value that parses as JSON is cleaned as a structure and dumped back, and a JSON string inside it is cleaned as a structure too, so the platform still parses it.
  • Declared values are masked, failing closed. Every value in the customer's secrets file, the simulator channel and the gateway file is replaced with ***, except names on NON_SECRET_SETTINGS: 27 platform settings such as model and provider names, the Observe project, locations and base URLs, which a span keeps. An unknown name is masked.
    • Each source is read on its own, so a name in two files masks both values.
    • A value of 8 or more characters is also masked in its JSON-escaped forms. A short all-digit value, such as a 4-digit PIN, is masked as a whole number.
    • Values are masked after traceai's cut rule and JSON reading. Those run on the value as recorded, so a declared secret can't change the length the cut rule reads.
  • The resource is cleaned once per resource and reused, and never with declared values: the platform routes a job's spans by its project_name.
  • JSON keys are cleaned as strings. Keys that clean alike get (2), (3), so no value is lost. JSON nested deeper than 64 levels is cleaned as text, and a span that fails while being gathered is dropped alone.

• The install (begin()). Straight after register(), the provider's processor tuple is replaced with one BatchSpanProcessor around the cleaner, and the old processors are shut down. add_span_processor isn't the install step:

  • on fi 1.0.0 (which uv.lock pins), it shuts down the default processor and turns export off;
  • on fi 1.2.x (what CI's pip install . resolves), it adds a processor beside the raw export.

• A run-time check. After the instrumentors are turned on, begin() checks that exactly one processor remains and that it exports through the cleaner. If not, tracing is off for the job:

  • every processor is taken off the provider, then shut down, then the provider itself;
  • one warning names only the error's class.

  On fi 1.0.0, a provider's shutdown alone leaves an added processor posting.

• Values cut by traceai. traceai cuts text at 5,000 characters (ending in ...), 2,000, 1,000 and 500. A cut can leave a numeric password looking like a port, which the URL cleaner keeps.

  • So a value of exactly one of those lengths that ends inside an address loses that address whole, from its scheme on (an empty scheme included).
  • This runs on the value as traceai recorded it, before _redact.

• Fails closed:

  • A span that can't be copied is dropped.
  • Any other error drops the batch and returns FAILURE, including an error in _redact or in the inner exporter.
  • A batch with every span dropped sends nothing.
  • No exception reaches the SDK, whose own log line would carry the text.

• CI: tests/harness/test_observability.py is added to sdk-smoke.yml, after tests/providers.

The private surface touched, the same in fi-instrumentation 1.0.0 and 1.2.2, and in OpenTelemetry SDK 1.39.1 and 1.42.1:

• TracerProvider._active_span_processor._span_processors;
• _default_processor;
• ReadableSpan, Event, Link, BoundedAttributes and BoundedList.

If any of these changes, tracing turns off or spans are dropped. It never exports raw.

Testing

• test_observability.py (in CI): 70 passed and 1 skipped, on both fi 1.0.0 with SDK 1.39.1 (the kit's venv) and fi 1.2.2 with SDK 1.42.1, run locally.
• The hosted and authoring entrypoint tests each drive the real entrypoint and check what reaches begin(). A customer secret is masked under a platform name, a credential name and an arbitrary name, and a name in both files masks both values. The model and the project come through.
  • The tests run through the real begin() and register(), with fake keys and a closed local port, so nothing leaves the machine.
  • T2b checks the real exporter at the transport.
  • The skip is the traceai-driven test, because traceai-claude-agent-sdk isn't installed here or in CI. It passed with the local traceAI 0.1.0 source on the path.
• The hosted entrypoint tests, run whole on the base and on this change, give identical results: 156 passed and 6 failed (livekit, the same on the base).
• CI clean-install on the pull_request run fails one speed budget, test_contract_model.py::test_a_crafted_file_of_block_comments_does_not_hold_up_the_check[a-file-of-openers] (0.5 s), at about 0.54–0.66 s. That file isn't in this PR. The base fails it on its own push run, 477efe88 (https://github.com/future-agi/agent-learning-kit/actions/runs/43169044737, 0.84 s, plus test_a_quote_that_grows_when_normalised_costs_little). This PR's own push run passes (https://github.com/future-agi/agent-learning-kit/actions/runs/43173801082).
• Planted bugs: 39, each run on both fi versions. All are caught on both, except one that only applies to fi 1.0.0 (the _default_processor line), which is caught on 1.0.0. The bugs include:
  • the wrapper bypassed;
  • add_span_processor used;
  • the run-time check removed or moved before the instrumentors;
  • each text field left raw;
  • tuples skipped;
  • the attribute keys passed to _redact;
  • each fail-closed path broken;
  • the cut rule moved, or a length or the ... dropped;
  • dropped counts lost;
  • the turn-off steps removed;
  • parent, kind, link context or event time lost;
  • a wrong endpoint in the real exporter.
• Fix-only checks after Dileep's fixes: three rounds, each with two fresh reviewers. The first two weren't clear: the declared values masked model names and the project, and then customer values under platform names went out. The rule now fails closed, and the third check cleared with no High. The chunking commit was dropped after measurement (see Known limits).
• Reviews: two design rounds, one plan round and one change round, each with two fresh reviewers. Their Mediums and Lows are folded in:
  • the run-time check moved after the instrumentors;
  • turning off empties the processor tuple before shutdown;
  • the copy's structure tested;
  • the real exporter tested;
  • event names cleaned;
  • no empty POST.

Known limits

• CI never runs fi 1.0.0, the version the hosted image installs from uv.lock. CI resolves the newest fi (1.2.2 today). This PR's 1.0.0 run is local, in the kit's venv. A CI step installing with uv export --frozen would close the gap; it isn't added here.
• traceai's names and cuts come from the 0.1.0 source, because no 0.1.1 wheel was on disk:
  • claude_agent.tool.input and .output (_attributes.py:50-51);
  • the 5,000 cut (_hooks.py:68, :88; tool input :167, output :266);
  • 2,000 (_client_wrapper.py:91, :356);
  • 1,000 (_hooks.py:200; _client_wrapper.py:381, :386);
  • 500 (_hooks.py:196, :284, :356);
  • the embedded 100 (_client_wrapper.py:412, which the URL cleaner refuses whole).
• Honest loss: a value of exactly a cut length that ends in an address loses that address, for example a 500-character Bash command ending curl -s https://api.example.com/health.
• TH-9500's patterns (bare keys, path tokens) reach these spans through the same _redact call once that lands, with no change here.
• Cleaning costs CPU in the voice process. On a heavy batch (512 spans × 20 attributes × 5,000 URL-dense characters), cleaning on the export thread adds about 10 ms per asyncio loop wake (0.8 ms idle; macOS, one run). The GIL changes hands every 5 ms, so the stall follows the total cleaning CPU, and a span-count bound doesn't help. Chunking was tried and dropped, because it didn't lower the stall and it multiplied the retry wait while Observe is down. A fix means less cleaning CPU, or cleaning outside the process's GIL. This is still open on Dileep's thread.
• Left to other tickets: the SDK's own run telemetry (fi/alk/telemetry) goes through only declared environment values today (TH-9530). The hosted chat workspace checkpoint is a file channel (TH-9521).

Related Issues

Refs: TH-9504
Related: #278 (TH-9491), #225 (TH-8804), TH-9500, TH-9530, TH-9521

[cdileep23]

Please check the comments.

[cdileep23]

P3. The cleaning runs as CPU work on the export thread, in the same process as voice runs. A batch of 512 spans, each with 20 attributes of 5,000 URL-dense characters, took 4.1 s in one export call locally. That holds the GIL away from the audio pipeline.

The input is extreme, but traceai fills 5,000-character tool outputs routinely.

Fix: skip strings without :// before they reach _redact (most text has none), or cap the batch size for this processor.

[khushalsonawat]

Not changed, with a reason. _redact_text already returns at once for a string with no '://' (agent.py on #278), so ordinary prose and most tool output cost almost nothing. Your 4.1 s batch is 512 spans × 20 attributes × 5,000 URL-dense characters, about 50 MB of URLs. Skipping strings without '://' here would also skip TH-9500's bare-key patterns once they land, and TH-9500 brings its own literal pre-check with a speed budget of under 1 s per 5 MiB. If you want a hard bound, I'd cap the processor's batch size rather than skip text; say if so.

[khushalsonawat]

A correction and an update. My reply above is out of date: with TH-9500 (#308), the cleaner also runs key patterns on text without ://, so skipping those strings isn't free any more.

I tried chunking the export (64 spans per call, with a yield between calls). It didn't help, so I've dropped it. A reviewer measured it with your shape: 512 spans × 20 attributes × 5,000 URL-dense characters, an asyncio loop with a 1 ms timer, and the export in another thread (macOS, one run each):

	Export time	Mean stall per wake	Worst stall
idle loop	–	0.8 ms	–
one call (this PR)	8.3 s	11.2 ms	25.4 ms
64-span chunks with a yield	11.5 s	10.9 ms	100.7 ms

• Why it doesn't help: CPython already hands the GIL over every 5 ms, so the stall follows the total cleaning CPU, and chunking doesn't reduce that. A span count doesn't bound the work either: one span holding a 2 MB value took 0.64 s on its own.
• What it costs: with Observe unreachable, each chunk gets the exporter's own retry budget. 256 spans took 26.7 s to flush instead of 6.6 s.

So your point is still open. On a heavy batch, cleaning adds about 10 ms per loop wake on the voice process. A real fix has to cut the cleaning CPU itself, or move it out of the process's GIL. Batch size can't fix it. I've added it to the PR's known limits. I'm leaving this thread unresolved for you.