Skip to content

feat(harness): report the agent's framework in the agent document - #229

Merged
khushalsonawat merged 1 commit into
feat/TH-8804-source-subfolderfrom
feat/TH-8805-framework-detection
Nov 4, 2026
Merged

khushalsonawat merged 1 commit into
feat/TH-8804-source-subfolderfrom
feat/TH-8805-framework-detection

Conversation

@khushalsonawat

Copy link
Copy Markdown
Member

Stacked on #225 (TH-8804), itself on #224 (TH-8806). This PR's base is feat/TH-8804-source-subfolder, so its diff shows only TH-8805. It retargets to dev once #225 merges.

Summary

Every build's agent document now says which framework the agent uses: extension.framework = {name, version}, plus label for other. The answer comes only from reading files, so the customer's code is never imported or run and no model is asked. The same detectors will be reused by TH-8950.

Ticket: TH-8805.

What the code shows name version
Builds with one of the 10 agreed frameworks, after the layering and voice rules that framework installed version, or null
One of 9 other known agent frameworks, and none of the 10 other, with label installed version, or null
Two unrelated frameworks, imported but never built with, a limit hit, no Python, or a failure unknown null
Would be none, but something was left unread unknown null
Everything read, and no framework none null
  • Tokens, not a parse tree. A tree took about 1 GB on a 1 MB file. Tokens take 20 to 41 MB.
  • Limits, not a clock. 64 MiB read, 5,000 Python files and a shared work counter, so the same files give the same answer on every retry.
  • A failure can't cost the run. Any failure gives unknown, and the document still goes out. The field counts in setup_hash.
  • The folder follows subfolder, then the contract's runtime.workdir, then the control process's working directory, then the source.

Known limits

  • Nothing reads the field yet. Platform #3661 at 5f6dbdf4b has no reader, and a user can't correct a wrong name until that lands.
  • Timings are laptop time. The worst case is about 480 s on 3.12.3, against the 3,720 s sandbox lifetime. Guest speed is unmeasured, so the margin holds if the guest is within about 7x of the laptop.
  • Memory at worst is about 1.1 GB: 858 MB measured in a 4 GiB container, plus an estimated 200 MB for the harness. Agent processes sharing the container weren't measured with it.
  • folder_findings can let a MemoryError out. Only detect_framework promises not to, so TH-8950 needs its own guard if it calls the first directly.
  • Reported none, though a framework is there: ones outside the lists (CAMEL, OpenAI Swarm), dynamic imports, and from livekit import *.
  • Reported unknown where a person could tell: a data table of several MiB can use up the work limit, and so can a repo over 64 MiB.
  • Docker agents report version: null, because there are no installed packages to read. Agents already built show nothing until their next version.

Differences from the ticket

  • Names are the Studio contract's 10, not the FRAMEWORK_PRESETS keys (owner's decision).
  • other covers a fixed list of 9 (owner's decision).
  • The tracing check isn't changed. TH-8977 moves it to the token scanner.

Verification

  • ruff check on the changed source files and the new test file. Full repo not run.
  • Focused pytest, the listed files on 3.11 and 3.12.14: 7 failed, 652 passed and 1 failed, 658 passed. The failures are the base's by name: 6 need livekit on 3.11, and test_grade's cap test fails on both.
  • 3.12.3, the sandbox's version, detector and scenario source: 274 passed.
  • 75 planted bugs, each caught by a named test.
  • Full uv run pytest -q: not run on the laptop. CI on this PR is the full run of sdk-smoke.
  • Round 3 of review was clear (High 0, Medium 2, Low 4). Not seen by any reviewer: its two Medium test additions and Low fixes, and the landing on feat(harness): build an agent from a repo's subfolder #225, which calls agent_folder for subfolder.

- Write extension.framework = {name, version} on every build, plus label
  for other, beside the contract.
- Read the agent's Python as tokens, never importing or running it.
- Report one of the 10 agreed names, other with a label from 9 known
  frameworks, none for plain code, or unknown when it can't decide.
- Read the version only from the build's installed package files.
- End in unknown on a limit: 64 MiB read, 5,000 Python files, or a shared
  work counter, with no clock, so the answer is the same every time.
- Keep detection failures away from the document and the run.

TH-8805

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
@khushalsonawat khushalsonawat self-assigned this Nov 2, 2026
@khushalsonawat
khushalsonawat merged commit a62389f into feat/TH-8804-source-subfolder Nov 4, 2026
11 checks passed
@khushalsonawat
khushalsonawat deleted the feat/TH-8805-framework-detection branch November 4, 2026 05:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant