[khushalsonawat]

Summary

TH-8808 is part of T43: qualify the hosted harness across frameworks. This is Pydantic AI's offline half, on the same pattern as the OpenAI Agents SDK reference (#301). A local end-to-end run with a real reading and a real model is summarised under Verification.

• examples/harness/pydantic_ai_support/ is a small support agent on Pydantic AI 2.61.0, with the same three tools as test(harness): an OpenAI Agents SDK reference agent for the hosted path (TH-8800) #301 so the frameworks can be compared:
  • Callback: a module-level agent_callback for the kit's callback bridge.
  • Conversation: the kit's conversation becomes Pydantic AI's prompt plus message history, so each turn is sent once.
  • Persona: given as instructions=. A system_prompt= is added only while the history is empty, so it would vanish from the second turn on.
  • Model: built per call as an OpenAIChatModel on a client closed with the call. A plain openai: model string maps to the Responses API, which the model gateway serves only for OpenAI-format routes.
  • Tools: the agent returns content only. Pydantic AI runs its plain tools on worker threads, and the kit's tracer records each call.
  • A key with no base URL is refused: if OPENAI_API_KEY is set and OPENAI_BASE_URL isn't, the callback raises before building a client, naming the missing setting.
• tests/harness/test_pydantic_ai_reference.py pins what the hosted path does with it, using a stand-in for the library (it isn't a kit dependency):
  • Detection: it's detected as pydantic_ai (run folder, subfolder, in place), and a shadowing pydantic_ai.py changes the result.
  • Build: the build runs it through the callback bridge, with the tracer bound to exactly its three tools, each checked against the example's real, plain function.
  • Tracer: the real tracer bootstrap records each tool as 2.61.0 calls it, and nothing for a same-named function in another module.
  • Glue: the callback's history conversion, persona, per-call client, content-only reply and error propagation.

No kit code changes. The tests use the shared helpers in tests/harness/reference_agents.py.

Stacked on #301 (feat/th-8800-openai-agents-reference): it uses #301's shared test helpers, so #301 merges first, with a merge commit.

Verification

• ruff check on the new files
• Focused pytest, run locally on Python 3.12.3: the new file, plus test_framework_detection.py, test_python_tool_trace_bootstrap.py and test_callback_http_adapter.py. 293 passed.
• Full uv run pytest -q: not run, because no shared runtime is touched.
• CI: sdk-smoke runs a fixed list of test files, which didn't include this one. It now does (tests/harness/test_pydantic_ai_reference.py).
• Fix-only checks: the fix made after the review cleared (the agent's comment on the Responses API) had a fix-only check: CLEAR, nothing above Low. The audit fixes (this CI-list line, and this description) had a fix-only check as well: CLEAR, nothing above Low.
• Local end-to-end run (12 Nov, a real reading, then the agent on vertex_ai/gemini-3.7-flash through the model gateway): it passes end to end. 3 right replies, and the trace file has the 3 calls with their arguments and results.
• Ten planted bugs, each caught by its named test. They include:
  • the new message also in history;
  • a dropped assistant turn;
  • an openai: model string;
  • a client built at import;
  • returned tool calls;
  • an async tool;
  • a shadowing pydantic_ai.py;
  • system_prompt= instead of instructions=.

Review round 1 (Dileep)

• The test file uses the shared helpers (from test(harness): an OpenAI Agents SDK reference agent for the hosted path (TH-8800) #301); the detection tests are one parametrised test, with a new shadowing case.
• The tracer subprocess has a timeout (180 s).
• The build test checks each tracer binding against the example's real function, so renaming a tool fails it.
• The example refuses a key given with no base URL, before building a client; the README says so, and says why agent-simulate is pinned.
• These had a fix-only check: CLEAR, with no findings (High 0, Medium 0, Low 0). Tests on Python 3.12.3: the Pydantic AI reference file (16 tests, from 13) and four neighbouring files, 310 passed; the planted bugs (a renamed tool, the refusal removed) were caught.

Known limits

• The tests use a stand-in for the library, because CI doesn't install it. Pydantic AI 2.61.0 itself ran only in the local end-to-end run.
• A customer's Pydantic AI agent written as openai:<model> and run on the gateway with a non-OpenAI model gets the gateway's 501, because it uses the Responses API. A guidance line to write openai-chat:<target_model>, next to fix(harness): let an OpenAI Agents SDK agent run on the model gateway (TH-8800) #303's, follows as a separate PR once fix(harness): let an OpenAI Agents SDK agent run on the model gateway (TH-8800) #303 merges (a register row).
• The agent's orders are its own state, not the run's world, as in test(harness): an OpenAI Agents SDK reference agent for the hosted path (TH-8800) #301 (a register row).

Checklist

• Public API names stay under agent_learning.*, agent-learn, or @future-agi/agent-learning-kit
• New user-facing workflow has docs or an example
• New release claim has executable gate coverage (no release claim)
• No hosted-service dependency was added to release-gated examples
• Security-sensitive output is redacted or justified (no credentials; the key and URL come from the environment)

[cdileep23]

Please check the comments.

[cdileep23]

P3. No timeout on this subprocess.run, so a hang in the driver hangs the whole suite. #305's equivalent uses timeout=180; add the same here.

[cdileep23]

P3. The bindings this asserts are the ones the test writes into the contract itself, and _source_tool_trace_environment copies them through unchanged, so the assert compares the test's input with itself. Renaming return_policy in the example leaves this test green (three other tests catch it).

Fix: assert the bindings against the example's real functions, or drop the bindings claim from the test's name.

[cdileep23]

P3. Same copied setup as the other reference-agent PRs (_copy_example, the stand-in writer, the fresh-import fixture, the detection tests, the build test skeleton, the tracer driver, the pin tests). See the shared-helper comment on #301: this file would import from that one module instead.

[cdileep23]

Please check the comments.

[cdileep23]

P3. This helper re-implements the shared fresh_import with its own sys.path and sys.modules handling. The example_module fixture above already gives a fresh import on the same stand-ins, and #305's build test uses its fixture this way. Taking example_module as a parameter of the build test and calling module = example_module() keeps all 16 tests green, still fails when a tool is renamed, and drops this helper and the importlib.util import.

The loop at lines 226-233 that checks each binding against the real function is also the same as #305's, and #301's build test still compares the bindings with its own input (test_openai_agents_reference.py:180). One assert_bindings_match(bindings, module, module_name) in reference_agents.py would serve all three.