[khushalsonawat]

Summary

TH-8811 is part of T43: qualify the hosted harness across frameworks. This is CrewAI's offline half, on the same desk as #301, #304, #305 and #306. The real run comes later; the local end-to-end run of 12 Nov didn't include CrewAI.

• examples/harness/crewai_support/ uses CrewAI 1.15's own chat surface: a standalone Agent whose kickoff_async(messages) takes the conversation (the last user message is the request; earlier turns are history), served through the kit's callback bridge.
  • Built per call: the Agent and its LLM are built on each call, because CrewAI's agent keeps per-run state and the bridge serves requests on parallel threads.
  • Model: LLM(model=MODEL, custom_openai=True, base_url=OPENAI_BASE_URL, api_key=OPENAI_API_KEY), CrewAI's native OpenAI-compatible client on chat completions, with the model name unchanged. custom_openai=True stops CrewAI switching to the Responses API after a "responses-only" error, which the gateway can't serve.
  • Telemetry: CrewAI's telemetry and first-run tracing are turned off before CrewAI is imported (CREWAI_DISABLE_TELEMETRY, CREWAI_TRACING_ENABLED=false). OTEL_SDK_DISABLED is not used, because it would turn off all OpenTelemetry in the process.
  • Tools: the same three tools as the other references, plain functions wrapped with CrewAI's tool() per call.
• tests/harness/test_crewai_reference.py uses a stand-in for CrewAI, since CrewAI isn't a kit dependency.
  • Detection: detected as crewai, with a crewai.py shadow flipping it.
  • Build: the callback bridge with the generic tracer. No ALK_CREWAI_TOOL_TRACE (see the known limits).
  • Tracer: the real tracer bootstrap records each tool as CrewAI calls it.
  • Glue: the conversation is sent once, the agent and model are built per call, the telemetry defaults are set before the import, a missing base URL fails the turn, and errors propagate.

No kit code changes.

Stacked on #301 (feat/th-8800-openai-agents-reference): it uses #301's shared test helpers, so #301 merges first, with a merge commit.

Review round 1 (Dileep)

• The test file uses the shared helpers in tests/harness/reference_agents.py (from test(harness): an OpenAI Agents SDK reference agent for the hosted path (TH-8800) #301); the four detection tests are one parametrised test, and the 21 tests are otherwise unchanged.
• The agent's comment and the README give the real reason for custom_openai=True.
• A README line says agent-simulate is pinned for the kit's callback bridge (fi.simulate.AgentInput).
• These had a fix-only check: CLEAR, one Low (a README sentence about plain openai/ strings was broader than CrewAI 1.15.33's routing). That sentence now says which names can switch to the Responses API, and a second fix-only check of that change was CLEAR, with no findings. Tests on Python 3.12.3: the CrewAI reference file and five neighbouring files, 322 passed; two planted bugs caught.

Overlap

examples/harness/multi_agent_compatibility/crewai/ (the CrewAI 1.6.1 multi-agent reference) is stale for CrewAI 1.15. Its string model "openai/vertex_ai/..." can't reach the gateway on 1.15, and its litellm==1.101.0 pin conflicts with 1.15's extra (<1.100.5). It's named here and not changed.

Verification

• ruff check on the new files
• Focused pytest, run locally on Python 3.12.3: the new file, framework detection, the tracer bootstrap, the callback adapter and the CrewAI observer tests. 308 passed.
• Full uv run pytest -q: not run, because no shared runtime is touched.
• CI: sdk-smoke runs a fixed list of test files, which didn't include this one. It now does (tests/harness/test_crewai_reference.py).
• Fix-only checks: the fix made after the review cleared (a comment on why the stand-in's str() differs from .raw) had a fix-only check: CLEAR, with one Low (a two-line comment), accepted. The audit fixes (this CI-list line, and this description) had a fix-only check as well: CLEAR, nothing above Low.
• Fifteen planted bugs, each caught by its named test. They include:
  • custom_openai dropped;
  • a model string;
  • the agent built at import or cached;
  • the new message sent twice;
  • a renamed or missing tool;
  • an async tool;
  • telemetry set after the import;
  • OTEL_SDK_DISABLED;
  • agent-simulate dropped;
  • a crewai.py shadow;
  • the sync kickoff.

Known limits

• Strict schemas: CrewAI sends function tools with "strict": true, and whether the model gateway's Gemini route accepts it is unverified until the real run.
• Observer: the kit's CrewAI observer is switched on only for one-shot command agents, so this chat agent's tool calls are recorded by the generic tracer: names, arguments and raw return values, without the observer's agent and task fields (a register row).
• Stand-in only: no CI job installs CrewAI. The 1.15.33 source was read in review, and the real run proves it.
• Unclosed clients: each turn leaves CrewAI's two SDK clients unclosed (CrewAI exposes no close; a register row).
• Shared state: the agent's orders are its own state, not the run's world, as in test(harness): an OpenAI Agents SDK reference agent for the hosted path (TH-8800) #301 (a register row).

Checklist

• Public API names stay under agent_learning.*, agent-learn, or @future-agi/agent-learning-kit
• New user-facing workflow has docs or an example
• New release claim has executable gate coverage (no release claim)
• No hosted-service dependency was added to release-gated examples
• Security-sensitive output is redacted or justified (no credentials; the key and URL come from the environment)

[cdileep23]

Please check the comments.

[cdileep23]

P3. On CrewAI 1.15.33 this isn't what happens. LLM(model="openai/vertex_ai/gemini-3.7-flash", base_url=..., api_key=...) goes to the native OpenAICompletion with model vertex_ai/gemini-3.7-flash, the gateway URL and api="completions" (llm.py:512-523). LiteLLM is picked only when the URL comes from the environment alone.

custom_openai=True is still right, for a different reason: it stops CrewAI switching to the Responses API after a "responses-only" error (openai/completion.py:751). Reword this comment, the README line and the description to give that reason.

[cdileep23]

P3. Same copied setup as the other reference-agent PRs (_copy_example, the stand-in writer, the fresh-import fixture, the detection tests, the build test skeleton, the tracer driver, the pin tests). See the shared-helper comment on #301: this file would import from that one module instead.

[cdileep23]

Please check the comments.

[cdileep23]

P3. The comment now gives only the Responses API reason and dropped the main one for this example's default model. vertex_ai/gemini-3.7-flash has no openai/ prefix and vertex_ai is not in CrewAI 1.15.33's PROVIDER_ALIASES (crewai/llm.py:211-229), so without custom_openai=True the name goes to LiteLLM whatever base_url says (llm.py:500-525), and LiteLLM isn't installed here, so LLM() fails.

Fix: add a clause, e.g. "and sends a non-openai/ name such as vertex_ai/... to the OpenAI-compatible client at all, where it would otherwise go to LiteLLM, which isn't installed". In the README narrow "an openai/ string whose URL comes only from the environment goes to LiteLLM" to "an openai/ name CrewAI doesn't know".

[cdileep23]

P3. Moving to the shared helper weakened the shadow check: it now passes when CrewAI is still detected with a blocker (reference_agents.py:197, assert not (found == expected and blockers == frozenset())), where the old test asserted detect_framework(run) != CREWAI. The PR body says the tests are otherwise unchanged. Fix in the helper (see the comment on #301).