Repository navigation
test: send real agent turns through AIMock; fix empty tool results - #59
Merged
Merged
Conversation
AIMock started on every test run but no active test sent a model call to
it: the one agent test was skipped, and default models go through the
Vercel AI Gateway, which AIMock can't mock.
- tests/helpers/aimock.ts points agents at direct anthropic/ ids with a
dummy key and reads AIMock's request journal.
- agent-turns.test.ts runs History agent turns end to end: plain reply,
a tool round trip, PII redaction and a provider error.
- Those tests caught a real bug: five custom tools returned
{ type: "text", text } from toModelOutput. The AI SDK reads `value`,
so the model always got an empty tool result. Fixed in search_history,
create_dashboard, save_document, preview_app and create_automation,
with a unit test pinning the shape.
- aimock.json and fixtures now use Foreman's real tool names and SDK
parameter names.
- Remove the skipped provider-swap test, the AIMock lines in the web
Playwright config that pointed at missing files, and the upstream
annotations issue draft (fixed upstream). Raise the AIMock floor to
^1.45.1, the version already locked.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YaNZt6oJYNXatfpSYMnNp5
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
hamchowderr
marked this pull request as ready for review
October 20, 2026 14:21
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Chowderr · project thread
Before: AIMock started on every test run, but no active test sent a model call to it. The one agent test was skipped, and the default models go through the Vercel AI Gateway, which AIMock can't mock. Five custom tools also returned their summary to the model in a field the AI SDK ignores, so after search_history, create_dashboard, save_document, preview_app or create_automation ran, the model saw an empty tool result.
After: four History agent turns run end to end through AIMock: a plain reply, a tool round trip, PII redaction and a provider error. Those tests caught the empty-result bug, which is now fixed in all five tools, and a unit test pins the correct output shape.
How:
tests/helpers/aimock.tspoints agents at directanthropic/...model ids with a dummy key, and reads AIMock's request journal so tests can check what was sent.tests/fixtures/aimock/agent-turns.json.aimock.jsonand the older fixtures now use Foreman's real tool names and the SDK's parameter names.toModelOutputnow returns{ type: "text", value }instead of{ type: "text", text }.annotationsissue draft, since that bug is fixed upstream.^1.45.1, the version already locked.rules/testing-and-db.mdon both points.Checks:
npm test(562 passed),tscfor agents and web, Biome, andverify:aimock. The new tests fail onmainand pass here.Generated by Claude Code