Skip to content

fix: harden tool approval decisions and predicates - #1344

Closed
mattapperson wants to merge 3 commits into
mainfrom
fix/approval-hardening
Closed

mattapperson wants to merge 3 commits into
mainfrom
fix/approval-hardening

Conversation

@mattapperson

@mattapperson mattapperson commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes three bugs in the human-in-the-loop tool approval flow. Each one lets an approval-gated tool run without the approval it was meant to require.

  • Approval predicate runs against raw model arguments before the input schema is applied #520: predicates see raw model arguments. toolRequiresApproval passed JSON.parse output to requireApproval predicates. The tool's inputSchema (defaults, coercions, transforms) was applied only later, at execution time. A model could omit a field defaulted to a sensitive value (e.g. destructive: z.boolean().default(true)), so the predicate saw undefined and skipped approval while the executor ran with true. Call-level and tool-level predicates now receive the inputSchema output, including type-changing transforms. The original call is left unchanged so the executor's own parse doesn't apply transforms twice. If the arguments fail validation, the raw arguments are passed through; the executor rejects such calls before execute runs.
  • Approval decisions are not de-duplicated, so an approval-gated tool can execute twice #518: decisions aren't de-duplicated. An ID listed twice in approveToolCalls executed the tool twice. An ID in both arrays executed it and recorded a rejection. ModelResult now throws at construction, before any tool runs or state changes, next to the existing "decisions require a state accessor" check.
  • Resuming an awaiting_approval conversation with no decision advances it without recording an approval #517: resuming without a decision advances the conversation. The resume path ran only when a decision was passed. With none, a persisted awaiting_approval conversation was saved as in_progress and a new model request was sent. Resuming now always goes through decision processing. With no decisions every call stays pending, so the conversation stays paused, using the existing "still pending" path.

Fixes #517
Fixes #518
Fixes #520

Behavior changes

  • Passing duplicate or conflicting approval IDs now throws instead of running tools twice.
  • Resuming a paused conversation without decisions no longer sends a request; requiresApproval() stays true. Any new input passed on that call is not sent until a decision is made.
  • On such a resume, getText() and getResponse() throw Conversation is awaiting approval for tool calls: <name> (<id>). Pass approveToolCalls or rejectToolCalls to continue. instead of the generic Response not available.

Testing

  • New tests/unit/approval-hardening.test.ts (12 tests):
    • Schema defaults and type-changing transforms reach both predicate types, and the original call isn't modified.
    • Each invalid decision shape throws, including from the ModelResult constructor.
    • A paused conversation resumed without decisions stays awaiting_approval, a recording fetcher sees no requests, and getResponse() explains the pending approval.
  • Against the original conversation-state.ts and model-result.ts, 7 of the first 10 tests fail; the other 3 are controls. The transform test also fails against this PR's first commit, which re-parsed the output.
  • Full unit suite: 223 passed, no type errors. Lint and build are clean.

Devin Review

- #520: run tool-call arguments through the tool's inputSchema before
  calling a call-level or tool-level requireApproval predicate, so the
  predicate sees the same defaults and coercions the executor runs
  with. The original call is not modified, so the executor's own parse
  does not apply transforms twice.
- #518: reject duplicate IDs within approveToolCalls or rejectToolCalls,
  and IDs present in both, when ModelResult is constructed, before any
  tool runs.
- #517: resuming an awaiting_approval conversation always goes through
  decision processing. With no decisions every call stays pending, so
  the conversation stays paused instead of sending a new model request.

Fixes #517
Fixes #518
Fixes #520
devin-ai-integration[bot]

This comment was marked as resolved.

The re-validation guard parsed the already-parsed arguments a second time.
A schema with a type-changing transform (e.g. a string transformed to a
boolean) fails that second parse, so predicates fell back to the raw
arguments and could allow a call the executor then ran with the
transformed value. Pass the inputSchema output whenever the first parse
succeeds.

Also explain a pending approval when getText() or getResponse() is called
on a resumed conversation that is still awaiting a decision, instead of
the generic "Response not available".

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 new potential issue.

Devin Review

Comment on lines +119 to +120
if (isCallApprovalCheck(callLevelCheck)) {
return callLevelCheck({ ...toolCall, arguments: args }, context);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟨 Invalid approval setting bypasses tool gate

When a JavaScript caller sets call-level requireApproval to true, isCallApprovalCheck ignores it instead of rejecting it. Tools without their own approval setting then execute without approval.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

@mattapperson

Copy link
Copy Markdown
Contributor Author

Closing: #1356 moved this code (model-result.ts, conversation-state.ts and the rest of callModel) to @openrouter/agent, so it can't merge here. The fixes are already covered in the agent repo:

One note for the agent code: a review here flagged that an exception thrown inside a tool's schema (e.g. a throwing transform or refine) can escape z4.safeParse in the approval check, so it fails the whole turn instead of becoming a tool error. The agent's toolRequiresApproval parses the same way, so it's worth checking there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant