Skip to content

Gemini harness fix: mandatory self-verification pass for draft final answers - #107

Merged
allocsys merged 8 commits into
mainfrom
gemini-verification-pass
Aug 27, 2026
Merged

Gemini harness fix: mandatory self-verification pass for draft final answers#107
allocsys merged 8 commits into
mainfrom
gemini-verification-pass

Conversation

@allocsys

Copy link
Copy Markdown
Owner

Why

Independent of the parked Groq/GLM work (see plan.md's status notes — unaffected by this PR), Gemini has a real, observed accuracy bug in long investigations: on a 13-step live run, Gemini had the complete file in context from its very first tool call, but its final synthesis incorrectly claimed a specific implementation detail was false — it appears to have trusted a later, narrower github_search_code snippet over the complete file it had already read earlier in the same run.

What changed

  1. SYSTEM_PREAMBLE addition (connectors/gemini/agent_delegate.js): an explicit rule that a full/direct read (github_read_file, github_get_file_at_commit, notion_get_page, etc.) outranks a narrower/derived result about the same fact (a github_search_code snippet, a mem0_search match), even if the narrower result was fetched more recently.
  2. Mandatory one-time self-verification pass (new mechanism, not just a prompt tweak): the first time the model produces a draft final answer with step budget still remaining, the loop pushes the draft back with a new no-tools VERIFICATION_PROMPT instructing it to re-check every specific claim against the raw tool results already in the conversation, applying the same full-read-outranks-narrow-result rule. Whatever comes back from that second call is what's actually returned. If the draft answer is itself the final allowed step, the pass is skipped (no budget left to check twice).
  3. pendingVerification state, threaded through agent_checkpoint.js's save/load exactly like the existing repeat-tracking fields, so a run that dies mid-verification resumes back into the verification turn rather than silently re-entering normal tool-use.

This is provider-agnostic — it lives in the loop body, not gemini-specific code — so it applies identically to GLM/Groq whenever those are unparked, even though Gemini is the only provider it's actually exercised against right now.

Cost tradeoff

One extra provider call (and step) on every successful investigation that would otherwise have finished with budget to spare — e.g. a max_steps: 6 run that used to finish in 3 steps now finishes in 4. Deliberate accuracy-for-cost tradeoff, not a bug.

Tests

test/agent-delegate-loop.test.js updated: every existing test that reaches a draft final answer with steps remaining now expects the extra verification-pass providerChat call (step/call counts bumped by one); added a dedicated test confirming the verification pass can change the returned answer and that its call carries no tools; added a dedicated test confirming the pass is skipped when the draft answer is itself the final allowed step. Full suite green in CI: 24 files / 350 tests, eslint . clean.

Not yet done

This is a harness/prompt change, verified only against the mocked test suite so far. Per plan.md's own note, the actual claim this is meant to fix has not been re-validated against a live Gemini call yet. Next step after merge: re-run a comparable investigation task live and confirm (a) the verification pass fires and completes, and (b) it either catches a similar contradiction or the original bug doesn't recur.

@allocsys
allocsys merged commit 757bf25 into main Aug 27, 2026
2 checks passed
@allocsys
allocsys deleted the gemini-verification-pass branch August 27, 2026 14:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant