Gemini harness fix: mandatory self-verification pass for draft final answers - #107
Merged
Conversation
…r resuming mid self-verification-pass
… + self-verification-pass mechanism
…swer with budget remaining
…heckpoint call sites
…im against main (pre-merge), and why it's inconclusive
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Independent of the parked Groq/GLM work (see plan.md's status notes — unaffected by this PR), Gemini has a real, observed accuracy bug in long investigations: on a 13-step live run, Gemini had the complete file in context from its very first tool call, but its final synthesis incorrectly claimed a specific implementation detail was false — it appears to have trusted a later, narrower
github_search_codesnippet over the complete file it had already read earlier in the same run.What changed
SYSTEM_PREAMBLEaddition (connectors/gemini/agent_delegate.js): an explicit rule that a full/direct read (github_read_file,github_get_file_at_commit,notion_get_page, etc.) outranks a narrower/derived result about the same fact (agithub_search_codesnippet, amem0_searchmatch), even if the narrower result was fetched more recently.VERIFICATION_PROMPTinstructing it to re-check every specific claim against the raw tool results already in the conversation, applying the same full-read-outranks-narrow-result rule. Whatever comes back from that second call is what's actually returned. If the draft answer is itself the final allowed step, the pass is skipped (no budget left to check twice).pendingVerificationstate, threaded throughagent_checkpoint.js's save/load exactly like the existing repeat-tracking fields, so a run that dies mid-verification resumes back into the verification turn rather than silently re-entering normal tool-use.This is provider-agnostic — it lives in the loop body, not gemini-specific code — so it applies identically to GLM/Groq whenever those are unparked, even though Gemini is the only provider it's actually exercised against right now.
Cost tradeoff
One extra provider call (and step) on every successful investigation that would otherwise have finished with budget to spare — e.g. a
max_steps: 6run that used to finish in 3 steps now finishes in 4. Deliberate accuracy-for-cost tradeoff, not a bug.Tests
test/agent-delegate-loop.test.jsupdated: every existing test that reaches a draft final answer with steps remaining now expects the extra verification-passproviderChatcall (step/call counts bumped by one); added a dedicated test confirming the verification pass can change the returned answer and that its call carries notools; added a dedicated test confirming the pass is skipped when the draft answer is itself the final allowed step. Full suite green in CI: 24 files / 350 tests,eslint .clean.Not yet done
This is a harness/prompt change, verified only against the mocked test suite so far. Per plan.md's own note, the actual claim this is meant to fix has not been re-validated against a live Gemini call yet. Next step after merge: re-run a comparable investigation task live and confirm (a) the verification pass fires and completes, and (b) it either catches a similar contradiction or the original bug doesn't recur.