Skip to content

fix: repair placeholder-only answers instead of completing them - #224

Draft
atlas-coding-agent wants to merge 2 commits into
devhims:mainfrom
atlas-coding-agent:fix/reject-placeholder-only-answers
Draft

atlas-coding-agent wants to merge 2 commits into
devhims:mainfrom
atlas-coding-agent:fix/reject-placeholder-only-answers

Conversation

@atlas-coding-agent

@atlas-coding-agent atlas-coding-agent commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

A Spanish session finished as Completed / Answered and charged 2 credits. The answer contained only "…" followed by citation chips, and reloading the page showed the same thing.

  • Session: c36be06c-9388-4eaf-bbea-c6004d7115d4
  • Run: f60dc202-9320-490b-bc2f-a372f031dd35, Oct 23 2026, 18:33 to 18:39 IST

Expected: a real answer, or an honest failure. Actual: a placeholder saved as a successful answer.

Diagnosis on 3d11c12 (upstream main)

fillerOnlyAnswer is the only content check before commit. It matches only an English fragment ("The") or an English promise ("I'll look up…"). A block of "…" with declared evidenceIds passes the schema's /\S/ check and the filler check. renderStructuredAnswer then appends the declared citations. The citation gate in buildAgentTurnResult sees resolvable citations and accepts the answer.

What changed

A block now counts as filler when its rendered prose has no letter or digit in any script, after the renderer's own [cite:…] markers are removed. The check runs the same alias, range, code, link and literal handling that rendering uses. A supported range like ref_1 through ref_2 therefore counts as citations only. Inline code, quoted literals, URLs and unsupported ranges count as visible text.

Filler answers keep their existing contract: one repair, then the existing failure "could not pass the answer validation checks after repair". The research path keeps its existing evidence fallback. One real block is still enough.

Tests

  • youtube-agent-structured-answer.test.ts covers placeholders such as …, citation-only text, zero-width text and range-only text like ref_1 through ref_2 and ref_1 to ref_2. It also covers text that must still pass: inline code ref_1, quoted "(ref_1)", indented code, links, URLs, an unsupported range, and short Spanish, numeric and CJK answers.
  • In youtube-agent-unified-finalizer.test.ts, a Spanish run with a cited … gets repaired and commits the real answer. A repeated placeholder fails without calling finalize.
  • CI on 48bf915: Platform checks ran 2153 tests, 2153 passed, 59 skipped, 0 failed. All 8 workflow jobs passed. The previous head ce99ac1 failed 3 existing literal-context tests, and this head fixes them without changing any existing assertion.

Verification limits

The sandbox's npm registry returned 403 for every package, so I could not run vitest or tsc locally. CI is the only full test run. Vercel reports "Authorization required to deploy", which is an access issue. I did not test against the live dashboard. The PR does not change completed runs that are already saved.

Separate issue, not changed here: the output-limit plus repair-timeout failure in run 2ad29797-b041-4748-b1e1-54fa99a65ebe has a different cause. Its missing receipt overlaps #183.

@vercel

vercel Bot commented Oct 24, 2026

Copy link
Copy Markdown

@kiro-agent is attempting to deploy a commit to the devhims' projects Team on Vercel.

A member of the Team first needs to authorize it.

devhims commented Oct 24, 2026

Copy link
Copy Markdown
Owner

Review verdict: changes needed for head ce99ac1c9178e0e29d7706f17a2c082d84847901.

The narrow ellipsis repair is appropriate for session c36be06c-9388-4eaf-bbea-c6004d7115d4, run f60dc202-9320-490b-bc2f-a372f031dd35, but the new detector and existing renderer disagree about what is a citation versus answer content:

  1. P2: legitimate literal/code answers are newly rejected — platform/src/agents/structured-answer.ts:56-65. CITATION_MARKUP strips aliases from the entire raw block, including protected literal/code contexts. For example, an answer consisting of inline code ref_1, or the quoted literal "(ref_4)", is now classified as a placeholder although the renderer intentionally preserves that content. This can waste the repair attempt or fail an otherwise valid literal answer. The exact-head Platform CI confirms the regression: the existing literal-context tests for "(ref_4 through ref_7)" and "(ref_4)", and the trailing indented-code test for [ref_2405, ref_2406], fail with the new placeholder ZodError. Preserve the renderer's literal/code handling rather than deleting those test expectations.

  2. P2: a supported bare range still bypasses the chips-only guard — platform/src/agents/structured-answer.ts:56-65 (compare resolveAliasForms at 215-234 and rendering at 270). For a block ref_1 through ref_2 with both aliases supplied, the detector removes only the endpoints, leaving through, and accepts it as answer text. The existing renderer resolves that whole valid range into [cite:e1] [cite:e2], so the resulting answer still has citations but no prose. ref_1 to ref_2 has the same issue. Check the same citation-normalized, literal-aware content that rendering uses; add regression tests for bare range-only answers as well as preserved literals.

Verification: independently executed the new regex classification on ellipsis, bare ranges, literal aliases, Spanish text and numeric answers, and inspected the alias/render and repair paths. This was a focused logic probe, not a full local application test. Exact-head CI Platform checks passed typechecking but failed unit tests: 3 failed, 2139 passed, 59 skipped. No live dashboard retest was run. Vercel's separate author-authorization failure is infrastructure/access, not evidence of a code defect. Keep this draft until the findings are fixed and the latest head's required checks are validated.

devhims commented Oct 24, 2026

Copy link
Copy Markdown
Owner

Re-review of head 48bf9156c5f0de2a54bd9223a4fb0b1d718d9e83.

Code review: no remaining material findings in this change. Overall verdict: changes needed / validation blocked, not yet approved for merge, solely pending clarification or resolution of the failed Vercel gate described below. No further code change is requested by this review.

Both P2 findings from the previous head are resolved:

  • platform/src/agents/structured-answer.ts:55-64,257-278: placeholder detection now uses the same inline-citation normalization as rendering, passes the supplied alias map, and preserves Markdown code/link and literal contexts. The previously failing quoted-literal and indented-code tests were retained and pass.
  • Supported bare ranges such as ref_1 through ref_2 and ref_1 to ref_2 are normalized before the content check, so their separator words can no longer masquerade as answer text. Added tests cover these range-only answers, preserved literals, and behavior without supplied aliases.

Verification: inspected the full PR and follow-up diff, existing citation protections, and repair flow. Exact-head Verify #577 completed successfully across all eight jobs, including both Dashboard E2E shards, Auth E2E, Platform integration, Web, documentation, library, and Platform checks. Platform logs report 2153 unit tests passed and 59 skipped; the relevant structured-answer (171), citation-range (33), and unified-finalizer (111) test files all passed, and typechecking passed. These are independently inspected CI results, not a claim that I ran the full suite locally. No live dashboard retest was performed, and saved historical answers are not backfilled.

Remaining validation blocker: Vercel reports failure for this exact head because the PR author needs team authorization. This is an access/deployment gate, not a code/test failure. The connector does not establish whether this check is required by branch policy, so I am withholding merge-ready approval until the repository owner confirms it is non-required or resolves it. I have not authorized a deployment or changed repository settings.

Scope remains the placeholder-only completion issue from session c36be06c-9388-4eaf-bbea-c6004d7115d4, run f60dc202-9320-490b-bc2f-a372f031dd35. This does not establish resolution of the separate output-limit/repair-timeout issue or broad Agent reliability.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants