Skip to content

Tune the AI wrapper and act on the independent review - #49

Merged
TechNomadCode merged 10 commits into
mainfrom
fix/ai-review-findings
Oct 22, 2026
Merged

TechNomadCode merged 10 commits into
mainfrom
fix/ai-review-findings

Conversation

@TechNomadCode

Copy link
Copy Markdown
Owner

Tunes how the app talks to the AI and fixes four findings from an independent review of that code.

What changes

  • Prompt version 6: the instructions cut by a quarter; an unsupported claim is saved as the product manager's own decision; the search described as it works.
  • Searches left: each search result tells the AI how many searches it has left.
  • Transport: the AI's reasoning is sent back as reasoning and tool results carry their tool's name, as Cloudflare's own provider sends them. A check of every recorded batch showed Cloudflare read the reasoning on every call before this change too; the earlier "sometimes dropped" finding was an error in that check, now corrected.
  • Broken replies refused: a reply that is nearly all one repeated character (Cloudflare twice sent 4,096 "!") now shows the usual error with "Try again", and the measurement counts it as not run.
  • Size bounds: a conversation is refused above 100,000 characters of what reaches the AI, an AI reply above 8,000, and a request body above 300,000. A full nine-section PRD still fits.
  • Reranker tried and switched off: calibrated on Cloudflare (55 recorded searches, 2,310 pairs) against 327 hand-labelled pairs, bge-reranker-base neither separated relevant passages from unrelated ones nor ordered them better than the search alone. The search returns the five closest passages, as before. The code stays, with its scores read as Cloudflare sends them and a threshold of 0.
  • Measurement tool: each batch's cost estimate comes from its own measurement's answers, and the reranker calibration command (npm run measure -- --calibrate-rerank).

Files in src/server and why

  • chat.ts, prompts/system-prompt.ts: prompt version 6, the searches-left note, the size bounds.
  • ai/cloudflare.ts: the transport, the broken-reply guard.
  • search.ts, ai/cloudflare-reranker.ts, ai/fake-reranker.ts, ai/reranker.ts, ai/providers.ts, ai/models.ts: the reranker, now off.
  • measurement/*: cost estimate, calibration, broken replies counted as not run.
  • src/app/api/chat/route.ts: the body bound.

Checks (local)

  • npm run verify: 554 unit tests and the production build passed.
  • npm run measure:fake: every request ran on the fake model.
  • Measured on the real model only together with the switch to gpt-oss-20b (lean-20b, 25 of 30 runs, on its own branch): dark mode 3 of 4 against 0 of 5 in the baseline, grocery delivery 3 of 4 against 3 of 5. Not run here: the end-to-end tests.

TechNomadCode and others added 10 commits October 22, 2026 03:59
…d one's

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he visitor's decision

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@TechNomadCode
TechNomadCode merged commit 7aa27dd into main Oct 22, 2026
5 checks passed
@TechNomadCode
TechNomadCode deleted the fix/ai-review-findings branch October 22, 2026 16:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant