Skip to content

feat(preview): add Workers AI Markdown fallback - #21

Merged
phatpham9 merged 2 commits into
mainfrom
feat/ai-preview-fallback
Nov 13, 2026
Merged

phatpham9 merged 2 commits into
mainfrom
feat/ai-preview-fallback

Conversation

@phatpham9

@phatpham9 phatpham9 commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Summary

  • Adds a Workers AI toMarkdown fallback for previews the built-in extractor cannot read: non-HTML documents (PDF/Office/CSV/XML), HTML pages with no readable blocks, and bodies truncated before content.
  • Core stays platform-free: fetchArticlePreview accepts an injected fallback, platforms/cloudflare/src/previewFallback.ts owns env.AI, and core/previewMarkdown.ts maps the converted Markdown into the same validated PreviewBlock shape.
  • The fallback runs only for no_content, too_large, and unsupported_content_type. blocked, timeout, rate_limited, upstream_error, and empty_body keep their existing paths.
  • Bounds: 5 MB non-HTML cap (skipped early via Content-Length; oversized documents now report too_large), 10-second AI timeout, existing 10-second fetch timeout, 120 blocks / 40k chars in the mapper. Kill switch: FEEDREADER_AI_PREVIEW_FALLBACK=false.
  • Bumps the preview cache key to preview:v10 so previously cached empty previews get a fallback attempt.
  • Updates README, ARCHITECTURE, DEPLOYMENT, and RUNBOOK.

Verification

  • npm run typecheck
  • npx vitest run --exclude '.claude/**' — 14 files, 119 tests
  • npm run db:migrate:local
  • wrangler deploy --dry-run — 359.01 KiB / 95.58 KiB gzip
  • Local end-to-end:
    • arXiv PDF → 200, 19 blocks, 1.8–3.0 s, 14,865 tokens
    • 6.7 MB PDF → 502 too_large in ~0.2 s, no 5 MB download (Content-Length guard)
    • HTML page with no readable blocks → 200 via AI in ~0.3 s
    • Normal GitHub article → unchanged primary extraction, no AI call
  • PR preview deployment (pr-21-6fbbc54, shared production D1):
    • 385 KB NY Fed PDF → 200, 86 blocks (4.3 s first extraction; cache hit afterwards)
    • 6.65 MB Postgres PDF → 502 too_large in ~0.9 s
    • isocpp PDF → 502 blocked (upstream_status: 403; fallback correctly skipped)
    • GitHub release article → 200, 100 blocks, cache-control: private, no-store
  • The existing preview cache hit and the private, no-store client header still behave as before.

Notes

  • Workers AI document text conversion is free; the fallback disables AI image-description conversion and skips standalone images.
  • Best-effort: AI errors/timeouts return null and preserve the original preview failure reason.

🤖 Generated with Claude Code

PDFs, Office documents, and HTML pages the built-in extractor cannot
read now fall back to Workers AI's toMarkdown conversion. The converted
Markdown is mapped into the same validated PreviewBlock shape, with
unsafe links and images dropped; core stays platform-free behind an
injected fallback, and FEEDREADER_AI_PREVIEW_FALLBACK=false disables it.

Bump the preview cache key to v10 so previously cached empty previews
get a fallback attempt. Bound non-HTML downloads at 5 MB, skip oversized
documents using Content-Length, and race the AI call against 10 seconds.
Strip PDF filename/page-marker headings and trailing JSON-LD.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

🔎 Cloudflare preview: https://ece1c824-feedreader.phatpham9.workers.dev

Uploaded from 6fbbc544860d53fd072e0f3010b1be848be61563. This is a Worker version preview — it shares the production D1 database, so it reads/writes real data; it does not receive cron-triggered refreshes.

When the AI fallback is enabled but the document exceeds the 5 MB cap
(declared by Content-Length or discovered while reading), return
too_large instead of unsupported_content_type so the client shows the
size message. No fallback, unsupported formats, and failed conversions
still report unsupported_content_type.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@phatpham9
phatpham9 merged commit 0aebd36 into main Nov 13, 2026
3 checks passed
@phatpham9
phatpham9 deleted the feat/ai-preview-fallback branch November 13, 2026 05:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant