Skip to content

fix(sources): fetch Hugging Face papers from the JSON API - #22

Merged
phatpham9 merged 1 commit into
mainfrom
fix/huggingface-daily-papers-api
Nov 13, 2026
Merged

phatpham9 merged 1 commit into
mainfrom
fix/huggingface-daily-papers-api

Conversation

@phatpham9

@phatpham9 phatpham9 commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Summary

Production Hugging Face refresh returned 429 with a 1.4 MB HTML challenge page, which was stored in sync_state.last_error and exposed by /healthz.

  • Switch the Hugging Face source from scraping /papers/trending (~1.4 MB HTML) to https://huggingface.co/api/daily_papers (~272 KB JSON, same 50 entries).
  • Send a descriptive User-Agent and Accept header on all source requests.
  • Honor Retry-After on 429s (capped at 5 seconds); otherwise use 1s/3s backoff for rate limits and the existing short backoff for 5xx.
  • Cap error details at 200 characters; HTML error pages become upstream returned an HTML error page before they reach D1, /healthz, or the UI.
  • Add parser, retry, header, and error-sanitization tests; update README/ARCHITECTURE/DEPLOYMENT/RUNBOOK.

Verification

  • npm run typecheck
  • npx vitest run --exclude '.claude/**' — 14 files, 126 tests
  • npm run db:migrate:local
  • wrangler deploy --dry-run
  • Cloudflare-egress smoke check: a temporary branch-preview route called the new HF source from a Worker running on Cloudflare's network — 200, 50 daily papers, 1.76s. The temporary route was removed and its branch deleted; it is not part of this diff.
  • The Hugging Face API also returned 200 locally with the same 50 entries.

🤖 Generated with Claude Code

Hugging Face returned a 429 HTML challenge page for the 1.4 MB
/papers/trending scrape, and the raw body ended up in sync_state and
/healthz. Read the same 50 daily-paper entries from
/api/daily_papers (about 272 KB of JSON) instead.

Source requests now send a descriptive User-Agent and Accept header,
honor Retry-After on 429s, and cap error details at 200 characters.
HTML error pages are reduced to a fixed message before they reach D1,
the health payload, or the UI.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

🔎 Cloudflare preview: https://70d154f5-feedreader.phatpham9.workers.dev

Uploaded from da6a831b55e84a1475c497e729d6afc7a509d1da. This is a Worker version preview — it shares the production D1 database, so it reads/writes real data; it does not receive cron-triggered refreshes.

@phatpham9
phatpham9 merged commit 562871b into main Nov 13, 2026
3 checks passed
@phatpham9
phatpham9 deleted the fix/huggingface-daily-papers-api branch November 13, 2026 07:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant