Repository navigation
fix(sources): fetch Hugging Face papers from the JSON API - #22
Merged
Merged
Conversation
Hugging Face returned a 429 HTML challenge page for the 1.4 MB /papers/trending scrape, and the raw body ended up in sync_state and /healthz. Read the same 50 daily-paper entries from /api/daily_papers (about 272 KB of JSON) instead. Source requests now send a descriptive User-Agent and Accept header, honor Retry-After on 429s, and cap error details at 200 characters. HTML error pages are reduced to a fixed message before they reach D1, the health payload, or the UI. Co-Authored-By: Claude Code <noreply@anthropic.com>
|
🔎 Cloudflare preview: https://70d154f5-feedreader.phatpham9.workers.dev Uploaded from |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Production Hugging Face refresh returned
429with a 1.4 MB HTML challenge page, which was stored insync_state.last_errorand exposed by/healthz./papers/trending(~1.4 MB HTML) tohttps://huggingface.co/api/daily_papers(~272 KB JSON, same 50 entries).User-AgentandAcceptheader on all source requests.Retry-Afteron429s (capped at 5 seconds); otherwise use 1s/3s backoff for rate limits and the existing short backoff for 5xx.upstream returned an HTML error pagebefore they reach D1,/healthz, or the UI.Verification
npm run typechecknpx vitest run --exclude '.claude/**'— 14 files, 126 testsnpm run db:migrate:localwrangler deploy --dry-run200, 50 daily papers, 1.76s. The temporary route was removed and its branch deleted; it is not part of this diff.🤖 Generated with Claude Code