Conversation
`fetchXSource` treated every failure except a rate limit as evidence about the account. So a provider outage, a dead session, or no provider being configured at all incremented `error_count` — and `markCrawlFailure` retires a feed at ten consecutive failures. An X source polls on a five-minute floor. Ten strikes is fifty minutes. That makes the ordinary order of operations destructive: setting `X_ENABLED=true` before a provider is reachable is how you find out whether it is reachable, and it would have marked every X source dead within the hour, logging "these accounts are broken" rather than "nothing is configured to collect with". The comment above the line already claimed the correct behaviour — that a deleted, suspended or protected account is the one failure genuinely about the source. The code did the opposite. Now only `XNoSuchSource` reaches `markCrawlFailure`; everything else returns `throttled`, which moves `next_fetch_at` and leaves `status`, `error_count`, `last_error` and `last_success_at` untouched. Retry intervals by cause, because they are three different guesses about when the situation changes: a rate limit uses the interval the server named; an outage waits twenty minutes; "no provider configured" waits an hour, since nothing changes until somebody deploys something. Five tests, four of which fail against the merged code. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q6QEgpuS4MLamogXtr2ZX6
ralyodio
added a commit
that referenced
this pull request
Aug 29, 2026
…ner (#158) X needs a collector and there was none. Rather than a second Railway service, RSSHub runs inside the poller — the only process in this system that collects anything — reachable on 127.0.0.1:1200. That is a smaller thing than a service beside it: no second billed container, no private-networking hop (Railway's outbound IPv6 is opt-in per service and fails as a fast 504, which is unpleasant to diagnose), and no public surface to remember to keep unexposed. The web service is untouched; it never collects, so it never needs any of this. Supervised, and its death is never the crawler's. RSSHub is a large third-party app talking to a hostile upstream and it will crash; a crash costs one restart with backoff and nothing else. If it never starts at all, X sources simply reschedule — #157 is what makes that safe. The image copies RSSHub's published artifact rather than resolving it: the npm package is only ever `1.0.0-master.<sha>` with 83 direct deps including a Playwright fork that downloads a browser. Three things were verified against RSSHub's own source rather than guessed, having been wrong on the first attempt: - the entry is `dist/index.mjs`, not `lib/index.js` (they publish no bin) - the base must be `node:24-trixie-slim`, matching theirs. It was `node:22-alpine`: musl cannot load glibc-linked artifacts and Node 22 cannot load a Node 24 ABI addon, and both fail as an image that builds cleanly and dies on first require - `--max-http-header-size=32768`, because X sends headers over Node's default and the failure reads like a broken upstream The bundled Chromium is deleted again — hundreds of megabytes for routes that scrape pages, and every route this poller asks for is an API call. Honest limits: the image build cannot be exercised here (no container runtime on this box), so the Dockerfile is verified by reading RSSHub's Dockerfile and package.json, not by building. The poller now runs a Node major CI does not exercise. And it collects nothing until X_SESSIONS carries a real X login. 7 tests on the supervisor; all 12 packages green. Claude-Session: https://claude.ai/code/session_01Q6QEgpuS4MLamogXtr2ZX6 Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
ralyodio
added a commit
that referenced
this pull request
Aug 29, 2026
…le (#159) Two more namespaces, and they cost very different amounts — which is the useful part of this change rather than an aside. Instagram is X's shape: no feeds, collected through the RSSHub daemon already running in the poller, mirrored here. It was a canonical.js, a fetch.js and a route, because the namespace machinery already existed. Facebook is not. Three doors, measured rather than assumed: - facebook.com/feeds/page.php?format=rss20 answers 404 (removed) - mbasic.facebook.com/<page> answers 200 with a login wall - RSSHub carries a thousand namespaces and no Facebook one The only remaining route is Meta's Graph API, which returns a Page's posts only to somebody who ADMINISTERS that Page — reading a stranger's public Page needs a permission Meta grants rarely and only after review. So /fb/ takes no open submissions: a Page appears once its operator connects it via FB_PAGE_TOKENS, and an unconnected Page is told plainly that it is not collectable rather than shown a button that does nothing. Nothing here drives a logged-in session against that login wall. The rule from #157 is now shared rather than per-platform. src/failure.js owns "only a source that does not exist may count against the source", and X was refactored onto it rather than Instagram and Facebook being written against a copy. Its 60 tests passed unchanged, which is the evidence the refactor was behaviour-identical. Two bugs the tests caught, both mine: - identify.js said in a comment that X is tried before Instagram and had Instagram first, so a bare @handle resolved to Instagram - the Facebook collector took its display name from the ref, which is lowercased because it is an identity — so every item title read "somepage:" instead of "SomePage:". Display now comes from feed_url, which kept the original casing; identity still comes from the ref Also: an uncrawled social row now renders its canonical name instead of whatever the OPML import called it. 50,026 subreddits are imported and a few hundred crawled, so /r/programming was headed "reddit.com". A render decision, not a rewrite of stored titles — the crawler owns those. 1,345 tests across 12 packages, all green. Claude-Session: https://claude.ai/code/session_01Q6QEgpuS4MLamogXtr2ZX6 Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fetchXSourcetreated every failure except a rate limit as evidence about the account. A provider outage, a dead session, or no provider configured at all incrementederror_count— andmarkCrawlFailureretires a feed at ten consecutive failures.An X source polls on a five-minute floor. Ten strikes is fifty minutes.
That makes the ordinary order of operations destructive: setting
X_ENABLED=truebefore a provider is reachable is how you find out whether it is reachable, and it would have marked every X source dead within the hour — logging "these accounts are broken" rather than "nothing is configured to collect with".The comment above the line already claimed the correct behaviour. The code did the opposite. Now only
XNoSuchSourcereachesmarkCrawlFailure; everything else returnsthrottled, which movesnext_fetch_atand leavesstatus,error_count,last_errorandlast_success_atexactly as they were.Retry intervals by cause — three different guesses about when the situation changes:
Five new tests; four fail against the merged code and pass here, including the exact scenario above. All 12 packages green.
Found while checking what switching X on would actually do: there is no RSSHub, Teapot or Nitter service in the Railway project yet, so the "no provider configured" path is the one prod would have taken.
🤖 Generated with Claude Code
https://claude.ai/code/session_01Q6QEgpuS4MLamogXtr2ZX6