Skip to content

Run RSSHub as a supervised daemon inside the crawler container - #158

Merged
ralyodio merged 1 commit into
mainfrom
x-rsshub
Aug 29, 2026
Merged

Run RSSHub as a supervised daemon inside the crawler container#158
ralyodio merged 1 commit into
mainfrom
x-rsshub

Conversation

@ralyodio

Copy link
Copy Markdown
Contributor

X needs a collector and there was none. Rather than a second Railway service, RSSHub runs inside the poller — the only process in this system that collects anything — on 127.0.0.1:1200.

That is a smaller thing than a service beside it: no second billed container, no private-networking hop (Railway's outbound IPv6 is opt-in per service and fails as a fast 504, which is unpleasant to diagnose), and no public surface to remember to keep unexposed. The web service is untouched — it never collects, so it never needs any of this.

Supervised, and its death is never the crawler's. RSSHub is a large third-party app talking to a hostile upstream and it will crash. A crash costs one restart with backoff and nothing else — not the crawl, not the queue, not the 50,026 Reddit feeds that have nothing to do with X. If it never starts at all, X sources simply reschedule; #157 is what makes that safe.

Three things I got wrong first, then checked against RSSHub's own source

Assumed Actually
entry lib/index.js dist/index.mjs — they publish no bin
base node:22-alpine must be node:24-trixie-slim, matching theirs
default header size --max-http-header-size=32768

The base is the one that would have hurt: musl cannot load glibc-linked artifacts and Node 22 cannot load a Node 24 ABI addon, and both fail as an image that builds cleanly and dies on first require. All three are now pinned by a test.

The image copies RSSHub's published artifact rather than resolving it — the npm package is only ever 1.0.0-master.<sha> with 83 direct deps including a Playwright fork that downloads a browser. The bundled Chromium is deleted again: hundreds of megabytes for routes that scrape pages, and every route this poller asks for is an API call.

Honest limits

  • The image build is not exercised here. There is no container runtime on this box, so the Dockerfile is verified by reading RSSHub's Dockerfile and package.json, not by building it. Watch the first deploy.
  • The poller now runs Node 24, which CI (Node 22) does not exercise. The suite passes on both and 22 is the stricter runner, so the gap is in the safe direction — but it is a gap.
  • It collects nothing yet. X_ENABLED is still false and X_SESSIONS is empty; RSSHub answers every timeline with nothing until it has a real X login.

7 tests on the supervisor — restart-with-backoff, backoff reset after a healthy run, no respawn during shutdown, and that the session cookie never reaches a log line. All 12 packages green.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Q6QEgpuS4MLamogXtr2ZX6

X needs a collector and there was none. Rather than a second Railway
service, RSSHub runs inside the poller — the only process in this system
that collects anything — reachable on 127.0.0.1:1200.

That is a smaller thing than a service beside it: no second billed
container, no private-networking hop (Railway's outbound IPv6 is opt-in
per service and fails as a fast 504, which is unpleasant to diagnose), and
no public surface to remember to keep unexposed. The web service is
untouched; it never collects, so it never needs any of this.

Supervised, and its death is never the crawler's. RSSHub is a large
third-party app talking to a hostile upstream and it will crash; a crash
costs one restart with backoff and nothing else. If it never starts at
all, X sources simply reschedule — #157 is what makes that safe.

The image copies RSSHub's published artifact rather than resolving it:
the npm package is only ever `1.0.0-master.<sha>` with 83 direct deps
including a Playwright fork that downloads a browser.

Three things were verified against RSSHub's own source rather than
guessed, having been wrong on the first attempt:

- the entry is `dist/index.mjs`, not `lib/index.js` (they publish no bin)
- the base must be `node:24-trixie-slim`, matching theirs. It was
  `node:22-alpine`: musl cannot load glibc-linked artifacts and Node 22
  cannot load a Node 24 ABI addon, and both fail as an image that builds
  cleanly and dies on first require
- `--max-http-header-size=32768`, because X sends headers over Node's
  default and the failure reads like a broken upstream

The bundled Chromium is deleted again — hundreds of megabytes for routes
that scrape pages, and every route this poller asks for is an API call.

Honest limits: the image build cannot be exercised here (no container
runtime on this box), so the Dockerfile is verified by reading RSSHub's
Dockerfile and package.json, not by building. The poller now runs a Node
major CI does not exercise. And it collects nothing until X_SESSIONS
carries a real X login.

7 tests on the supervisor; all 12 packages green.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q6QEgpuS4MLamogXtr2ZX6
@ralyodio
ralyodio merged commit 124b30f into main Aug 29, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant