Reddit at /r/, X at /x/, and a provider stack behind X

.env.example

@@ -46,3 +46,44 @@ CARD_BACKFILL_SECONDS=20
 # Nonessential maintenance can be paused while a large first-crawl backlog owns
 # the database write path. The work is resumable when these are switched on.
 CLUSTER_BACKFILL=1
+
+# --------------------------------------------------------------------- X / Twitter
+# X publishes no feeds, so posts are collected through a provider and mirrored
+# at /x/<handle>. Off by default: with this unset nothing is collected, existing
+# X feeds keep serving what they hold, and no source is marked unhealthy for it.
+X_ENABLED=false
+
+# Failover order. The provider never appears in a public URL, so this can change
+# under a live subscriber without their reader noticing.
+X_PRIMARY_PROVIDER=rsshub
+X_FALLBACK_PROVIDERS=teapot,official
+
+# Self-hosted, alongside the app, and not exposed publicly. A provider whose
+# base URL is unset is skipped rather than guessed at.
+RSSHUB_BASE_URL=
+RSSHUB_ACCESS_KEY=
+TEAPOT_BASE_URL=
+
+# The official API: the only provider that costs money per request, hence the
+# caps. 0 means unlimited.
+X_API_BEARER_TOKEN=
+X_API_DAILY_READS=0
+X_API_MONTHLY_READS=0
+X_API_MAX_RPM=0
+
+# Logged-in X sessions for the unofficial providers, as JSON:
+#   [{"id":"x-1","authToken":"...","ct0":"..."}]
+#
+# These are a full login to an X account - whoever holds them can post as it and
+# change its password. They live here rather than in a table so that a leaked
+# database dump carries none of them; x_sessions holds health state only. Use
+# dedicated accounts, and separate ones for production and development.
+#
+# The positional pair X_AUTH_TOKENS / X_CT0_TOKENS also works and is a trap:
+# the lists are matched by index, so removing one dead account from the middle
+# of the first and forgetting the second pairs every later token with the wrong
+# cookie. Prefer the JSON form, which cannot express that.
+X_SESSIONS=
+
+X_FETCH_TIMEOUT_MS=15000
+X_SESSION_COOLDOWN_SECONDS=900

README.md

@@ -23,6 +23,7 @@ packages/feed/   Feed discovery, RSS/Atom/JSON Feed parsing, OPML, SSRF guards
 packages/ingest/ Submit + crawl orchestration
 packages/db/     Turso/libSQL client, migrations and every query
 packages/notify/ Alerts — web push, email digests and webhooks
+packages/social/ X and Reddit: canonical identity, and X's provider adapters
 ```
 
 Everything outside the Next app is plain ESM with JSDoc types — no build step, so Docker stays
@@ -56,6 +57,98 @@ pnpm --filter @rssamplifier/db migrate
 | `/discoveries/{id}` | Progress of one keyword run: what was added, and why the rest was not |
 | `/crawlstats` | Crawler and discovery queues, live (`/crawlstatus` redirects here) |
 | `/random` | Redirect to a random blog — the toolbar's ✦ |
+| `/r` | Every subreddit and Reddit user in the directory |
+| `/r/<subreddit>` | One community: `/r/programming`, `.rss` `.atom` `.json` `.md` |
+| `/r/u/<user>` | One Reddit user, under the same prefix |
+| `/x` | Every X account, search and list in the directory |
+| `/x/<handle>` | One timeline: `/x/OpenAI`, plus `/replies` and `/media` |
+| `/x/list/<id>` | One X list |
+| `/x/search?q=` | An X search as a feed — X's own operators pass through |
+| `/x/status` | Which provider is collecting X, and how it is doing |
+
+### The two social namespaces
+
+Reddit and X both live under a prefix of their own, and for the same reason from
+opposite directions. Reddit publishes real RSS, so a subreddit resolves down the
+ordinary path and lands as an untyped row at a slug of its own — which is how
+50,099 of them ended up filed among the blogs. X publishes nothing at all, so
+without a provider it is not submittable in the first place.
+
+`packages/social` answers one question for both: **what is the canonical identity
+of this thing?** `@OpenAI`, `x.com/OpenAI` and `https://twitter.com/openai/` are
+one source (`x:user:openai`, at `/x/OpenAI`); `/r/programming`, `/r/Programming/`
+and `/r/programming/new/.rss` are one community (`r:sub:programming`, at
+`/r/programming`). One identity means one row, which means **one polling job no
+matter how many people subscribe** — the thing that matters most when the
+upstream rate-limits per account.
+
+A social source is an ordinary row in `feeds`, not a parallel `sources` table.
+That is what lets it inherit dedupe, backoff, interval learning, keyword
+extraction, full-text search, alerts, sitemaps and all five syndication formats
+without a line of X-specific code in any of them — and it is why an X post can
+appear in `/topics/artificial-intelligence.rss` beside a blog post with nothing
+to tell them apart.
+
+`/{slug}` still answers for both, and always will: it is the permanent identity
+of a row and links already point at it. The `/r/` and `/x/` address is the
+canonical one, which is what search engines are told.
+
+## Collecting X
+
+X has no feeds, so posts are collected through a provider and mirrored here.
+**Which provider never appears in a public URL** — that is the whole design.
+A reader subscribed to `/x/OpenAI.rss` through RSSHub is still subscribed when
+the collection method is replaced under them.
+
+```
+RSSHub (primary) → Teapot (fallback) → official X API (paid) → cached items
+```
+
+Failover is per attempt and the order is fixed; three failures in a row set a
+provider aside for a few minutes and one success clears it. Nothing here can
+empty a feed: items are only ever written on success, so an outage leaves
+yesterday's posts exactly where they were and the public route goes on serving
+them.
+
+| Variable | Default | What |
+| --- | --- | --- |
+| `X_ENABLED` | `false` | The kill switch. Off means no collection; existing feeds keep serving. |
+| `X_PRIMARY_PROVIDER` | `rsshub` | First provider tried. |
+| `X_FALLBACK_PROVIDERS` | `teapot,official` | Tried in order after it. |
+| `RSSHUB_BASE_URL` | — | A self-hosted RSSHub. Unset means the provider is skipped. |
+| `RSSHUB_ACCESS_KEY` | — | If that instance requires one. |
+| `TEAPOT_BASE_URL` | — | A Teapot or Nitter-shaped instance. |
+| `X_API_BEARER_TOKEN` | — | Official API. Unset means that provider is skipped. |
+| `X_API_DAILY_READS` | `0` | Spend cap, `0` for unlimited. Also `X_API_MONTHLY_READS`, `X_API_MAX_RPM`. |
+| `X_SESSIONS` | — | JSON: `[{"id":"x-1","authToken":"…","ct0":"…"}]` |
+| `X_FETCH_TIMEOUT_MS` | `15000` | Upstream deadline. |
+| `X_SESSION_COOLDOWN_SECONDS` | `900` | How long a rate-limited session rests. |
+
+**Session credentials live in the environment and never in a table.**
+`auth_token` and `ct0` are a full login to an X account — whoever holds them can
+post as it and change its password. `x_sessions` and `x_provider_state` hold
+health state only, so a leaked database dump carries no credentials, and nothing
+renderable about a session is one. Use dedicated accounts, and separate ones for
+production and development.
+
+§47 of the PRD also allows the positional pair `X_AUTH_TOKENS` / `X_CT0_TOKENS`.
+It works, and it is a trap worth naming: the lists are matched by position, so
+deleting one dead account from the middle of the first and forgetting the second
+silently pairs every later token with the wrong cookie — a pool of sessions that
+all authenticate as nobody. `X_SESSIONS` cannot express that state.
+
+Both unofficial providers keep their own logged-in sessions by default, so the
+pool has nothing to hand them unless the deployment exposes a per-request
+parameter (`RSSHUB_SESSION_PARAM`, `TEAPOT_SESSION_HEADER`). Nothing here guesses
+at those names: guessing would mean putting a live cookie on a query string
+somebody else's instance might log.
+
+**Not built:** the `/admin/x` buttons of §34 — disable a provider, clear a
+cooldown, force a refresh. This codebase has no notion of an administrator to
+guard them with, and shipping the levers before the lock is how a kill switch
+becomes a way for anyone to turn collection off. `X_ENABLED` and
+`X_PRIMARY_PROVIDER` cover the two that matter without a code deploy.
+`/x/status` is the read-only half, and is where those buttons go.
 
 ## Agent endpoints
 

apps/poller/package.json

@@ -12,6 +12,7 @@
     "@rssamplifier/discover": "workspace:*",
     "@rssamplifier/feed": "workspace:*",
     "@rssamplifier/ingest": "workspace:*",
-    "@rssamplifier/notify": "workspace:*"
+    "@rssamplifier/notify": "workspace:*",
+    "@rssamplifier/social": "workspace:*"
   }
 }

apps/poller/src/index.js

@@ -5,6 +5,7 @@ import {
   q,
   accounts,
   alerts,
+  social,
   takeWriteTally,
   warmStatsCache,
   warmDirectoryCache,
@@ -24,6 +25,7 @@ import {
 import { runDueSources, discoverFromOwnTopics } from '@rssamplifier/discover';
 import { findFeedCard } from '@rssamplifier/feed';
 import { deliverAlerts, vapidConfig } from '@rssamplifier/notify';
+import { createXRuntime, xEnabled } from '@rssamplifier/social';
 
 import { createRecorder, toEntry, writeFailure } from './log.js';
 
@@ -294,7 +296,12 @@ async function tick() {
       batchSize,
       concurrency,
       publishLog ? recorder.record : null,
-      { perHost },
+      // `crawl` is handed to crawlFeed as-is. The X runtime travels here rather
+      // than being built per feed because it is the thing that *remembers*:
+      // which provider is in cooldown, which session is resting. Rebuilt per
+      // crawl it would be a system with no memory, rediscovering the same
+      // outage on every source in the batch.
+      { perHost, crawl: { xRuntime } },
     );
     if (crawled || failed) {
       // The backlog is the number worth watching: crawled/failed only say the
@@ -578,6 +585,42 @@ try {
   process.exit(1);
 }
 
+/**
+ * The X collection runtime, built once and shared by every crawl in this process.
+ *
+ * Built after the migration, because `hydrate()` reads `x_provider_state` and
+ * `x_sessions` — the tables that migration creates — to restore cooldowns
+ * across a redeploy. Without that a service that redeploys ten times in a day
+ * forgets ten outages and re-walks into each of them.
+ *
+ * `null` when X is switched off, and that is a first-class state rather than a
+ * failure: `crawlFeed` sees no runtime and reschedules its X sources without
+ * touching their health, so existing feeds keep serving what they hold and
+ * nothing is retired while the integration is off (§42's kill switch).
+ *
+ * Logged either way. "The X sources stopped updating" is the kind of thing that
+ * goes unnoticed for a week, and a boot line saying `x=false` is what turns
+ * that into a five-second answer — the same reason the push half of the alerts
+ * stack prints `push=true` here.
+ */
+const xRuntime = xEnabled(env)
+  ? await createXRuntime({
+      env,
+      providerStore: social.providerStore(db),
+      sessionStore: social.sessionStore(db),
+      // Straight onto the same live log the crawl writes to, so a failover or a
+      // session cooldown appears on /crawlstats beside the crawl it affected
+      // rather than in a stream nobody has open (§35).
+      onEvent: publishLog ? (event, fields) => recorder.record(toEntry(event, fields)) : null,
+    })
+  : null;
+
+log('x-runtime', {
+  enabled: Boolean(xRuntime),
+  providers: xRuntime ? xRuntime.registry.candidates().map((p) => p.name) : [],
+  sessions: xRuntime ? xRuntime.sessions.size : 0,
+});
+
 /**
  * Key the items stored before the grouping column existed.
  *

apps/web/next.config.mjs

@@ -206,6 +206,43 @@ const nextConfig = {
         // The directory itself: what was added to it, newest first.
         { source: '/feed.:format(rss|atom|json|xml|md)', destination: '/api/directory/feed/:format' },
 
+        // The two social namespaces.
+        //
+        // No playlist spellings, for the same reason /following has none: a
+        // timeline and a subreddit carry no enclosures, so an `.m3u` of one
+        // would be an empty file with a confident name.
+        //
+        // Ordered narrowest-first within each prefix, because `:username` and
+        // `:subreddit` match anything: every fixed address under /x has to be
+        // named before /x/:username, and /r/u before /r/:subreddit. A rewrite
+        // parameter never spans a slash, so the two-segment rules cannot be
+        // shadowed by the one-segment ones — but the fixed segments can be, and
+        // silently.
+        {
+          source: '/x/search.:format(rss|atom|json|xml|md)',
+          destination: '/api/x/search/feed/:format',
+        },
+        {
+          source: '/x/list/:listId.:format(rss|atom|json|xml|md)',
+          destination: '/api/x/list/:listId/feed/:format',
+        },
+        {
+          source: '/x/:username/:mode(replies|media).:format(rss|atom|json|xml|md)',
+          destination: '/api/x/:username/:mode/feed/:format',
+        },
+        {
+          source: '/x/:username.:format(rss|atom|json|xml|md)',
+          destination: '/api/x/:username/feed/:format',
+        },
+        {
+          source: '/r/u/:username.:format(rss|atom|json|xml|md)',
+          destination: '/api/r/u/:username/feed/:format',
+        },
+        {
+          source: '/r/:subreddit.:format(rss|atom|json|xml|md)',
+          destination: '/api/r/:subreddit/feed/:format',
+        },
+
         // One category of it. The segments are the category pages' own paths,
         // duplicated from CATEGORIES in apps/web/src/lib/categories.js — this
         // file is evaluated before the workspace resolves, so it cannot import

apps/web/package.json

@@ -19,6 +19,7 @@
     "@rssamplifier/mail": "workspace:*",
     "@rssamplifier/notify": "workspace:*",
     "@rssamplifier/search": "workspace:*",
+    "@rssamplifier/social": "workspace:*",
     "@rssamplifier/translate": "workspace:*",
     "@simplewebauthn/browser": "^13.0.0",
     "@swc/helpers": "^0.5.23",

apps/web/src/app/AddSocialSource.jsx

@@ -0,0 +1,68 @@
+import { siteUrl } from '../lib/db.js';
+
+/**
+ * What `/r/somewhere` or `/x/somebody` shows when nobody has added it yet.
+ *
+ * A 404 would be the easy answer and the wrong one. The address is well formed,
+ * the thing at the other end almost certainly exists, and the visitor has
+ * already told us exactly what they want by typing it — so the page offers to
+ * add it rather than telling them they were wrong to ask.
+ *
+ * A plain `<form method="post">` to `/api/submit`, like every other control on
+ * this site: it works with JavaScript off, and the endpoint answers an HTML
+ * caller with a 303 back to the source's own page. Nothing is fetched from X or
+ * Reddit while the visitor waits — the row is written, the poller collects on
+ * its next tick, and this page is replaced by the real one within the minute
+ * (§17, §37).
+ *
+ * @param {{ network: 'x'|'reddit', label: string, input: string, canonical: string }} props
+ *   `input` is what gets submitted — the canonical upstream URL, not what was
+ *   typed, so the source that gets created is the one this page is about.
+ */
+export default function AddSocialSource({ network, label, input, canonical }) {
+  const platform = network === 'x' ? 'X' : 'Reddit';
+
+  return (
+    <main className="prose">
+      <h1>{label}</h1>
+
+      <p>
+        Nobody has added this {platform} source to the directory yet. Add it and RSS Amplifier
+        will start collecting it — usually within a minute.
+      </p>
+
+      <form method="post" action="/api/submit">
+        <input type="hidden" name="input" value={input} />
+        <button type="submit">Add {label} to the directory</button>
+      </form>
+
+      <p>
+        Once it is here, it will be at{' '}
+        <code>
+          {siteUrl()}
+          {canonical}
+        </code>{' '}
+        in every format this site publishes:{' '}
+        <code>.rss</code>, <code>.atom</code>, <code>.json</code> and <code>.md</code>. That
+        address does not change, whatever we have to do behind it to keep collecting.
+      </p>
+
+      {network === 'x' ? (
+        <p>
+          X publishes no feeds of its own, so this is collected on your behalf and mirrored here.
+          Protected accounts are not collected, and posts arrive as fast as we can read them
+          rather than in real time.
+        </p>
+      ) : (
+        <p>
+          Reddit publishes its own feed for this, and we read it on a schedule and keep a copy —
+          so the address above works whether or not Reddit is answering right now.
+        </p>
+      )}
+
+      <p>
+        <a href={network === 'x' ? '/x' : '/r'}>Browse what is already here</a>
+      </p>
+    </main>
+  );
+}