Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -87,3 +87,21 @@ X_SESSIONS=

X_FETCH_TIMEOUT_MS=15000
X_SESSION_COOLDOWN_SECONDS=900

# ------------------------------------------------------------------- Instagram
# Collected through the same embedded RSSHub daemon as X, so it needs no base
# URL of its own — only a logged-in Instagram cookie on the RSSHub side.
# Accounts and hashtags only; stories expire and are not collected.
IG_COOKIE=

# -------------------------------------------------------------------- Facebook
# Facebook publishes no public feed, serves no page without a login, and has no
# bridge. The only route in is Meta's Graph API, which returns a Page's posts
# only to somebody who ADMINISTERS that Page - so /fb/ takes no open
# submissions, and a Page appears only once its operator connects it here.
#
# FB_PAGE_TOKENS=[{"page":"MyPage","token":"EAA..."}]
#
# A Page Access Token can post as the Page. Treat it exactly like the X session
# cookies above: vault, not a service env, and never a database column.
FB_PAGE_TOKENS=
57 changes: 54 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,17 +65,25 @@ pnpm --filter @rssamplifier/db migrate
| `/x/list/<id>` | One X list |
| `/x/search?q=` | An X search as a feed — X's own operators pass through |
| `/x/status` | Which provider is collecting X, and how it is doing |
| `/ig` | Every Instagram account and hashtag in the directory |
| `/ig/<handle>` | One account: `/ig/nasa`, and `/ig/tag/<tag>` for a hashtag |
| `/fb` | Facebook Pages whose operators have connected them |
| `/fb/<page>` | One connected Page |

### The two social namespaces
### The four social namespaces

Reddit and X both live under a prefix of their own, and for the same reason from
opposite directions. Reddit publishes real RSS, so a subreddit resolves down the
ordinary path and lands as an untyped row at a slug of its own — which is how
50,099 of them ended up filed among the blogs. X publishes nothing at all, so
without a provider it is not submittable in the first place.

`packages/social` answers one question for both: **what is the canonical identity
of this thing?** `@OpenAI`, `x.com/OpenAI` and `https://twitter.com/openai/` are
Instagram is X's shape again — no feeds, collected through RSSHub — and cost
almost nothing to add, which is the point of having built the namespace once.
Facebook is its own thing entirely; see below.

`packages/social` answers one question for all four: **what is the canonical
identity of this thing?** `@OpenAI`, `x.com/OpenAI` and `https://twitter.com/openai/` are
one source (`x:user:openai`, at `/x/OpenAI`); `/r/programming`, `/r/Programming/`
and `/r/programming/new/.rss` are one community (`r:sub:programming`, at
`/r/programming`). One identity means one row, which means **one polling job no
Expand All @@ -93,6 +101,49 @@ to tell them apart.
of a row and links already point at it. The `/r/` and `/x/` address is the
canonical one, which is what search engines are told.

## Facebook, and what `/fb/` can honestly be

**There is no way to read an arbitrary public Facebook Page.** Three doors, all
measured on 2026-08-29 rather than assumed:

- the old `facebook.com/feeds/page.php?format=rss20` endpoint answers **404** —
removed, not deprecated
- `mbasic.facebook.com/<page>` answers 200 with a **login wall**
- RSSHub, which carries a thousand namespaces and maintains Twitter and
Instagram, has **no Facebook namespace at all**

The one remaining door is Meta's Graph API, and it only opens for Pages the
caller **administers**. Reading somebody else's public Page needs the
`Page Public Content Access` feature, which requires App Review plus business
verification and is granted rarely.

So `/fb/` is the one namespace here that does not take open submissions: a Page
appears when its operator connects it, by putting a Page Access Token in
`FB_PAGE_TOKENS`. A Page nobody has connected is not "not crawled yet", it is
not collectable, and the page says exactly that rather than offering a button
that would quietly do nothing.

```bash
FB_PAGE_TOKENS='[{"page":"MyPage","token":"EAA..."}]'
```

What is deliberately absent: anything that drives a logged-in Facebook session
against that login wall. It breaks constantly, it is against Meta's terms, and
it would risk an account of ours to serve a directory nobody pays for.

## Collecting Instagram

Same shape as X, through the same RSSHub daemon — the `/instagram/2/…` web-api
routes, which authenticate with a cookie rather than the private-api routes,
which want a username and password.

```bash
IG_COOKIE= # on the RSSHub side; see apps/poller/src/rsshub.js
```

Accounts and hashtags only. Stories expire, and a feed of things that have
already gone is worse than no feed.

## Collecting X

X has no feeds, so posts are collected through a provider and mirrored here.
Expand Down
12 changes: 12 additions & 0 deletions apps/web/next.config.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -242,6 +242,18 @@ const nextConfig = {
source: '/r/:subreddit.:format(rss|atom|json|xml|md)',
destination: '/api/r/:subreddit/feed/:format',
},
{
source: '/ig/tag/:tag.:format(rss|atom|json|xml|md)',
destination: '/api/ig/tag/:tag/feed/:format',
},
{
source: '/ig/:username.:format(rss|atom|json|xml|md)',
destination: '/api/ig/:username/feed/:format',
},
{
source: '/fb/:page.:format(rss|atom|json|xml|md)',
destination: '/api/fb/:page/feed/:format',
},

// One category of it. The segments are the category pages' own paths,
// duplicated from CATEGORIES in apps/web/src/lib/categories.js — this
Expand Down
115 changes: 82 additions & 33 deletions apps/web/src/app/AddSocialSource.jsx
Original file line number Diff line number Diff line change
@@ -1,33 +1,82 @@
import { siteUrl } from '../lib/db.js';

/**
* What `/r/somewhere` or `/x/somebody` shows when nobody has added it yet.
* What `/r/somewhere`, `/x/somebody`, `/ig/somebody` or `/fb/SomePage` shows
* when it is not in the directory yet.
*
* A 404 would be the easy answer and the wrong one. The address is well formed,
* the thing at the other end almost certainly exists, and the visitor has
* already told us exactly what they want by typing it — so the page offers to
* add it rather than telling them they were wrong to ask.
* A 404 would be the easy answer and the wrong one for three of the four. The
* address is well formed, the thing at the other end almost certainly exists,
* and the visitor has already said exactly what they want by typing it — so the
* page offers to add it rather than telling them they were wrong to ask.
*
* A plain `<form method="post">` to `/api/submit`, like every other control on
* this site: it works with JavaScript off, and the endpoint answers an HTML
* caller with a 303 back to the source's own page. Nothing is fetched from X or
* Reddit while the visitor waits — the row is written, the poller collects on
* caller with a 303 back to the source's own page. Nothing is fetched from the
* platform while the visitor waits — the row is written, the poller collects on
* its next tick, and this page is replaced by the real one within the minute
* (§17, §37).
*
* @param {{ network: 'x'|'reddit', label: string, input: string, canonical: string }} props
* **Facebook is the exception, and says so.** There is no public feed, no
* unauthenticated HTML and no provider; the only way in is a Page Access Token
* from whoever administers the Page. Offering an "add" button there would be a
* button that quietly does nothing, so it gets an explanation instead.
*
* @param {{ network: 'x'|'reddit'|'instagram'|'facebook', label: string, input: string, canonical: string }} props
* `input` is what gets submitted — the canonical upstream URL, not what was
* typed, so the source that gets created is the one this page is about.
*/
export default function AddSocialSource({ network, label, input, canonical }) {
const platform = network === 'x' ? 'X' : 'Reddit';
const platform = PLATFORMS[network] ?? PLATFORMS.x;
const address = (
<>
<code>
{siteUrl()}
{canonical}
</code>{' '}
in every format this site publishes: <code>.rss</code>, <code>.atom</code>,{' '}
<code>.json</code> and <code>.md</code>
</>
);

if (network === 'facebook') {
return (
<main className="prose">
<h1>{label}</h1>

<p>
This Facebook Page is not connected, and unlike the rest of the directory it cannot be
added by anyone who happens to want it.
</p>

<p>
Facebook publishes no feed for a Page, serves no page without a login, and has no
third-party bridge we can use. The only remaining route is Meta&rsquo;s own Graph API,
and it will only return a Page&rsquo;s posts to somebody who <strong>administers that
Page</strong> — reading a stranger&rsquo;s public Page needs a permission Meta grants
rarely and only after review.
</p>

<p>
So if this is your Page, it can be connected: an administrator supplies a Page Access
Token and it appears here at {address}, collected on the same schedule as everything
else. If it is not your Page, there is nothing we can honestly offer — and we would
rather say that than mirror a scraper that breaks every few weeks.
</p>

<p>
<a href="/fb">What is connected</a> · <a href="/x">X</a> · <a href="/ig">Instagram</a> ·{' '}
<a href="/r">Reddit</a>
</p>
</main>
);
}

return (
<main className="prose">
<h1>{label}</h1>

<p>
Nobody has added this {platform} source to the directory yet. Add it and RSS Amplifier
Nobody has added this {platform.name} source to the directory yet. Add it and RSS Amplifier
will start collecting it — usually within a minute.
</p>

Expand All @@ -36,33 +85,33 @@ export default function AddSocialSource({ network, label, input, canonical }) {
<button type="submit">Add {label} to the directory</button>
</form>

<p>
Once it is here, it will be at{' '}
<code>
{siteUrl()}
{canonical}
</code>{' '}
in every format this site publishes:{' '}
<code>.rss</code>, <code>.atom</code>, <code>.json</code> and <code>.md</code>. That
address does not change, whatever we have to do behind it to keep collecting.
</p>
<p>Once it is here, it will be at {address}. That address does not change, whatever we have
to do behind it to keep collecting.</p>

{network === 'x' ? (
<p>
X publishes no feeds of its own, so this is collected on your behalf and mirrored here.
Protected accounts are not collected, and posts arrive as fast as we can read them
rather than in real time.
</p>
) : (
<p>
Reddit publishes its own feed for this, and we read it on a schedule and keep a copy —
so the address above works whether or not Reddit is answering right now.
</p>
)}
<p>{platform.note}</p>

<p>
<a href={network === 'x' ? '/x' : '/r'}>Browse what is already here</a>
<a href={platform.index}>Browse what is already here</a>
</p>
</main>
);
}

/** What to call each platform, and the one thing worth saying about it. */
const PLATFORMS = {
x: {
name: 'X',
index: '/x',
note: 'X publishes no feeds of its own, so this is collected on your behalf and mirrored here. Protected accounts are not collected, and posts arrive as fast as we can read them rather than in real time.',
},
reddit: {
name: 'Reddit',
index: '/r',
note: 'Reddit publishes its own feed for this, and we read it on a schedule and keep a copy — so the address above works whether or not Reddit is answering right now.',
},
instagram: {
name: 'Instagram',
index: '/ig',
note: 'Instagram publishes no feeds, so this is collected on your behalf and mirrored here. Private accounts are not collected, and stories are not either — they expire, and a feed of things that have already gone is worse than no feed.',
},
};
Loading
Loading