diff --git a/.env.example b/.env.example index 21491ea..3d4bf8b 100644 --- a/.env.example +++ b/.env.example @@ -95,13 +95,23 @@ X_SESSION_COOLDOWN_SECONDS=900 IG_COOKIE= # -------------------------------------------------------------------- Facebook -# Facebook publishes no public feed, serves no page without a login, and has no -# bridge. The only route in is Meta's Graph API, which returns a Page's posts -# only to somebody who ADMINISTERS that Page - so /fb/ takes no open -# submissions, and a Page appears only once its operator connects it here. +# Facebook publishes no feed and shows nothing without a login, so a Page is +# read with a logged-in session off mbasic.facebook.com - the same bargain as +# the X and Instagram cookies above, not a different one. # -# FB_PAGE_TOKENS=[{"page":"MyPage","token":"EAA..."}] +# A normal browser session, copied from devtools. c_user and xs are the two that +# matter; extra cookies are harmless. +# +# FB_COOKIE='c_user=...; xs=...' # -# A Page Access Token can post as the Page. Treat it exactly like the X session -# cookies above: vault, not a service env, and never a database column. +# This is a login to somebody's Facebook account. Vault, not a service env, and +# never a database column. Use a dedicated account: expect it to be challenged, +# and expect /fb/ to be the least dependable namespace on the site. +FB_COOKIE= + +# Optional, and almost never applicable. Meta's Graph API returns a Page's posts +# only to somebody who ADMINISTERS that Page; where a token exists it is used in +# preference to scraping, because a supported API beats guessing at markup. +# +# FB_PAGE_TOKENS=[{"page":"MyPage","token":"EAA..."}] FB_PAGE_TOKENS= diff --git a/README.md b/README.md index a342f2e..6e0df72 100644 --- a/README.md +++ b/README.md @@ -101,35 +101,54 @@ to tell them apart. of a row and links already point at it. The `/r/` and `/x/` address is the canonical one, which is what search engines are told. -## Facebook, and what `/fb/` can honestly be +## Facebook, and what `/fb/` is honestly worth -**There is no way to read an arbitrary public Facebook Page.** Three doors, all -measured on 2026-08-29 rather than assumed: +Facebook publishes nothing readable without a login. Three doors, measured on +2026-08-29 rather than assumed: - the old `facebook.com/feeds/page.php?format=rss20` endpoint answers **404** — removed, not deprecated -- `mbasic.facebook.com/` answers 200 with a **login wall** +- `mbasic.facebook.com/` answers 200 and **redirects to `login.php`** - RSSHub, which carries a thousand namespaces and maintains Twitter and Instagram, has **no Facebook namespace at all** -The one remaining door is Meta's Graph API, and it only opens for Pages the -caller **administers**. Reading somebody else's public Page needs the -`Page Public Content Access` feature, which requires App Review plus business -verification and is granted rarely. - -So `/fb/` is the one namespace here that does not take open submissions: a Page -appears when its operator connects it, by putting a Page Access Token in -`FB_PAGE_TOKENS`. A Page nobody has connected is not "not crawled yet", it is -not collectable, and the page says exactly that rather than offering a button -that would quietly do nothing. +So a Page is read the way X and Instagram are read: with a logged-in session, +off the HTML that `mbasic.facebook.com` still renders server-side. ```bash -FB_PAGE_TOKENS='[{"page":"MyPage","token":"EAA..."}]' +FB_COOKIE='c_user=...; xs=...' # a normal browser session, not an app +FB_PAGE_TOKENS= # optional; see below ``` -What is deliberately absent: anything that drives a logged-in Facebook session -against that login wall. It breaks constantly, it is against Meta's terms, and -it would risk an account of ours to serve a directory nobody pays for. +Where a Page Access Token happens to exist for a Page, Meta's Graph API is used +instead — a supported API beats guessing at markup. It is never required and +almost never present, because Graph only returns a Page's posts to somebody who +**administers** that Page. + +**What that third bullet is telling you.** RSSHub maintains Twitter and +Instagram and not this, and that is a verdict rather than an oversight: Facebook +is the most hostile of the three to being read this way. Expect the markup to +change, expect the reading account to be challenged, and expect `/fb/` to be the +least dependable namespace on the site. + +Three things follow from that, and they are in the code rather than in this +paragraph: + +- **Every selector is in one place.** `SELECTORS` in + `packages/social/src/facebook/scrape.js`. When it breaks, fetch a page with a + live cookie, look at the HTML, and edit that object — nothing else. +- **A checkpoint retires the session, never the Page.** A login wall is + classified as an auth failure, which reschedules and leaves every health + column alone. Ten of those in a row would otherwise mark every Facebook Page + in the directory dead, since a feed is retired after ten failures. +- **It is polled hourly, not every five minutes.** Each platform has its own + floor (`FLOOR_MINUTES`): X 5, Instagram 30, Facebook 60. The cost of asking + too often here is not a 429, it is a lock on the account and every Facebook + source going dark at once. A Page that posts twice a week loses nothing to an + hourly floor. + +A personal profile cannot be read by any of this and is rejected rather than +half-supported. ## Collecting Instagram diff --git a/apps/web/src/app/AddSocialSource.jsx b/apps/web/src/app/AddSocialSource.jsx index 7a31016..9669ffc 100644 --- a/apps/web/src/app/AddSocialSource.jsx +++ b/apps/web/src/app/AddSocialSource.jsx @@ -16,10 +16,10 @@ import { siteUrl } from '../lib/db.js'; * its next tick, and this page is replaced by the real one within the minute * (§17, §37). * - * **Facebook is the exception, and says so.** There is no public feed, no - * unauthenticated HTML and no provider; the only way in is a Page Access Token - * from whoever administers the Page. Offering an "add" button there would be a - * button that quietly does nothing, so it gets an explanation instead. + * Facebook was the exception here for one revision, on the grounds that only a + * Page's own administrator could connect it. That is no longer true: it is read + * with a session like X and Instagram are, so it takes open submissions like + * they do — see facebook/scrape.js for what that costs in reliability. * * @param {{ network: 'x'|'reddit'|'instagram'|'facebook', label: string, input: string, canonical: string }} props * `input` is what gets submitted — the canonical upstream URL, not what was @@ -38,39 +38,6 @@ export default function AddSocialSource({ network, label, input, canonical }) { ); - if (network === 'facebook') { - return ( -
-

{label}

- -

- This Facebook Page is not connected, and unlike the rest of the directory it cannot be - added by anyone who happens to want it. -

- -

- Facebook publishes no feed for a Page, serves no page without a login, and has no - third-party bridge we can use. The only remaining route is Meta’s own Graph API, - and it will only return a Page’s posts to somebody who administers that - Page — reading a stranger’s public Page needs a permission Meta grants - rarely and only after review. -

- -

- So if this is your Page, it can be connected: an administrator supplies a Page Access - Token and it appears here at {address}, collected on the same schedule as everything - else. If it is not your Page, there is nothing we can honestly offer — and we would - rather say that than mirror a scraper that breaks every few weeks. -

- -

- What is connected · X · Instagram ·{' '} - Reddit -

-
- ); - } - return (

{label}

@@ -114,4 +81,9 @@ const PLATFORMS = { index: '/ig', note: 'Instagram publishes no feeds, so this is collected on your behalf and mirrored here. Private accounts are not collected, and stories are not either — they expire, and a feed of things that have already gone is worse than no feed.', }, + facebook: { + name: 'Facebook', + index: '/fb', + note: 'Facebook publishes no feeds and shows nothing without a login, so this is read on your behalf and mirrored here. It is the least reliable of the four by some distance — Facebook changes its markup often and without warning — and it is checked hourly rather than every few minutes, because asking faster is how the reading account gets locked.', + }, }; diff --git a/apps/web/src/app/SocialIndex.jsx b/apps/web/src/app/SocialIndex.jsx index 721ce87..2fb5831 100644 --- a/apps/web/src/app/SocialIndex.jsx +++ b/apps/web/src/app/SocialIndex.jsx @@ -60,12 +60,12 @@ const LOOKS = { }, facebook: { platform: 'Facebook', - noun: 'connected Pages', + noun: 'Pages', base: '/fb', - placeholder: '', - addLabel: '', + placeholder: 'fb/SomePage', + addLabel: 'Add a Facebook Page', blurb: - 'Facebook is the one platform here that cannot be added by whoever wants it. There is no public feed, no page without a login, and no bridge — only Meta’s Graph API, which returns a Page’s posts to somebody who administers that Page. So these are Pages whose operators connected them, and nothing else can be.', + 'Facebook publishes no feeds and shows nothing without a login, so these are read on your behalf and mirrored here. Expect them to be the least dependable part of this directory: Facebook changes its markup without warning and works harder than the others to stop anyone reading it this way, so a Page here goes quiet from time to time. Pages only — a personal profile cannot be read at all.', }, }; @@ -95,16 +95,11 @@ export default async function SocialIndex({ network, page = 1 }) {

{blurb}

- {/* Facebook has no add form, and that is not an oversight: a Page can - only be connected by somebody who administers it, so a box inviting - anyone to paste a Page would be a box that quietly does nothing. */} - {network === 'facebook' ? null : ( -
- - - -
- )} +
+ + + +
{rows.length >= FILTER_FROM ? ( }} ctx @@ -24,7 +24,7 @@ export async function GET(req, { params }) { format, 400, `not a Facebook Page name: ${page}`, - 'Page names are 5-60 characters of A-Z, 0-9 and dot.', + 'Page names are 3-60 characters of A-Z, 0-9 and dot.', ); } diff --git a/apps/web/src/app/fb/[page]/page.jsx b/apps/web/src/app/fb/[page]/page.jsx index 7817ee2..028ce9b 100644 --- a/apps/web/src/app/fb/[page]/page.jsx +++ b/apps/web/src/app/fb/[page]/page.jsx @@ -10,11 +10,10 @@ export const dynamic = 'force-dynamic'; /** * One Facebook Page, at `/fb/SomePage`. * - * The one namespace of the four where "not here yet" usually means "not - * possible" rather than "nobody has got round to it". Facebook has no public - * feed, no unauthenticated HTML and no provider — the only way in is a Page - * Access Token from whoever administers the Page — so the empty state explains - * that rather than offering a button that would quietly do nothing. + * Read with a session against mbasic, like X and Instagram are — so this takes + * open submissions like they do. It is the least reliable of the four by some + * distance; see @rssamplifier/social's facebook/scrape.js for why, and for the + * one place to look when it stops working. * * @param {{ params: Promise<{ page: string }> }} props */ diff --git a/packages/db/src/social.js b/packages/db/src/social.js index 34bb612..24b4444 100644 --- a/packages/db/src/social.js +++ b/packages/db/src/social.js @@ -20,6 +20,9 @@ import { newId, nowIso } from './client.js'; * @typedef {import('@libsql/client').Client} Client */ +/** The interval a source starts on when its caller names none. */ +const DEFAULT_START_MINUTES = 60; + /** * The one source behind a canonical ref. * @@ -110,7 +113,11 @@ export async function countSocialFeeds(db, network) { * @param {{ * network: string, ref: string, slug: string, title: string, feedUrl: string, * siteUrl?: string|null, config?: object|null, priority?: number, - * }} source + * intervalMinutes?: number, + * }} source `intervalMinutes` is the platform's floor, supplied by the caller — + * this package deliberately does not import @rssamplifier/social to look it + * up, because the dependency would run the wrong way: social already reads + * nothing from db, and db has no other reason to know what a platform is. * @returns {Promise<{ id: string, slug: string, created: boolean }>} */ export async function upsertSocialSource(db, source) { @@ -147,7 +154,7 @@ export async function upsertSocialSource(db, source) { // Reddit is excluded deliberately — it is a real feed on somebody else's // server, and 50,026 of them at five minutes is how you get rate-limited // off a platform. See markHostThrottled in queries.js. - source.network === 'reddit' ? 60 : 5, + Math.max(1, Number(source.intervalMinutes) || DEFAULT_START_MINUTES), now, now, now, diff --git a/packages/ingest/src/crawl.js b/packages/ingest/src/crawl.js index c1f7651..4685a8b 100644 --- a/packages/ingest/src/crawl.js +++ b/packages/ingest/src/crawl.js @@ -1,6 +1,6 @@ import { resolveFeed, scrapeFeed, feedTopics } from '@rssamplifier/feed'; import { q, authors } from '@rssamplifier/db'; -import { fetchSocialSource, isCollected } from '@rssamplifier/social'; +import { fetchSocialSource, floorMinutesFor, isCollected } from '@rssamplifier/social'; import { prepareCredits } from './enrich.js'; import { @@ -226,11 +226,13 @@ export async function crawlFeed(db, feed, opts = {}) { // distinction, so adding a platform never edits this file. const social = isCollected(feed); - // A provider-backed source polls on a five-minute floor rather than an hour's - // — see SOCIAL_MIN_INTERVAL. The floor is passed to every scheduling call - // below rather than read from a global, so this row's cadence is decided here - // and nowhere else. - const floor = social ? SOCIAL_MIN_INTERVAL : FLOOR_DEFAULT; + // How fast this row may be asked, which is a property of its platform rather + // than of this codebase — X through RSSHub tolerates five minutes, a personal + // Instagram or Facebook session does not, and asking too often there costs a + // checkpoint on the account rather than a 429. @rssamplifier/social owns the + // table; the floor is passed to every scheduling call below so this row's + // cadence is decided here and nowhere else. + const floor = social ? floorMinutesFor(feed) : FLOOR_DEFAULT; // What the server told us last time, sent back so it can answer "still the // same" without sending the document again. Scraped sources are excluded: what diff --git a/packages/ingest/src/submit.js b/packages/ingest/src/submit.js index d8d9490..e41efac 100644 --- a/packages/ingest/src/submit.js +++ b/packages/ingest/src/submit.js @@ -1,6 +1,6 @@ import { resolveFeed, scrapeFeed, normalizeUrl, parseOpml, uniqueSlug } from '@rssamplifier/feed'; import { q, social } from '@rssamplifier/db'; -import { socialSourceFrom } from '@rssamplifier/social'; +import { floorMinutesFor, socialSourceFrom } from '@rssamplifier/social'; import { queueFeeds } from './queue.js'; import { refreshFeedKeywords } from './crawl.js'; @@ -208,6 +208,10 @@ async function submitSocial(db, source) { feedUrl: source.feedUrl, siteUrl: source.siteUrl, priority: 1, + // The platform's floor, so a new row starts at the fastest it will ever be + // asked and the crawler backs it off from there. A Facebook Page read with + // a personal session starts hourly; an X timeline starts at five minutes. + intervalMinutes: floorMinutesFor({ social_network: source.network }), }); if (!stored.id) { diff --git a/packages/social/index.js b/packages/social/index.js index b0575b3..d6e6dac 100644 --- a/packages/social/index.js +++ b/packages/social/index.js @@ -79,7 +79,14 @@ export { export { fetchFacebookSource, pageToken, connectedPages } from './src/facebook/fetch.js'; export { failureResult, retryAfterFor, ANOMALY_SECONDS, UNCONFIGURED_SECONDS } from './src/failure.js'; -export { fetchSocialSource, isCollected } from './src/collect.js'; +export { + fetchSocialSource, + isCollected, + floorMinutesFor, + FLOOR_MINUTES, + DEFAULT_FLOOR_MINUTES, +} from './src/collect.js'; +export { scrapeFacebookPage } from './src/facebook/scrape.js'; export { socialSourceFrom, socialPathFor, SOCIAL_NETWORKS } from './src/identify.js'; export { socialDisplayTitle } from './src/display.js'; diff --git a/packages/social/package.json b/packages/social/package.json index 21497a5..053f060 100644 --- a/packages/social/package.json +++ b/packages/social/package.json @@ -15,6 +15,7 @@ "test": "node --test test/*.test.js" }, "dependencies": { - "@rssamplifier/feed": "workspace:*" + "@rssamplifier/feed": "workspace:*", + "linkedom": "^0.18.13" } } diff --git a/packages/social/src/collect.js b/packages/social/src/collect.js index 07baae6..e629ad2 100644 --- a/packages/social/src/collect.js +++ b/packages/social/src/collect.js @@ -30,6 +30,40 @@ const COLLECTORS = { facebook: fetchFacebookSource, }; +/** + * The fastest each platform may be asked, in minutes. + * + * Not one number, because the three are not one risk. X goes through RSSHub, + * which is built to be polled and is reading a medium where an hour old is + * visibly stale — five minutes is the point of the exercise. Instagram and + * Facebook are read with a personal session against a platform that watches for + * exactly that, and the cost of asking too often is not a 429, it is a + * checkpoint on the account and every source on that platform going dark at + * once. + * + * So the slow ones are slow on purpose. A Facebook Page that posts twice a week + * loses nothing to an hourly floor, and the account survives to keep reading it. + * + * These are floors, not intervals: the crawler's cadence code still backs a + * quiet source further off on its own. Nothing here makes a source faster. + */ +export const FLOOR_MINUTES = { + x: 5, + instagram: 30, + facebook: 60, +}; + +/** The default for anything fetched rather than collected. */ +export const DEFAULT_FLOOR_MINUTES = 60; + +/** + * @param {{ social_network?: string|null }} feed + * @returns {number} minutes + */ +export function floorMinutesFor(feed) { + return FLOOR_MINUTES[String(feed?.social_network ?? '')] ?? DEFAULT_FLOOR_MINUTES; +} + /** * Is this row collected through a provider rather than fetched from a document? * diff --git a/packages/social/src/facebook/canonical.js b/packages/social/src/facebook/canonical.js index dbdc47a..85dc8e6 100644 --- a/packages/social/src/facebook/canonical.js +++ b/packages/social/src/facebook/canonical.js @@ -1,43 +1,46 @@ /** - * Facebook — and the honest limits on what `/fb/` can ever be. + * Facebook — and the honest limits on what `/fb/` is worth. * * Read this before adding to it, because the shape of this file is decided by * something outside the codebase. * - * **There is no way to read an arbitrary public Facebook Page.** Three doors, - * all measured rather than assumed, on 2026-08-29: + * **Facebook publishes nothing readable without a login.** Three doors, + * measured rather than assumed on 2026-08-29: * * - the old `facebook.com/feeds/page.php?format=rss20` endpoint answers 404; * it was removed, not deprecated - * - `mbasic.facebook.com/` answers 200 with a login wall + * - `mbasic.facebook.com/` answers 200 and redirects to `login.php` * - RSSHub, which carries a thousand namespaces and maintains Twitter and * Instagram, has no Facebook namespace at all * - * The remaining door is the Graph API, and it only opens for Pages the caller - * **administers**: reading somebody else's public Page needs the - * `Page Public Content Access` feature, which requires App Review and business - * verification and is granted rarely. So a Facebook source here is not - * something a stranger can submit — it is something the Page's own operator - * connects, by supplying a Page Access Token. + * So a Page is read the way X and Instagram are read: with a logged-in session, + * off the HTML that mbasic renders server-side. That is `./scrape.js`, and it + * is the default. `./fetch.js` will use Meta's Graph API instead for a Page + * somebody administers and has a token for, because a supported API beats + * guessing at markup — but a token is never required and almost never present. * - * That is a different bargain from the rest of this directory, where anyone may - * submit anything, and `/fb/` should not pretend otherwise: a page nobody has - * connected a token for is not "not crawled yet", it is "not collectable", and - * the page says so. + * **What that third bullet is telling you.** RSSHub maintains Twitter and + * Instagram and not this, and that is a verdict rather than an oversight: + * Facebook is the most hostile of the three to being read this way. Expect the + * markup to change, expect the reading account to be challenged, and expect + * `/fb/` to be the least dependable namespace on the site. The failure handling + * downstream is built for that — a checkpoint retires the *session*, never the + * Page — but no amount of care makes the underlying surface stable. * - * What is deliberately *not* here: anything that drives a logged-in Facebook - * session against the login wall. It breaks constantly, it is against Meta's - * terms, and it would put an account of ours at risk to serve a directory - * nobody is paying for. + * A personal profile is not readable by any of this. `profile.php` and friends + * are rejected below rather than half-supported. */ /** * A Page's public name — the `/PageName` in a Facebook URL. * - * Facebook calls it a "username" or "vanity URL" and permits letters, digits - * and dots, minimum five characters. + * Letters, digits and dots. Facebook documents a five-character minimum and + * that minimum is wrong to enforce: it applies to usernames created now, and + * plenty of long-standing Pages are shorter — facebook.com/NASA is four. A + * regex written from the documentation rejects real Pages, so the floor is + * three and `NOT_A_PAGE` below does the work of excluding furniture. */ -const VANITY = /^[A-Za-z0-9.]{5,60}$/; +const VANITY = /^[A-Za-z0-9.]{3,60}$/; /** A numeric Page id, which is what the Graph API actually addresses. */ const PAGE_ID = /^[0-9]{6,25}$/; @@ -95,7 +98,7 @@ export function parseFacebookInput(input) { if (!raw) return null; // `fb/SomePage` and `fb.com/SomePage` shorthands, plus a bare numeric id. - const short = /^\/?(?:fb|facebook)\/([A-Za-z0-9.]{5,60})\/?$/i.exec(raw); + const short = /^\/?(?:fb|facebook)\/([A-Za-z0-9.]{3,60})\/?$/i.exec(raw); if (short) return { mode: 'page', page: short[1] }; // A bare string of digits is deliberately NOT read as a Page id. It is also a @@ -196,6 +199,6 @@ export function facebookSource(input) { * @returns {{ mode: 'page', page: string }|null} */ export function facebookSpecFromRef(ref) { - const match = /^fb:page:([A-Za-z0-9.]{5,60}|[0-9]{6,25})$/.exec(String(ref ?? '')); + const match = /^fb:page:([A-Za-z0-9.]{3,60}|[0-9]{6,25})$/.exec(String(ref ?? '')); return match ? { mode: 'page', page: match[1] } : null; } diff --git a/packages/social/src/facebook/fetch.js b/packages/social/src/facebook/fetch.js index 1e6fde9..7792ba6 100644 --- a/packages/social/src/facebook/fetch.js +++ b/packages/social/src/facebook/fetch.js @@ -1,27 +1,30 @@ /** - * Collecting Facebook, for the Pages somebody has actually connected. + * Collecting Facebook, by session first and by API where one is available. * - * See `./canonical.js` for why this is the only shape available: there is no - * way to read an arbitrary public Page, so a Facebook source is not something a - * stranger submits — it is something a Page's operator connects by supplying a - * Page Access Token. + * Two ways in, and the better one is not the default because it is almost never + * possible. Meta's Graph API returns a Page's posts only to somebody who + * administers that Page — reading anyone else's needs `Page Public Content + * Access`, which wants App Review and business verification — so a token + * reaches a handful of Pages and a session reaches the rest. `FB_PAGE_TOKENS` + * is consulted first and `FB_COOKIE` carries everything else. * - * **Tokens live in the environment, keyed by Page**, never in a table, for - * exactly the reason X's session cookies do not: a Page Access Token can post - * as the Page. `FB_PAGE_TOKENS` is a JSON array, and a Page with no entry in it - * is not "not crawled yet" — it is not collectable, and both the crawler and - * the page say so rather than retrying for ever. + * Both produce the same items, keyed the same way (`fb:`), so a Page + * that gains a token later does not change identity and nobody's reader marks + * the feed unread. * - * This is the one collector in the package that talks to a real, supported, - * documented API rather than a bridge, which makes it the least likely of the - * three to break and the one with the smallest reach. That trade is Meta's, not - * ours. + * **Credentials live in the environment, never in a table**, for the reason X's + * session cookies do not: a Page token can post as the Page, and a Facebook + * session cookie is a login to somebody's account. See §36 and AC-7. + * + * On what this is worth, and the failure handling that follows from it, see + * `./scrape.js`. */ import { providerGet } from '../x/providers/http.js'; import { XUnavailable, XNoSuchSource } from '../x/errors.js'; import { failureResult } from '../failure.js'; import { facebookSpecFromRef } from './canonical.js'; +import { scrapeFacebookPage } from './scrape.js'; /** * Pinned rather than floating. Graph deprecates a version roughly every two @@ -64,12 +67,53 @@ export async function fetchFacebookSource(feed, opts) { const display = displayName(feed?.feed_url) ?? spec.page; try { + // Two ways in, and the cheap one is not the default. + // + // A Page Access Token is strictly better where it exists — a supported API, + // structured posts, no markup to guess at — but it only ever exists for a + // Page somebody administers, which is almost none of them. So Graph is used + // when a token happens to be configured for this Page, and the session + // scrape carries everything else. Neither needs configuring per source. const token = pageToken(env, spec.page); + if (!token) { - // Not an outage and not a broken Page: nobody has connected it. Phrased - // so `retryAfterFor` gives it the hour it deserves rather than retrying - // every twenty minutes for a token that is not coming. - throw new XUnavailable(`facebook: page ${spec.page} is not connected`); + const cookie = String(env.FB_COOKIE ?? '').trim(); + onEvent('facebook.fetch.started', { ref: feed.social_ref, via: 'mbasic' }); + + const scraped = await scrapeFacebookPage(spec, { + cookie, + timeoutMs: Number(env.X_FETCH_TIMEOUT_MS) || undefined, + fetch: opts.runtime?.fetch, + signal: opts.signal, + }); + + const items = scraped.posts.map((post) => fromScrape(post, display)).filter(Boolean); + + if (items.length === 0 && Number(feed?.item_count ?? 0) > 0) { + onEvent('facebook.fetch.failed', { ref: feed.social_ref, error: 'empty-result' }); + return { ok: false, throttled: true, retryAfter: 20 * 60, error: 'empty-result' }; + } + + onEvent('facebook.fetch.success', { + ref: feed.social_ref, + via: 'mbasic', + itemCount: items.length, + }); + + return { + ok: true, + feedUrl: feed.feed_url, + feed: { + title: `${scraped.displayName ?? display} on Facebook`, + description: `Posts from the ${display} Page on Facebook, mirrored by RSS Amplifier.`, + siteUrl: String(feed.feed_url), + language: null, + imageUrl: null, + categories: [], + kind: 'blog', + items, + }, + }; } const url = new URL(`${GRAPH}/${encodeURIComponent(spec.page)}/posts`); @@ -131,6 +175,37 @@ export async function fetchFacebookSource(feed, opts) { } } +/** + * One scraped post as one of our items. + * + * Deliberately the same shape `toItem` produces from Graph, so that a Page + * which gains a token later does not change identity: both key on the post id, + * so `fb:` is the same guid whichever way the post was read, and switching + * does not mark a subscriber's whole feed unread. + * + * @param {{ id: string, url: string, text: string, createdAt: string|null, image: string|null }} post + * @param {string} display + */ +function fromScrape(post, display) { + if (!post?.id) return null; + + const first = String(post.text ?? '').split('\n').find(Boolean) ?? ''; + const title = first ? clip(first, 110) : '(photo)'; + + return { + guid: `fb:${post.id}`, + url: post.url, + title: `${display}: ${title}`, + summary: clip(post.text, 400) || null, + contentHtml: html(post.text, post.image, post.url), + author: display, + publishedAt: post.createdAt, + imageUrl: post.image, + categories: [], + audio: null, + }; +} + /** * One Graph post as one of our items. * @@ -243,7 +318,7 @@ export function connectedPages(env = process.env) { * @returns {string|null} */ function displayName(feedUrl) { - const match = /facebook\.com\/([A-Za-z0-9.]{5,60})\/?$/.exec(String(feedUrl ?? '')); + const match = /facebook\.com\/([A-Za-z0-9.]{3,60})\/?$/.exec(String(feedUrl ?? '')); return match ? match[1] : null; } diff --git a/packages/social/src/facebook/scrape.js b/packages/social/src/facebook/scrape.js new file mode 100644 index 0000000..951b99f --- /dev/null +++ b/packages/social/src/facebook/scrape.js @@ -0,0 +1,248 @@ +import { parseHTML } from 'linkedom'; + +import { providerGet } from '../x/providers/http.js'; +import { XAuthFailed, XNoSuchSource, XUnavailable } from '../x/errors.js'; + +/** + * Reading a Facebook Page the way the rest of this package reads X and + * Instagram: with a logged-in session, off the HTML. + * + * **Why this exists after `./fetch.js` said it would not.** The Graph API path + * next door works and needs no scraping, but it only ever reaches Pages the + * caller administers, which makes `/fb/` a namespace almost nobody can add to. + * The rest of the directory does not work that way, and neither do the other + * two collected platforms: X is read with an `auth_token` cookie and Instagram + * with an `IG_COOKIE`. A session cookie here is the same bargain, not a new one. + * + * **`mbasic.facebook.com`, not `www`.** It is the no-JavaScript version, so a + * page is server-rendered HTML that a parser can read — where `www` is a React + * application whose content arrives in GraphQL payloads and whose class names + * are generated afresh on every deploy. mbasic still answers (measured + * 2026-08-29: it 302s to `login.php` without a session rather than 404ing, which + * is what tells you it is alive and gated rather than gone). + * + * ## What this is honestly worth + * + * Less than the other two, and the difference is worth stating plainly rather + * than discovering in production: + * + * - **Nobody maintains this shape but us.** RSSHub carries a thousand + * namespaces and keeps Twitter and Instagram working; it has no Facebook one. + * That is not an oversight, it is a verdict — Facebook is the most hostile of + * the three to automation, and the tooling ecosystem reflects it. + * - **It will break**, and on Facebook's schedule rather than ours. Every + * selector below is a guess about somebody else's markup. They are collected + * in `SELECTORS` so that fixing them is one edit in one place. + * - **The account will be challenged.** Expect checkpoints, and expect them + * sooner if this polls quickly — which is why Facebook gets its own, much + * slower interval rather than the five minutes X gets. + * + * None of that is a reason not to build it. It is a reason for the failure + * handling below to be careful: a checkpoint must retire the *session*, never + * the Page, or one bad afternoon deletes the namespace. + */ + +/** The mobile HTML host. The only one of the three that renders server-side. */ +const MBASIC = 'https://mbasic.facebook.com'; + +/** + * A phone, because mbasic serves its simplest markup to one. + * + * Not a disguise — the crawler identifies itself in `providerGet`'s default + * agent everywhere it is not required to look like a browser. Here the user + * agent selects a *rendering*, and asking for the desktop one gets a page this + * parser cannot read. + */ +const UA = 'Mozilla/5.0 (Android 10; Mobile; rv:109.0) Gecko/109.0 Firefox/115.0'; + +/** + * Everything that depends on Facebook's markup, in one place. + * + * When this stops working — and it will — the fix is almost certainly here and + * nowhere else. Fetch a page with a live cookie, look at the HTML, and update + * the list. Each is tried in order and the first that matches wins, so an old + * selector can be left in place while a new one is added above it. + */ +const SELECTORS = { + /** A post. mbasic marks each story with a `data-ft` JSON blob. */ + post: ['div[data-ft*="top_level_post_id"]', 'div[data-ft*="mf_story_key"]', 'article'], + /** The prose inside one. The first non-empty match is taken as the caption. */ + text: ['div[data-ft] > div > span', 'div[data-ft] > div > div > span', 'p'], + /** The permalink, which also carries the post id. */ + link: ['a[href*="/story.php?story_fbid="]', 'a[href*="/posts/"]', 'a[href*="story_fbid"]'], + /** The timestamp. mbasic still uses , which is the only date on offer. */ + time: ['abbr'], + /** An attached photo. */ + image: ['img[src*="scontent"]', 'img'], +}; + +/** + * Is the response the login wall rather than the page? + * + * Checked on the final URL and on the body, because the redirect and the + * interstitial are two different ways of being told the same thing. + */ +const LOGIN_WALL = /\/login\.php|\/login\/\?|name="login"|checkpoint/i; + +/** + * Scrape one Page. + * + * @param {{ page: string }} spec + * @param {{ + * cookie: string, + * timeoutMs?: number, + * fetch?: typeof fetch, + * signal?: AbortSignal, + * }} opts + * @returns {Promise<{ posts: Array, displayName: string|null }>} + */ +export async function scrapeFacebookPage(spec, opts) { + if (!opts?.cookie) { + throw new XUnavailable('facebook: no session (FB_COOKIE is not set)'); + } + + const url = `${MBASIC}/${encodeURIComponent(spec.page)}`; + + const { body, url: landed } = await providerGet(url, { + provider: 'facebook-mbasic', + headers: { + cookie: opts.cookie, + 'user-agent': UA, + // mbasic serves a different, heavier page to a client that claims to + // want the modern one. + accept: 'text/html,application/xhtml+xml', + 'accept-language': 'en-US,en;q=0.9', + }, + timeoutMs: opts.timeoutMs, + fetch: opts.fetch, + signal: opts.signal, + }); + + // The session is gone, or Facebook wants a challenge solved. Either way it is + // a fact about our login and not about this Page — `failureResult` routes + // XAuthFailed to a reschedule, and the session pool retires the credential. + // Blaming the Page here is how ten crawls delete the namespace. + if (LOGIN_WALL.test(String(landed ?? '')) || LOGIN_WALL.test(body.slice(0, 4000))) { + throw new XAuthFailed('facebook: session rejected (login or checkpoint)'); + } + + const { document } = parseHTML(body); + + // Facebook answers a missing Page with a real page saying so, not a 404. + if (/isn't available|content isn't available|page you requested|couldn't find/i.test(body.slice(0, 20_000))) { + throw new XNoSuchSource(`facebook: no such page ${spec.page}`); + } + + const nodes = firstMatch(document, SELECTORS.post); + const posts = [...nodes].map((node) => toPost(node, spec)).filter(Boolean); + + return { posts, displayName: displayName(document) }; +} + +/** + * The first selector in a list that matches anything. + * + * The list is a fallback chain rather than a union, so a newer selector can be + * added above an older one without the two both matching and doubling every + * post. + */ +function firstMatch(root, selectors) { + for (const selector of selectors) { + const found = root.querySelectorAll(selector); + if (found.length > 0) return found; + } + return []; +} + +/** + * One story element as a post. + * + * @param {any} node + * @param {{ page: string }} spec + */ +function toPost(node, spec) { + const id = postId(node); + + // No id, no post. An item that cannot be deduplicated arrives again on every + // crawl for ever, which is invisible until the feed is nothing but + // duplicates — the same rule the X and Instagram collectors apply. + if (!id) return null; + + const link = firstMatch(node, SELECTORS.link)[0]; + const href = link?.getAttribute?.('href') ?? null; + + const text = [...firstMatch(node, SELECTORS.text)] + .map((el) => String(el.textContent ?? '').trim()) + .filter(Boolean) + .join('\n\n') + .trim(); + + const time = firstMatch(node, SELECTORS.time)[0]; + const image = firstMatch(node, SELECTORS.image)[0]?.getAttribute?.('src') ?? null; + + return { + id, + url: absolute(href) ?? `https://www.facebook.com/${spec.page}`, + text, + // mbasic writes a human date ("Yesterday at 14:03"). `Date.parse` handles + // the absolute forms and returns NaN for the relative ones, which becomes + // null — and a null date is correct rather than a guess. The crawler's + // cadence code already copes with an undated feed; inventing "now" for + // every post would make the feed look permanently fresh, which is the one + // failure `publishedTimes` exists to prevent. + createdAt: parseDate(time?.getAttribute?.('data-utime'), time?.textContent), + image: image && image.startsWith('http') ? image : null, + }; +} + +/** + * The post id, from the `data-ft` blob mbasic attaches to each story. + * + * Read from the attribute rather than from the permalink, because the permalink + * varies (`/story.php?story_fbid=`, `//posts/`) while this does not — + * and because a Page rename rewrites every permalink it has ever had, so a + * URL-keyed identity would re-ingest the whole Page the day that happens. + */ +function postId(node) { + const raw = node?.getAttribute?.('data-ft'); + if (raw) { + try { + const parsed = JSON.parse(raw); + const id = parsed?.top_level_post_id ?? parsed?.mf_story_key; + if (id) return String(id); + } catch { + // Fall through to the link. + } + } + + const href = firstMatch(node, SELECTORS.link)[0]?.getAttribute?.('href') ?? ''; + return /story_fbid=(\d+)/.exec(href)?.[1] ?? /\/posts\/(\d+)/.exec(href)?.[1] ?? null; +} + +/** The Page's own name, for the feed title. */ +function displayName(document) { + const title = String(document.querySelector('title')?.textContent ?? '').trim(); + if (!title) return null; + // mbasic titles are " - Home | Facebook" and similar. + return title.replace(/\s*[-|]\s*(Home\s*)?\|?\s*Facebook\s*$/i, '').trim() || null; +} + +/** + * @param {string|null|undefined} utime epoch seconds, when mbasic gives one + * @param {string|null|undefined} text the human date otherwise + * @returns {string|null} ISO 8601 + */ +function parseDate(utime, text) { + const seconds = Number(utime); + if (Number.isFinite(seconds) && seconds > 0) return new Date(seconds * 1000).toISOString(); + + const parsed = Date.parse(String(text ?? '')); + return Number.isFinite(parsed) ? new Date(parsed).toISOString() : null; +} + +/** mbasic links are relative and point at mbasic; ours must point at Facebook. */ +function absolute(href) { + if (!href) return null; + const path = href.startsWith('http') ? href : `https://www.facebook.com${href}`; + return path.replace('//mbasic.facebook.com', '//www.facebook.com').split('&refid')[0]; +} diff --git a/packages/social/src/failure.js b/packages/social/src/failure.js index 73319eb..0d742e0 100644 --- a/packages/social/src/failure.js +++ b/packages/social/src/failure.js @@ -61,7 +61,14 @@ export function retryAfterFor(error) { // Distinguished by message rather than by type, because `XUnavailable` covers // both "nothing is configured" and "the thing that is configured is down", // and those deserve very different patience. - if (/no .* provider is configured|no RSSHUB_BASE_URL|not connected/i.test(String(error?.message ?? ''))) { + // Every shape of "nothing is set up to collect with": no provider in the + // registry, no bridge URL, no session cookie. All of them are a deployment + // that has not happened, and none of them changes in twenty minutes. + if ( + /no .* provider is configured|no RSSHUB_BASE_URL|not connected|no session/i.test( + String(error?.message ?? ''), + ) + ) { return UNCONFIGURED_SECONDS; } diff --git a/packages/social/src/x/providers/http.js b/packages/social/src/x/providers/http.js index c0432f8..26b124a 100644 --- a/packages/social/src/x/providers/http.js +++ b/packages/social/src/x/providers/http.js @@ -34,7 +34,7 @@ const MAX_BYTES = 4 * 1024 * 1024; * fetch?: typeof fetch, * signal?: AbortSignal, * }} [opts] - * @returns {Promise<{ body: string, status: number, headers: Headers }>} + * @returns {Promise<{ body: string, status: number, headers: Headers, url: string }>} */ export async function providerGet(url, opts = {}) { const { @@ -75,7 +75,10 @@ export async function providerGet(url, opts = {}) { }); if (failure) throw failure; - return { body, status: res.status, headers: res.headers }; + // `url` is where the response actually came from after redirects. The + // Facebook scraper needs it: mbasic answers a missing session with a 302 + // to login.php and a 200 body, so the status alone says nothing. + return { body, status: res.status, headers: res.headers, url: res.url ?? String(url) }; } catch (error) { if (error?.name?.startsWith('X')) throw error; throw new XUnavailable(`${provider}: ${redact(error)}`, { provider, sessionId, cause: null }); diff --git a/packages/social/test/facebook-scrape.test.js b/packages/social/test/facebook-scrape.test.js new file mode 100644 index 0000000..9bae502 --- /dev/null +++ b/packages/social/test/facebook-scrape.test.js @@ -0,0 +1,206 @@ +import assert from 'node:assert/strict'; +import { test } from 'node:test'; + +import { scrapeFacebookPage } from '../src/facebook/scrape.js'; +import { fetchFacebookSource } from '../src/facebook/fetch.js'; +import { floorMinutesFor, FLOOR_MINUTES } from '../src/collect.js'; + +/* + * The scraper, against a fixture shaped like mbasic's own output. + * + * What matters here is not that the selectors are right — only a live cookie + * proves that, and they will change anyway — but that everything *around* them + * is: a login wall must retire the session and not the Page, a post with no id + * must be dropped rather than stored undeduplicatable, and a relative mbasic + * link must come out pointing at facebook.com. + */ + +const PAGE = [ + 'NASA - Home | Facebook', + '
', + '
', + '
We are going to the Moon.
', + ' 29 August at 10:00', + ' Full story', + ' ', + '
', + '
', + '
A second post.
', + ' 29 August at 09:00', + ' Full story', + '
', + '
Furniture with no data-ft and no link at all.
', + '
', +].join('\n'); + +/** + * A Response that reports where it landed. + * + * `Response.url` is getter-only, so it has to be defined rather than assigned — + * and it has to be set at all, because that is the only thing distinguishing + * "mbasic served the page" from "mbasic redirected to the login wall and served + * that instead". Both are a 200. + */ +const respond = (body, url = 'https://mbasic.facebook.com/nasa') => async () => { + const res = new Response(body, { status: 200, headers: { 'content-type': 'text/html' } }); + Object.defineProperty(res, 'url', { value: url }); + return res; +}; + +test('a page of stories becomes posts keyed on the post id', async () => { + const { posts, displayName } = await scrapeFacebookPage( + { page: 'nasa' }, + { cookie: 'c_user=1; xs=secret', fetch: respond(PAGE) }, + ); + + assert.deepEqual( + posts.map((p) => p.id), + ['123456789', '987654321'], + 'the element with no data-ft and no permalink is furniture, not a post', + ); + + assert.equal(displayName, 'NASA'); + assert.equal(posts[0].text, 'We are going to the Moon.'); + assert.equal(posts[0].createdAt, new Date(1756461600 * 1000).toISOString()); + assert.equal(posts[0].image, 'https://scontent.example/photo1.jpg'); +}); + +test('links come out pointing at facebook.com, not at mbasic', async () => { + const { posts } = await scrapeFacebookPage( + { page: 'nasa' }, + { cookie: 'c_user=1; xs=secret', fetch: respond(PAGE) }, + ); + + for (const post of posts) { + assert.match(post.url, /^https:\/\/www\.facebook\.com\//, post.url); + assert.doesNotMatch(post.url, /mbasic/); + // The tracking parameter mbasic appends is not part of the address. + assert.doesNotMatch(post.url, /refid/); + } +}); + +test('no session is a configuration problem, not a broken Page', async () => { + await assert.rejects( + () => scrapeFacebookPage({ page: 'nasa' }, { cookie: '', fetch: respond(PAGE) }), + /FB_COOKIE/, + ); +}); + +test('the login wall retires the session, never the Page', async () => { + // Both shapes of being told the same thing: the redirect, and the body. + const byUrl = scrapeFacebookPage( + { page: 'nasa' }, + { + cookie: 'c_user=1; xs=stale', + fetch: respond('anything', 'https://mbasic.facebook.com/login.php?next=x'), + }, + ); + await assert.rejects(byUrl, (error) => error.name === 'XAuthFailed'); + + const byBody = scrapeFacebookPage( + { page: 'nasa' }, + { + cookie: 'c_user=1; xs=stale', + fetch: respond('
…
'), + }, + ); + await assert.rejects(byBody, (error) => error.name === 'XAuthFailed'); +}); + +test('a Page that does not exist is the one failure about the source', async () => { + const missing = scrapeFacebookPage( + { page: 'nosuchpagehere' }, + { + cookie: 'c_user=1; xs=secret', + fetch: respond("This content isn't available right now"), + }, + ); + + await assert.rejects(missing, (error) => error.name === 'XNoSuchSource'); +}); + +test('an undated post is stored undated rather than stamped now', async () => { + // mbasic writes relative dates ("Yesterday at 14:03") that do not parse. A + // null date is correct; inventing `now` would make the feed look permanently + // fresh, which is exactly what publishedTimes exists to prevent. + const relative = PAGE.replace('29 August at 10:00', 'Yesterday at 14:03'); + + const { posts } = await scrapeFacebookPage( + { page: 'nasa' }, + { cookie: 'c_user=1; xs=secret', fetch: respond(relative) }, + ); + + assert.equal(posts[0].createdAt, null); +}); + +test('the collector turns a scrape into items, and a stale session into a throttle', async () => { + const feed = { social_ref: 'fb:page:nasa', feed_url: 'https://www.facebook.com/NASA' }; + + const ok = await fetchFacebookSource(feed, { + runtime: { env: { FB_COOKIE: 'c_user=1; xs=secret' }, onEvent: () => {}, fetch: respond(PAGE) }, + }); + + assert.equal(ok.ok, true); + assert.deepEqual( + ok.feed.items.map((i) => i.guid), + ['fb:123456789', 'fb:987654321'], + ); + // Display casing comes off feed_url, not off the lowercased ref. + assert.match(ok.feed.items[0].title, /^NASA: /); + + const stale = await fetchFacebookSource(feed, { + runtime: { + env: { FB_COOKIE: 'c_user=1; xs=stale' }, + onEvent: () => {}, + fetch: respond('x', 'https://mbasic.facebook.com/login.php'), + }, + }); + + // A dead cookie must reschedule. Ten of these in a row would otherwise retire + // every Facebook Page in the directory. + assert.equal(stale.ok, false); + assert.equal(stale.throttled, true); +}); + +test('a Page token is preferred over scraping where one exists', async () => { + let scraped = 0; + const feed = { social_ref: 'fb:page:nasa', feed_url: 'https://www.facebook.com/NASA' }; + + const result = await fetchFacebookSource(feed, { + runtime: { + env: { + FB_PAGE_TOKENS: '[{"page":"nasa","token":"EAAtoken"}]', + FB_COOKIE: 'c_user=1; xs=secret', + }, + onEvent: () => {}, + fetch: async (url) => { + if (String(url).includes('mbasic')) scraped += 1; + return new Response( + JSON.stringify({ data: [{ id: '1_2', message: 'via graph', created_time: '2026-08-29T10:00:00+0000' }] }), + { status: 200, headers: { 'content-type': 'application/json' } }, + ); + }, + }, + }); + + assert.equal(result.ok, true); + assert.equal(scraped, 0, 'a supported API beats guessing at markup'); + assert.deepEqual(result.feed.items.map((i) => i.guid), ['fb:1_2']); +}); + +test('each platform is asked no faster than it tolerates', () => { + // Not one number: X goes through a bridge built to be polled, while Instagram + // and Facebook are read with a personal session against platforms that watch + // for exactly that. The cost of asking too fast there is a locked account, + // not a 429. + assert.equal(floorMinutesFor({ social_network: 'x' }), 5); + assert.equal(floorMinutesFor({ social_network: 'instagram' }), 30); + assert.equal(floorMinutesFor({ social_network: 'facebook' }), 60); + + // Reddit is fetched rather than collected and takes the ordinary hour. + assert.equal(floorMinutesFor({ social_network: 'reddit' }), 60); + assert.equal(floorMinutesFor({}), 60); + + // Nothing collected may be asked faster than X. + for (const minutes of Object.values(FLOOR_MINUTES)) assert.ok(minutes >= 5); +}); diff --git a/packages/social/test/platforms.test.js b/packages/social/test/platforms.test.js index 31d08e7..78a1a04 100644 --- a/packages/social/test/platforms.test.js +++ b/packages/social/test/platforms.test.js @@ -122,15 +122,18 @@ test('Page tokens come from structured config, keyed case-insensitively', () => assert.deepEqual(connectedPages({}), []); }); -test('an unconnected Page reschedules for an hour rather than being retired', async () => { +test('a Page with neither a token nor a session reschedules, and is not retired', async () => { const result = await fetchFacebookSource( { social_ref: 'fb:page:somepage', feed_url: 'https://www.facebook.com/SomePage' }, { runtime: { env: {}, onEvent: () => {} } }, ); assert.equal(result.ok, false); - assert.equal(result.throttled, true, 'not collectable is not the same as broken'); + // Nothing is configured to read with, which is a deployment state rather than + // a broken Page — an hour, and no mark against the source. + assert.equal(result.throttled, true, 'not configured is not the same as broken'); assert.equal(result.retryAfter, UNCONFIGURED_SECONDS); + assert.match(result.error, /FB_COOKIE/); }); test('Graph answering 200 with an error object is still a failure', async () => { diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 22d35f5..1f3e4da 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -179,6 +179,9 @@ importers: '@rssamplifier/feed': specifier: workspace:* version: link:../feed + linkedom: + specifier: ^0.18.13 + version: 0.18.13 packages/translate: dependencies: