Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ npx lighthouse https://reader.boringcode.dev --view
- **Scheduled refresh** every 1 hour on wall-clock boundaries
- **Manual refresh** from the header re-fetches the current feed view from backend stored items only; it does **not** re-fetch upstream sources
- **Persisted visited-link dimming** for feed card titles across reload/reopen using local storage
- **Optional article preview** (off by default, toggled in Reader settings) that opens a reader view beside the feed on wide screens or full-screen on phones, with inline links/emphasis, image captions, tables, publication date/reading time, a Workers AI Markdown fallback for PDFs, Office documents, and HTML pages with no readable blocks, source-specific Hacker News comments, and an **Open source** button
- **Optional article preview** (off by default, toggled in Reader settings) that opens a reader view beside the feed on wide screens or full-screen on phones, with inline links/emphasis, image captions, tables, publication date/reading time, a Workers AI Markdown fallback for PDFs, Office documents, and HTML pages with no readable blocks, a Browser Run fallback for JavaScript-rendered pages, source-specific Hacker News comments, and an **Open source** button
- **Saved articles** with an optional per-card bookmark toggle, a `Saved` filter, and a preview-panel toggle; bookmark controls are disabled by default and can be enabled in Reader settings, with saved items stored locally under `feedreader.bookmarks`
- **PWA-ready assets and offline caching** including manifest, service worker, touch icons, cached shell assets, cached `/api/items` responses for previously visited views, a standalone launch veil, and skeleton loading states
- **Reconnect list refresh** re-fetches the current view from backend stored items only; it does **not** refresh upstream sources
Expand Down Expand Up @@ -214,6 +214,8 @@ Bindings configured in `platforms/cloudflare/wrangler.toml`:
- `DB` — D1 database binding
- `ASSETS` — Workers Static Assets binding for `web-static/`
- `SELF` — service binding back to the same Worker for per-source refresh fan-out
- `AI` — Workers AI binding for personalization and the Markdown preview fallback
- `BROWSER` — Browser Run binding for the JavaScript-rendered preview fallback

Runtime configuration:

Expand All @@ -222,6 +224,7 @@ Runtime configuration:
| `FEEDREADER_ITEMS_PER_SOURCE` | `20` | Per-source item count used in source dashboard/health contexts |
| `FEEDREADER_USER_AGENT` | `feedreader/0.1` | Outbound fetch user agent |
| `FEEDREADER_AI_PREVIEW_FALLBACK` | `true` | Workers AI Markdown fallback for PDFs/Office documents and empty previews; `false` disables it |
| `FEEDREADER_BROWSER_PREVIEW_FALLBACK` | `true` | Browser Run fallback for JavaScript-rendered HTML previews; `false` disables it |
| `REFRESH_SECRET` | _required secret_ | Shared secret for internal per-source refresh fan-out routes |

---
Expand Down Expand Up @@ -267,7 +270,7 @@ Returns a reader view for a stored feed item: rich text segments, image blocks w

- `url` — required; must exactly match the URL of an item currently stored in D1

The Worker fetches the article with a 10-second timeout and a 400 KB HTML cap, rejects non-public hosts, and never returns upstream markup. When the built-in extractor finds no readable blocks or the response is a document (PDF/Office), the Worker can convert it through Workers AI's Markdown conversion with a 5 MB cap; set `FEEDREADER_AI_PREVIEW_FALLBACK=false` to disable that fallback. Inline targets and images are restricted to validated public http(s) URLs. Responses are edge-cached for 30 minutes. Returns `404` for unknown URLs and `502` when the upstream page cannot be previewed.
The Worker fetches the article with a 10-second timeout and a 400 KB HTML cap, rejects non-public hosts, and never returns upstream markup. When the built-in extractor finds no readable blocks or the response is a document (PDF/Office), the Worker can convert it through Workers AI's Markdown conversion with a 5 MB cap; if an HTML page still has no readable content, it can render the page through Browser Run. Set `FEEDREADER_AI_PREVIEW_FALLBACK=false` or `FEEDREADER_BROWSER_PREVIEW_FALLBACK=false` to disable either fallback. Inline targets and images are restricted to validated public http(s) URLs. Responses are edge-cached for 30 minutes. Returns `404` for unknown URLs and `502` when the upstream page cannot be previewed.

### `POST /api/refresh`

Expand Down
22 changes: 21 additions & 1 deletion core/previewMarkdown.ts
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,10 @@ export function markdownToPreviewBlocks(
markdown: string,
options: MarkdownPreviewOptions,
): PreviewBlock[] {
const source = stripTrailingJsonLd(markdown.replace(/\r\n?/g, "\n"));
const normalized = stripLeadingFrontmatter(
markdown.replace(/\r\n?/g, "\n"),
);
const source = stripTrailingJsonLd(normalized);
const lines = source.split("\n");
const blocks: PreviewBlock[] = [];
const seenImages = new Set<string>();
Expand Down Expand Up @@ -627,6 +630,23 @@ function codePoint(fallback: string, value: number): string {
return String.fromCodePoint(value);
}

/**
* Browser Run prefixes Markdown with YAML frontmatter when the page exposes
* metadata. Strip a leading `---` block only when it contains at least one
* `key:` line, so a document that starts with a horizontal rule is left
* alone.
*/
function stripLeadingFrontmatter(markdown: string): string {
if (!markdown.startsWith("---\n")) return markdown;
const end = markdown.indexOf("\n---", 4);
if (end === -1) return markdown;
const frontmatter = markdown.slice(4, end);
if (!/^[A-Za-z0-9_-]+:/m.test(frontmatter)) return markdown;
const after = markdown.slice(end + 4);
if (after !== "" && !after.startsWith("\n")) return markdown;
return after.replace(/^\n+/, "");
}

/**
* HTML conversion appends the page's JSON-LD at the end of the Markdown.
* Remove only a trailing JSON-LD-shaped object/array so a legitimate JSON
Expand Down
49 changes: 49 additions & 0 deletions core/test/previewMarkdown.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -303,6 +303,55 @@ A long enough paragraph that should remain in the preview.`;
).toEqual([{ type: "paragraph", text: markdown }]);
});

it("strips YAML frontmatter returned by Browser Run", () => {
const markdown = `---
title: "TodoMVC: React"
meta:
description: "A TodoMVC written in React."
---

### Rendered

A long enough paragraph rendered after JavaScript executes.`;

expect(
markdownToPreviewBlocks(markdown, {
pageUrl: "https://example.com/posts/rendered",
}),
).toEqual([
{ type: "heading", text: "Rendered" },
{
type: "paragraph",
text: "A long enough paragraph rendered after JavaScript executes.",
},
]);
});

it("does not treat horizontal rules as frontmatter", () => {
const markdown = `---

A long enough paragraph that stays after the first rule.

---

Another long enough paragraph after the second rule.`;

expect(
markdownToPreviewBlocks(markdown, {
pageUrl: "https://example.com/posts/rules",
}),
).toEqual([
{
type: "paragraph",
text: "A long enough paragraph that stays after the first rule.",
},
{
type: "paragraph",
text: "Another long enough paragraph after the second rule.",
},
]);
});

it("returns no blocks for empty or JSON-LD-only input", () => {
expect(
markdownToPreviewBlocks("", {
Expand Down
5 changes: 3 additions & 2 deletions docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
| [`platforms/cloudflare/src/index.ts`](../platforms/cloudflare/src/index.ts) | Worker request router and scheduled entrypoint. |
| [`platforms/cloudflare/src/repository.ts`](../platforms/cloudflare/src/repository.ts) | D1-backed `FeedRepository` implementation. |
| [`platforms/cloudflare/src/previewFallback.ts`](../platforms/cloudflare/src/previewFallback.ts) | Workers AI `toMarkdown` adapter for document/empty previews. |
| [`platforms/cloudflare/src/browserPreviewFallback.ts`](../platforms/cloudflare/src/browserPreviewFallback.ts) | Browser Run markdown adapter for JavaScript-rendered pages. |
| [`platforms/cloudflare/migrations/`](../platforms/cloudflare/migrations) | D1 schema migrations. |
| [`web-static/`](../web-static) | Browser assets: CSS, JS, icons, manifest, service worker. |

Expand Down Expand Up @@ -77,7 +78,7 @@ This keeps refresh logic, rendering, and feed shaping independent of D1-specific
## Caching

- `GET /` is cached per URL (including the raw query string) for five minutes through the Cache API. `APP_VERSION` is part of the key and changes on every deploy, so a release invalidates old HTML; local `wrangler dev` keeps `APP_VERSION` at `dev` and skips the home cache so template/CSS edits are never hidden. The browser-facing response is `private, no-store`: the five-minute copy is internal to the Cache API, which keeps Cloudflare's Browser Cache TTL setting from rewriting the client TTL.
- `GET /api/items` is edge-cached until the underlying sources next refresh (the key includes `sync_state.last_success_at`). Successful article previews, including AI-converted ones, are cached for 30 minutes. Both endpoints use the same split as the home page: the Cache API holds the edge copy, while clients receive `private, no-store`, so the CDN/Browser Cache TTL cannot extend the client TTL.
- `GET /api/items` is edge-cached until the underlying sources next refresh (the key includes `sync_state.last_success_at`). Successful article previews, including AI- and Browser Run-converted ones, are cached for 30 minutes. Both endpoints use the same split as the home page: the Cache API holds the edge copy, while clients receive `private, no-store`, so the CDN/Browser Cache TTL cannot extend the client TTL.
- `style.css` and `theme-init.js` are read from the `ASSETS` binding once per isolate and inlined into the server-rendered HTML, so first paint has no render-blocking network requests. The files remain the single source of truth and are still served directly for cached shells.
- [`web-static/_headers`](../web-static/_headers) marks versioned static assets as immutable for a year; `service-worker.js` keeps `must-revalidate` so updates land.

Expand All @@ -99,7 +100,7 @@ The optional article preview is served by `GET /api/preview?url=…`.
- The requested URL must exactly match a stored `items.url`. This check, not the fetch layer, is what keeps the endpoint from acting as an open proxy; `feedreader.articlePreviewEnabled` only controls the client behavior.
- `core/preview.ts` additionally rejects non-http(s) schemes, credentials, non-standard ports, and loopback/private/link-local hosts, and re-checks the final URL after redirects.
- Article fetches time out after 10 seconds and read at most 400 KB of HTML (5 MB when a document is handed to the optional fallback). Transient upstream `429`/`5xx` responses are retried once with jitter. `PreviewUnavailableError` carries a structured reason (`blocked`, `rate_limited`, `upstream_error`, `timeout`, `unsupported_content_type`, `too_large`, `empty_body`, `redirect`, `invalid_url`) plus the optional upstream status; the endpoint logs `preview_unavailable`/`preview_empty` events and returns the reason so the client can show a specific message. The extractor removes scripts/navigation/footer noise and returns structured text segments, image blocks, and table blocks only — upstream HTML is never returned or injected. Text renders with `textContent`/DOM nodes; inline anchors are limited to URLs that pass the same public http(s) validation, and image URLs are re-validated before the browser loads them directly with `loading="lazy"` and `referrerpolicy="no-referrer"`.
- If the built-in extractor finds no blocks (`no_content`), truncation left no blocks (`too_large`), or the upstream is a non-HTML document (`unsupported_content_type`), `fetchArticlePreview` may call an injected fallback. Core stays AI-free: the Worker supplies `platforms/cloudflare/src/previewFallback.ts`, which calls `env.AI.toMarkdown`, maps the converted Markdown through `core/previewMarkdown.ts`, and returns the same validated `PreviewBlock` shape. The mapper re-checks every link and image with `normalizePreviewUrl`, strips trailing JSON-LD, and drops PDF filename/page-marker headings. Blocked, timeout, rate-limited, and empty-body failures never reach the fallback. `FEEDREADER_AI_PREVIEW_FALLBACK=false` disables it.
- If the built-in extractor finds no blocks (`no_content`), truncation left no blocks (`too_large`), or the upstream is a non-HTML document (`unsupported_content_type`), `fetchArticlePreview` may call an injected fallback. Core stays platform-free: the Worker supplies a chain (`platforms/cloudflare/src/previewFallbackChain.ts`). Workers AI (`previewFallback.ts`) calls `env.AI.toMarkdown`; Browser Run (`browserPreviewFallback.ts`) calls `env.BROWSER.quickAction("markdown", …)` only for `no_content` after AI fails. Both map the converted Markdown through `core/previewMarkdown.ts`, which re-checks every link and image with `normalizePreviewUrl`, strips YAML frontmatter and trailing JSON-LD, and drops PDF filename/page-marker headings. Browser Run waits for `networkidle2` with 8-second navigation/action and 9-second outer timeouts, rejects images/media/fonts and literal private-host requests, and validates the rendered page status and `meta.finalUrl`. Blocked, timeout, rate-limited, and empty-body failures never reach the fallback. `FEEDREADER_AI_PREVIEW_FALLBACK=false` and `FEEDREADER_BROWSER_PREVIEW_FALLBACK=false` disable each adapter.
- Image blocks may include a `figcaption`-derived caption and intrinsic width/height to reserve layout space. Known avatar/profile-image hosts and paths, SVG wordmarks, and common badge hosts are dropped before the image cap so author-list and README chrome doesn't displace article figures. Table blocks carry a header flag and capped rows/cells. The response also includes byline, publication date, and an estimated reading time. For Hugging Face paper URLs the extractor starts at the Abstract section, then appends the paper's resource links and citing model, dataset, space, and collection lists while excluding community comment chrome; GitHub titles drop the site prefix. The preview and card `…` dialogs expose a stored Hacker News comments URL for HN items; other sources omit it.
- The client highlights the card matching the open preview and provides previous/next navigation over the currently rendered card list, skipping duplicate or unusable cards. When the last rendered card is previewed, Next remains enabled if another page exists and appends it before advancing. On mobile the open panel pins the body scroll position (and reapplies it after rotation), the glass header/footer auto-hide on scroll down and return on scroll up, and the `theme-color` tint switches to the panel surface while the panel covers the viewport. A left-edge rightward drag follows the finger and reveals the preserved feed position underneath; releasing past a distance/velocity threshold slides the panel away and closes it, while a shorter drag springs back. Vertical edge drags remain native scrolling and the gesture is inactive while the `…` dialog is open.
- Card host rows keep the optional bookmark toggle and a more-actions dialog trigger. The dialog exposes Share, the stored Hacker News comments URL when present, and Open source; choosing an action does not open the preview.
Expand Down
2 changes: 1 addition & 1 deletion docs/DEPLOYMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,6 @@ The checked-in `wrangler.toml` config does not hardcode a route. Domain attachme
- Source refreshes fetch one page per source per hourly fan-out. Hugging Face reads about 272 KB of JSON from `/api/daily_papers` instead of the ~1.4 MB `/papers/trending` HTML page; source error details are capped at 200 characters, and HTML error pages are reduced to a fixed message before they reach D1 or the UI.
- The five-minute home-page cache absorbs most `GET /` traffic, so only cache misses run the feed query and render the HTML. The browser response is `private, no-store`; edge caching is the Worker Cache API, not the CDN, so the zone's Browser Cache TTL setting cannot rewrite the client TTL. Each cold-isolate cache miss also reads `style.css` and `theme-init.js` through the `ASSETS` binding (two local subrequests), then reuses them in memory.
- The standalone launch veil and skeleton styles are inlined with the existing HTML/CSS, so they add no extra requests. Bump the shell cache version when the HTML shell or `app.js` changes; see [RUNBOOK.md](RUNBOOK.md#pwa-shell-updates).
- Article previews add one outbound fetch (and one `node-html-parser` parse) per uncached `/api/preview` request. The 400 KB HTML body cap and 30-minute edge cache in `core/preview.ts` bound that cost; transient upstream `429`/`5xx` responses are retried once. PDFs/Office documents and HTML pages with no readable blocks can additionally call Workers AI's `toMarkdown` (5 MB download cap; document text conversion is free and the fallback disables AI image-description conversion). That call runs only on a cache miss and its result is cached for the same 30 minutes; set `FEEDREADER_AI_PREVIEW_FALLBACK=false` to disable it. Browser-facing preview and `/api/items` responses are `private, no-store`; their edge copies live in the Cache API. Watch the CPU-time panel if preview usage is heavy — bursts can hit the Workers Free 10 ms limit, so see [RUNBOOK.md](RUNBOOK.md#article-preview-failures). Richer preview blocks (inline links, captions, tables) increase the cached response payload but not the upstream fetch count; the same byte cap and cache TTL apply.
- Article previews add one outbound fetch (and one `node-html-parser` parse) per uncached `/api/preview` request. The 400 KB HTML body cap and 30-minute edge cache in `core/preview.ts` bound that cost; transient upstream `429`/`5xx` responses are retried once. PDFs/Office documents and HTML pages with no readable blocks can additionally call Workers AI's `toMarkdown` (5 MB download cap; document text conversion is free and the fallback disables AI image-description conversion). That call runs only on a cache miss and its result is cached for the same 30 minutes; set `FEEDREADER_AI_PREVIEW_FALLBACK=false` to disable it. HTML pages that still have no readable content after AI conversion can call Browser Run's `markdown` Quick Action; Workers Free includes 10 browser-minutes/day and paces Quick Actions to one request per 10 seconds. Browser Run runs only on a cache miss, uses no browser sessions (so the concurrent-browser limit does not apply), and its result is cached for 30 minutes; set `FEEDREADER_BROWSER_PREVIEW_FALLBACK=false` to disable it. Browser-facing preview and `/api/items` responses are `private, no-store`; their edge copies live in the Cache API. Watch the CPU-time panel if preview usage is heavy — bursts can hit the Workers Free 10 ms limit, so see [RUNBOOK.md](RUNBOOK.md#article-preview-failures). Richer preview blocks (inline links, captions, tables) increase the cached response payload but not the upstream fetch count; the same byte cap and cache TTL apply.

Check the Cloudflare dashboard's D1 usage and Workers CPU-time panels periodically — see [RUNBOOK.md](RUNBOOK.md) for what to do if a source consistently nears the CPU ceiling.
Loading
Loading