Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -270,7 +270,11 @@ Returns a reader view for a stored feed item: rich text segments, image blocks w

- `url` — required; must exactly match the URL of an item currently stored in D1

The Worker fetches the article with a 10-second timeout and a 400 KB HTML cap, rejects non-public hosts, and never returns upstream markup. When the built-in extractor finds no readable blocks or the response is a document (PDF/Office), the Worker can convert it through Workers AI's Markdown conversion with a 5 MB cap; if an HTML page still has no readable content, it can render the page through Browser Run. Set `FEEDREADER_AI_PREVIEW_FALLBACK=false` or `FEEDREADER_BROWSER_PREVIEW_FALLBACK=false` to disable either fallback. Inline targets and images are restricted to validated public http(s) URLs. Responses are edge-cached for 30 minutes (empty `no_content` results for 5 minutes); cache-miss previews are limited to 20 per minute per IP, while cached previews are unrestricted. Returns `404` for unknown URLs and `502` when the upstream page cannot be previewed.
The Worker fetches the article with a 10-second timeout and a 400 KB HTML cap, rejects non-public hosts, and never returns upstream markup. When the built-in extractor finds no readable blocks or the response is a document (PDF/Office), the Worker can convert it through Workers AI's Markdown conversion with a 5 MB cap; if an HTML page still has no readable content, it can render the page through Browser Run. Set `FEEDREADER_AI_PREVIEW_FALLBACK=false` or `FEEDREADER_BROWSER_PREVIEW_FALLBACK=false` to disable either fallback. Inline targets and images are restricted to validated public http(s) URLs. Responses are edge-cached for 30 minutes (empty `no_content` results for 5 minutes); cache-miss previews are limited to 20 per minute per IP, while cached previews are unrestricted. The client retries a transient `503` once. Returns `404` for unknown URLs and `502` when the upstream page cannot be previewed.

### `POST /api/personalize`

Returns a `For You` ranking for an interests string. Cache-miss requests are limited to 10 per minute per IP; cached rankings are unrestricted.

### `POST /api/refresh`

Expand Down
2 changes: 1 addition & 1 deletion core/render.ts
Original file line number Diff line number Diff line change
Expand Up @@ -195,7 +195,7 @@ export function renderIndexPage(data: PageData): string {
<link rel="apple-touch-icon" href="/apple-touch-icon.png?v=8" />
<script>${data.themeInitScript}</script>
<style>${data.inlineCss}</style>
<script src="/static/app.js?v=109" defer></script>
<script src="/static/app.js?v=110" defer></script>
</head>
<body data-app-version="${escapeHtml(data.appVersion)}">
<div class="boot-veil" data-boot-veil aria-hidden="true">
Expand Down
2 changes: 1 addition & 1 deletion core/test/render.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ describe("renderIndexPage", () => {
expect(html).toContain(
"<script>document.documentElement.dataset.theme='dark';</script>",
);
expect(html).toContain('/static/app.js?v=109');
expect(html).toContain('/static/app.js?v=110');
expect(html).not.toContain("style.css");
expect(html).not.toContain("theme-init.js");
});
Expand Down
6 changes: 3 additions & 3 deletions docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
| [`platforms/cloudflare/src/repository.ts`](../platforms/cloudflare/src/repository.ts) | D1-backed `FeedRepository` implementation. |
| [`platforms/cloudflare/src/previewFallback.ts`](../platforms/cloudflare/src/previewFallback.ts) | Workers AI `toMarkdown` adapter for document/empty previews. |
| [`platforms/cloudflare/src/browserPreviewFallback.ts`](../platforms/cloudflare/src/browserPreviewFallback.ts) | Browser Run markdown adapter for JavaScript-rendered pages. |
| [`platforms/cloudflare/src/previewRateLimit.ts`](../platforms/cloudflare/src/previewRateLimit.ts) | Per-IP fixed-window limiter for preview cache misses. |
| [`platforms/cloudflare/src/rateLimit.ts`](../platforms/cloudflare/src/rateLimit.ts) | Per-IP fixed-window limiter for preview and personalization cache misses. |
| [`platforms/cloudflare/migrations/`](../platforms/cloudflare/migrations) | D1 schema migrations. |
| [`web-static/`](../web-static) | Browser assets: CSS, JS, icons, manifest, service worker. |

Expand All @@ -41,7 +41,7 @@ Browser-only preferences and reading state live in `localStorage`: theme prefere

- The inline pre-paint theme script also records `display-mode: standalone` (and iOS `navigator.standalone`) as `data-standalone`. The server renders a lightweight boot veil; CSS only displays it in an installed app, and `app.js` fades and removes it after the first content frame. Browser tabs never see it, so it does not affect LCP.
- Client-side feed loads use skeleton cards. A 150 ms delay avoids flashing placeholders on fast responses when a feed is already on screen (an empty grid shows them immediately); replace loads snapshot the current grid so a failed request can restore it, and append loads add three skeleton rows at the bottom.
- The reader preview uses the same delayed-skeleton idea: the card title stays visible, a preview skeleton replaces the visible loading message after 150 ms, and the loaded title/content fade in. Aborts and failures clear the skeleton and restore the readable status message.
- The reader preview uses the same delayed-skeleton idea: the card title stays visible, a preview skeleton replaces the visible loading message after 150 ms, and the loaded title/content fade in. Aborts and failures clear the skeleton and restore the readable status message. A transient `503` from the preview endpoint is retried once after 500 ms; `429`s are not retried because the limiter's `Retry-After` is 60 seconds.
- Skeleton cards are `aria-hidden`; the grid sets `aria-busy`, and a visually hidden live region carries the loading message.
- Completed replace loads crossfade through the View Transitions API when available, with a Web Animations fade fallback. `prefers-reduced-motion: reduce` disables shimmer and both transition paths.
- The search field hides the browser's native `type="search"` clear control; the app's search toggle/close button is the only clear affordance.
Expand Down Expand Up @@ -79,7 +79,7 @@ This keeps refresh logic, rendering, and feed shaping independent of D1-specific
## Caching

- `GET /` is cached per URL (including the raw query string) for five minutes through the Cache API. `APP_VERSION` is part of the key and changes on every deploy, so a release invalidates old HTML; local `wrangler dev` keeps `APP_VERSION` at `dev` and skips the home cache so template/CSS edits are never hidden. The browser-facing response is `private, no-store`: the five-minute copy is internal to the Cache API, which keeps Cloudflare's Browser Cache TTL setting from rewriting the client TTL.
- `GET /api/items` is edge-cached until the underlying sources next refresh (the key includes `sync_state.last_success_at`). Successful article previews, including AI- and Browser Run-converted ones, are cached for 30 minutes; empty `no_content` results use a 5-minute TTL so a transient fallback failure can retry. Cache-miss previews pass a per-IP fixed-window limiter (Cache API, 20 per minute, per-colo, fail-open); cache hits bypass it. Both endpoints use the same split as the home page: the Cache API holds the edge copy, while clients receive `private, no-store`, so the CDN/Browser Cache TTL cannot extend the client TTL.
- `GET /api/items` is edge-cached until the underlying sources next refresh (the key includes `sync_state.last_success_at`). Successful article previews, including AI- and Browser Run-converted ones, are cached for 30 minutes; empty `no_content` results use a 5-minute TTL so a transient fallback failure can retry. Cache-miss previews pass a per-IP fixed-window limiter (Cache API, 20 per minute, per-colo, fail-open); cache-miss `/api/personalize` requests use the same limiter under a separate 10-per-minute bucket, while cached rankings bypass it. Both endpoints use the same split as the home page: the Cache API holds the edge copy, while clients receive `private, no-store`, so the CDN/Browser Cache TTL cannot extend the client TTL.
- `style.css` and `theme-init.js` are read from the `ASSETS` binding once per isolate and inlined into the server-rendered HTML, so first paint has no render-blocking network requests. The files remain the single source of truth and are still served directly for cached shells.
- [`web-static/_headers`](../web-static/_headers) marks versioned static assets as immutable for a year; `service-worker.js` keeps `must-revalidate` so updates land.

Expand Down
3 changes: 2 additions & 1 deletion docs/DEPLOYMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@ The checked-in `wrangler.toml` config does not hardcode a route. Domain attachme
- Source refreshes fetch one page per source per hourly fan-out. Hugging Face reads about 272 KB of JSON from `/api/daily_papers` instead of the ~1.4 MB `/papers/trending` HTML page; source error details are capped at 200 characters, and HTML error pages are reduced to a fixed message before they reach D1 or the UI.
- The five-minute home-page cache absorbs most `GET /` traffic, so only cache misses run the feed query and render the HTML. The browser response is `private, no-store`; edge caching is the Worker Cache API, not the CDN, so the zone's Browser Cache TTL setting cannot rewrite the client TTL. Each cold-isolate cache miss also reads `style.css` and `theme-init.js` through the `ASSETS` binding (two local subrequests), then reuses them in memory.
- The standalone launch veil and skeleton styles are inlined with the existing HTML/CSS, so they add no extra requests. Bump the shell cache version when the HTML shell or `app.js` changes; see [RUNBOOK.md](RUNBOOK.md#pwa-shell-updates).
- Article previews add one outbound fetch (and one `node-html-parser` parse) per uncached `/api/preview` request. The 400 KB HTML body cap, 30-minute edge cache (5 minutes for empty results), and a per-IP limiter (20 cache-miss previews per minute; cached previews bypass it) bound that cost; transient upstream `429`/`5xx` responses are retried once. PDFs/Office documents and HTML pages with no readable blocks can additionally call Workers AI's `toMarkdown` (5 MB download cap; document text conversion is free and the fallback disables AI image-description conversion). That call runs only on a cache miss and its result is cached for the same 30 minutes; set `FEEDREADER_AI_PREVIEW_FALLBACK=false` to disable it. HTML pages that still have no readable content after AI conversion can call Browser Run's `markdown` Quick Action; Workers Free includes 10 browser-minutes/day and paces Quick Actions to one request per 10 seconds. Browser Run runs only on a cache miss, uses no browser sessions (so the concurrent-browser limit does not apply), and its result is cached for 30 minutes; set `FEEDREADER_BROWSER_PREVIEW_FALLBACK=false` to disable it. Hugging Face paper pages parse only `<head>`+`<main>` with one combined noise selector, reducing their extraction CPU. Browser-facing preview and `/api/items` responses are `private, no-store`; their edge copies live in the Cache API. Watch the CPU-time panel if preview usage is heavy — bursts can hit the Workers Free 10 ms limit, so see [RUNBOOK.md](RUNBOOK.md#article-preview-failures). Richer preview blocks (inline links, captions, tables) increase the cached response payload but not the upstream fetch count; the same byte cap and cache TTL apply.
- Article previews add one outbound fetch (and one `node-html-parser` parse) per uncached `/api/preview` request. The 400 KB HTML body cap, 30-minute edge cache (5 minutes for empty results), and a per-IP limiter (20 cache-miss previews per minute; cached previews bypass it) bound that cost; transient upstream `429`/`5xx` responses are retried once. PDFs/Office documents and HTML pages with no readable blocks can additionally call Workers AI's `toMarkdown` (5 MB download cap; document text conversion is free and the fallback disables AI image-description conversion). That call runs only on a cache miss and its result is cached for the same 30 minutes; set `FEEDREADER_AI_PREVIEW_FALLBACK=false` to disable it. HTML pages that still have no readable content after AI conversion can call Browser Run's `markdown` Quick Action; Workers Free includes 10 browser-minutes/day and paces Quick Actions to one request per 10 seconds. Browser Run runs only on a cache miss, uses no browser sessions (so the concurrent-browser limit does not apply), and its result is cached for 30 minutes; set `FEEDREADER_BROWSER_PREVIEW_FALLBACK=false` to disable it. Hugging Face paper pages parse only `<head>`+`<main>` with one combined noise selector, reducing their extraction CPU. The browser retries a preview `503` once after 500 ms; `429`s are not retried. Browser-facing preview and `/api/items` responses are `private, no-store`; their edge copies live in the Cache API. Watch the CPU-time panel if preview usage is heavy — bursts can hit the Workers Free 10 ms limit, so see [RUNBOOK.md](RUNBOOK.md#article-preview-failures). Richer preview blocks (inline links, captions, tables) increase the cached response payload but not the upstream fetch count; the same byte cap and cache TTL apply.
- `/api/personalize` cache misses are limited to 10 per minute per IP by the shared Cache API limiter; cached rankings bypass it. This protects the embedding and optional LLM-polish budget from varied interests strings; blocked requests return `429` with `Retry-After: 60`.

Check the Cloudflare dashboard's D1 usage and Workers CPU-time panels periodically — see [RUNBOOK.md](RUNBOOK.md) for what to do if a source consistently nears the CPU ceiling.
4 changes: 3 additions & 1 deletion docs/RUNBOOK.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,8 @@ New migrations go in `platforms/cloudflare/migrations/` as `NNNN_description.sql

**Neuron budget**: the LLM polish step (one chat-model call over up to ~30 item titles/summaries) dominates Workers AI spend, not the embedder (one batched embed call per source per refresh cycle, only for items missing a vector, plus one tiny embed call per personalize cache-miss request). Under budget pressure, the first lever to pull is `FEEDREADER_PERSONALIZE_POLISH_POOL_SIZE`, not the embedding model.

**Rate limit**: cache-miss `/api/personalize` requests are limited to 10 per minute per IP by the shared Cache API limiter (`platforms/cloudflare/src/rateLimit.ts`, namespace `personalize`). Cache hits bypass it. A blocked request returns `429` with `reason: "rate_limited"` and logs `personalize_rate_limited`; interests and client IPs are not logged. The same limiter protects preview cache misses under its own 20-per-minute bucket.

**Manual verification** (no D1/Miniflare integration tests exist for this — same convention as the rest of `D1Repository`):

```bash
Expand Down Expand Up @@ -91,7 +93,7 @@ Before returning `unsupported_content_type` for a non-HTML response, `too_large`

If the AI conversion returns nothing for an HTML page, the chain calls Browser Run's `markdown` Quick Action (`platforms/cloudflare/src/browserPreviewFallback.ts`) before returning `no_content`. It is limited to `no_content`; PDFs/Office documents, `too_large`, `blocked`, `timeout`, and rate-limited failures never reach it. Browser Run waits for `networkidle2` with 8-second navigation/action and 9-second outer timeouts, blocks images/media/fonts and literal private-host requests, and validates the rendered page status and final URL. Workers Free includes 10 browser-minutes/day and paces Quick Actions to one request per 10 seconds; a `429` is logged and treated as no preview. Set `FEEDREADER_BROWSER_PREVIEW_FALLBACK=false` to disable it. Browser Run successes are cached for 30 minutes; the preview cache key is `preview:v11`.

The Worker logs `preview_unavailable` and `preview_empty` events with the URL, reason, and upstream status; `preview_empty` also records `cache_ttl_seconds`. It logs a `preview_ai_fallback` event for each AI attempt (`kind`, `outcome` = `ok`/`empty`/`error`, `ms`, `blocks`, `tokens`, and a capped `error`), a `preview_browser_fallback` event for each Browser Run attempt (`outcome`, `status`, `page_status`, `browser_ms`, `blocks`, and a capped `error`), and `preview_rate_limited` when the local per-IP limiter rejects a cache miss. Successful previews, including AI- and Browser Run-converted ones, are edge-cached for 30 minutes; empty `no_content` results use a 5-minute TTL so a transient fallback failure can retry. Cache-miss previews pass a per-IP fixed-window limiter (Cache API, 20 per minute, per-colo, fail-open); cache hits bypass it. Because the in-Worker limiter is per-colo and fail-open, add a zone-level Cloudflare Rate Limiting rule for `/api/preview` if you need a global limit. Browser-facing responses are `private, no-store`. Transient upstream `429`/`5xx` responses are retried once with jitter. If bursts produce transient `503`s, check Workers Logs/CPU: the extractor runs in the request invocation, and sequential retries succeeding usually points at the Workers Free 10 ms CPU limit rather than a lasting upstream block. Hugging Face paper pages are parsed as `<head>`+`<main>` with one combined noise selector to keep them closer to that budget; if `exceededCpu` 503s recur for HF URLs, the next lever is the paper Markdown-content endpoint or a stored-summary fallback. Lowering `DEFAULT_MAX_BYTES` in `core/preview.ts` (400 KB today) reduces the parse cost.
The Worker logs `preview_unavailable` and `preview_empty` events with the URL, reason, and upstream status; `preview_empty` also records `cache_ttl_seconds`. It logs a `preview_ai_fallback` event for each AI attempt (`kind`, `outcome` = `ok`/`empty`/`error`, `ms`, `blocks`, `tokens`, and a capped `error`), a `preview_browser_fallback` event for each Browser Run attempt (`outcome`, `status`, `page_status`, `browser_ms`, `blocks`, and a capped `error`), and `preview_rate_limited` when the local per-IP limiter rejects a cache miss. Successful previews, including AI- and Browser Run-converted ones, are edge-cached for 30 minutes; empty `no_content` results use a 5-minute TTL so a transient fallback failure can retry. Cache-miss previews pass a per-IP fixed-window limiter (Cache API, 20 per minute, per-colo, fail-open); cache hits bypass it. Because the in-Worker limiter is per-colo and fail-open, add a zone-level Cloudflare Rate Limiting rule for `/api/preview` if you need a global limit. The client retries a preview `503` once after 500 ms; `429`s are not retried because `Retry-After` is 60 seconds. Browser-facing responses are `private, no-store`. Transient upstream `429`/`5xx` responses are retried once with jitter. If bursts produce transient `503`s, check Workers Logs/CPU: the extractor runs in the request invocation, and sequential retries succeeding usually points at the Workers Free 10 ms CPU limit rather than a lasting upstream block. Hugging Face paper pages are parsed as `<head>`+`<main>` with one combined noise selector to keep them closer to that budget; if `exceededCpu` 503s recur for HF URLs, the next lever is the paper Markdown-content endpoint or a stored-summary fallback. Lowering `DEFAULT_MAX_BYTES` in `core/preview.ts` (400 KB today) reduces the parse cost.

## Local SSR smoke procedure

Expand Down
Loading
Loading