Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -270,7 +270,7 @@ Returns a reader view for a stored feed item: rich text segments, image blocks w

- `url` — required; must exactly match the URL of an item currently stored in D1

The Worker fetches the article with a 10-second timeout and a 400 KB HTML cap, rejects non-public hosts, and never returns upstream markup. When the built-in extractor finds no readable blocks or the response is a document (PDF/Office), the Worker can convert it through Workers AI's Markdown conversion with a 5 MB cap; if an HTML page still has no readable content, it can render the page through Browser Run. Set `FEEDREADER_AI_PREVIEW_FALLBACK=false` or `FEEDREADER_BROWSER_PREVIEW_FALLBACK=false` to disable either fallback. Inline targets and images are restricted to validated public http(s) URLs. Responses are edge-cached for 30 minutes. Returns `404` for unknown URLs and `502` when the upstream page cannot be previewed.
The Worker fetches the article with a 10-second timeout and a 400 KB HTML cap, rejects non-public hosts, and never returns upstream markup. When the built-in extractor finds no readable blocks or the response is a document (PDF/Office), the Worker can convert it through Workers AI's Markdown conversion with a 5 MB cap; if an HTML page still has no readable content, it can render the page through Browser Run. Set `FEEDREADER_AI_PREVIEW_FALLBACK=false` or `FEEDREADER_BROWSER_PREVIEW_FALLBACK=false` to disable either fallback. Inline targets and images are restricted to validated public http(s) URLs. Responses are edge-cached for 30 minutes (empty `no_content` results for 5 minutes); cache-miss previews are limited to 20 per minute per IP, while cached previews are unrestricted. Returns `404` for unknown URLs and `502` when the upstream page cannot be previewed.

### `POST /api/refresh`

Expand Down
5 changes: 3 additions & 2 deletions docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
| [`platforms/cloudflare/src/repository.ts`](../platforms/cloudflare/src/repository.ts) | D1-backed `FeedRepository` implementation. |
| [`platforms/cloudflare/src/previewFallback.ts`](../platforms/cloudflare/src/previewFallback.ts) | Workers AI `toMarkdown` adapter for document/empty previews. |
| [`platforms/cloudflare/src/browserPreviewFallback.ts`](../platforms/cloudflare/src/browserPreviewFallback.ts) | Browser Run markdown adapter for JavaScript-rendered pages. |
| [`platforms/cloudflare/src/previewRateLimit.ts`](../platforms/cloudflare/src/previewRateLimit.ts) | Per-IP fixed-window limiter for preview cache misses. |
| [`platforms/cloudflare/migrations/`](../platforms/cloudflare/migrations) | D1 schema migrations. |
| [`web-static/`](../web-static) | Browser assets: CSS, JS, icons, manifest, service worker. |

Expand Down Expand Up @@ -78,7 +79,7 @@ This keeps refresh logic, rendering, and feed shaping independent of D1-specific
## Caching

- `GET /` is cached per URL (including the raw query string) for five minutes through the Cache API. `APP_VERSION` is part of the key and changes on every deploy, so a release invalidates old HTML; local `wrangler dev` keeps `APP_VERSION` at `dev` and skips the home cache so template/CSS edits are never hidden. The browser-facing response is `private, no-store`: the five-minute copy is internal to the Cache API, which keeps Cloudflare's Browser Cache TTL setting from rewriting the client TTL.
- `GET /api/items` is edge-cached until the underlying sources next refresh (the key includes `sync_state.last_success_at`). Successful article previews, including AI- and Browser Run-converted ones, are cached for 30 minutes. Both endpoints use the same split as the home page: the Cache API holds the edge copy, while clients receive `private, no-store`, so the CDN/Browser Cache TTL cannot extend the client TTL.
- `GET /api/items` is edge-cached until the underlying sources next refresh (the key includes `sync_state.last_success_at`). Successful article previews, including AI- and Browser Run-converted ones, are cached for 30 minutes; empty `no_content` results use a 5-minute TTL so a transient fallback failure can retry. Cache-miss previews pass a per-IP fixed-window limiter (Cache API, 20 per minute, per-colo, fail-open); cache hits bypass it. Both endpoints use the same split as the home page: the Cache API holds the edge copy, while clients receive `private, no-store`, so the CDN/Browser Cache TTL cannot extend the client TTL.
- `style.css` and `theme-init.js` are read from the `ASSETS` binding once per isolate and inlined into the server-rendered HTML, so first paint has no render-blocking network requests. The files remain the single source of truth and are still served directly for cached shells.
- [`web-static/_headers`](../web-static/_headers) marks versioned static assets as immutable for a year; `service-worker.js` keeps `must-revalidate` so updates land.

Expand All @@ -104,6 +105,6 @@ The optional article preview is served by `GET /api/preview?url=…`.
- Image blocks may include a `figcaption`-derived caption and intrinsic width/height to reserve layout space. Known avatar/profile-image hosts and paths, SVG wordmarks, and common badge hosts are dropped before the image cap so author-list and README chrome doesn't displace article figures. Table blocks carry a header flag and capped rows/cells. The response also includes byline, publication date, and an estimated reading time. For Hugging Face paper URLs the extractor starts at the Abstract section, then appends the paper's resource links and citing model, dataset, space, and collection lists while excluding community comment chrome; GitHub titles drop the site prefix. The preview and card `…` dialogs expose a stored Hacker News comments URL for HN items; other sources omit it.
- The client highlights the card matching the open preview and provides previous/next navigation over the currently rendered card list, skipping duplicate or unusable cards. When the last rendered card is previewed, Next remains enabled if another page exists and appends it before advancing. On mobile the open panel pins the body scroll position (and reapplies it after rotation), the glass header/footer auto-hide on scroll down and return on scroll up, and the `theme-color` tint switches to the panel surface while the panel covers the viewport. A left-edge rightward drag follows the finger and reveals the preserved feed position underneath; releasing past a distance/velocity threshold slides the panel away and closes it, while a shorter drag springs back. Vertical edge drags remain native scrolling and the gesture is inactive while the `…` dialog is open.
- Card host rows keep the optional bookmark toggle and a more-actions dialog trigger. The dialog exposes Share, the stored Hacker News comments URL when present, and Open source; choosing an action does not open the preview.
- Successful previews are edge-cached for 30 minutes via the Cache API; the browser-facing response is `private, no-store`, so the CDN cannot extend the client TTL.
- Successful previews are edge-cached for 30 minutes (empty results for 5 minutes) via the Cache API; cache-miss requests are limited to 20 per minute per IP. The browser-facing response is `private, no-store`, so the CDN cannot extend the client TTL.

See [RUNBOOK.md](RUNBOOK.md) for failure modes.
2 changes: 1 addition & 1 deletion docs/DEPLOYMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,6 @@ The checked-in `wrangler.toml` config does not hardcode a route. Domain attachme
- Source refreshes fetch one page per source per hourly fan-out. Hugging Face reads about 272 KB of JSON from `/api/daily_papers` instead of the ~1.4 MB `/papers/trending` HTML page; source error details are capped at 200 characters, and HTML error pages are reduced to a fixed message before they reach D1 or the UI.
- The five-minute home-page cache absorbs most `GET /` traffic, so only cache misses run the feed query and render the HTML. The browser response is `private, no-store`; edge caching is the Worker Cache API, not the CDN, so the zone's Browser Cache TTL setting cannot rewrite the client TTL. Each cold-isolate cache miss also reads `style.css` and `theme-init.js` through the `ASSETS` binding (two local subrequests), then reuses them in memory.
- The standalone launch veil and skeleton styles are inlined with the existing HTML/CSS, so they add no extra requests. Bump the shell cache version when the HTML shell or `app.js` changes; see [RUNBOOK.md](RUNBOOK.md#pwa-shell-updates).
- Article previews add one outbound fetch (and one `node-html-parser` parse) per uncached `/api/preview` request. The 400 KB HTML body cap and 30-minute edge cache in `core/preview.ts` bound that cost; transient upstream `429`/`5xx` responses are retried once. PDFs/Office documents and HTML pages with no readable blocks can additionally call Workers AI's `toMarkdown` (5 MB download cap; document text conversion is free and the fallback disables AI image-description conversion). That call runs only on a cache miss and its result is cached for the same 30 minutes; set `FEEDREADER_AI_PREVIEW_FALLBACK=false` to disable it. HTML pages that still have no readable content after AI conversion can call Browser Run's `markdown` Quick Action; Workers Free includes 10 browser-minutes/day and paces Quick Actions to one request per 10 seconds. Browser Run runs only on a cache miss, uses no browser sessions (so the concurrent-browser limit does not apply), and its result is cached for 30 minutes; set `FEEDREADER_BROWSER_PREVIEW_FALLBACK=false` to disable it. Browser-facing preview and `/api/items` responses are `private, no-store`; their edge copies live in the Cache API. Watch the CPU-time panel if preview usage is heavy — bursts can hit the Workers Free 10 ms limit, so see [RUNBOOK.md](RUNBOOK.md#article-preview-failures). Richer preview blocks (inline links, captions, tables) increase the cached response payload but not the upstream fetch count; the same byte cap and cache TTL apply.
- Article previews add one outbound fetch (and one `node-html-parser` parse) per uncached `/api/preview` request. The 400 KB HTML body cap, 30-minute edge cache (5 minutes for empty results), and a per-IP limiter (20 cache-miss previews per minute; cached previews bypass it) bound that cost; transient upstream `429`/`5xx` responses are retried once. PDFs/Office documents and HTML pages with no readable blocks can additionally call Workers AI's `toMarkdown` (5 MB download cap; document text conversion is free and the fallback disables AI image-description conversion). That call runs only on a cache miss and its result is cached for the same 30 minutes; set `FEEDREADER_AI_PREVIEW_FALLBACK=false` to disable it. HTML pages that still have no readable content after AI conversion can call Browser Run's `markdown` Quick Action; Workers Free includes 10 browser-minutes/day and paces Quick Actions to one request per 10 seconds. Browser Run runs only on a cache miss, uses no browser sessions (so the concurrent-browser limit does not apply), and its result is cached for 30 minutes; set `FEEDREADER_BROWSER_PREVIEW_FALLBACK=false` to disable it. Browser-facing preview and `/api/items` responses are `private, no-store`; their edge copies live in the Cache API. Watch the CPU-time panel if preview usage is heavy — bursts can hit the Workers Free 10 ms limit, so see [RUNBOOK.md](RUNBOOK.md#article-preview-failures). Richer preview blocks (inline links, captions, tables) increase the cached response payload but not the upstream fetch count; the same byte cap and cache TTL apply.

Check the Cloudflare dashboard's D1 usage and Workers CPU-time panels periodically — see [RUNBOOK.md](RUNBOOK.md) for what to do if a source consistently nears the CPU ceiling.
3 changes: 2 additions & 1 deletion docs/RUNBOOK.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,7 @@ Use Cloudflare dashboard → Workers & Pages → `feedreader` → Triggers → C
- `422 invalid_url` — the stored URL is not public http(s) (for example an upstream feed emitted a `javascript:` or localhost link)
- `502 blocked` — the upstream returned `401`/`403` (for example DataDome/paywall protection)
- `502 rate_limited` / `502 upstream_error` — the upstream returned `429`/`5xx` after one retry
- `429 rate_limited` — the app's per-IP limiter blocked a cache-miss preview (20 per minute, `Retry-After: 60`). Upstream 429s after retries surface as `502 rate_limited`; the `preview_rate_limited` event identifies the local limiter.
- `504 timeout` — the fetch exceeded the 10-second timeout
- `502 too_large` — the HTML exceeded the 400 KB cap before readable content, or a document exceeded the 5 MB fallback cap
- `502 unsupported_content_type` / `empty_body` / `redirect` / `fetch_failed` — other fetch or extraction failures. `unsupported_content_type` also covers a document the AI fallback could not convert (disabled, timed out, or unsupported by `toMarkdown`).
Expand All @@ -90,7 +91,7 @@ Before returning `unsupported_content_type` for a non-HTML response, `too_large`

If the AI conversion returns nothing for an HTML page, the chain calls Browser Run's `markdown` Quick Action (`platforms/cloudflare/src/browserPreviewFallback.ts`) before returning `no_content`. It is limited to `no_content`; PDFs/Office documents, `too_large`, `blocked`, `timeout`, and rate-limited failures never reach it. Browser Run waits for `networkidle2` with 8-second navigation/action and 9-second outer timeouts, blocks images/media/fonts and literal private-host requests, and validates the rendered page status and final URL. Workers Free includes 10 browser-minutes/day and paces Quick Actions to one request per 10 seconds; a `429` is logged and treated as no preview. Set `FEEDREADER_BROWSER_PREVIEW_FALLBACK=false` to disable it. Browser Run successes are cached for 30 minutes; the preview cache key is `preview:v11`.

The Worker logs `preview_unavailable` and `preview_empty` events with the URL, reason, and upstream status, plus a `preview_ai_fallback` event for each AI attempt (`kind`, `outcome` = `ok`/`empty`/`error`, `ms`, `blocks`, `tokens`, and a capped `error`) and a `preview_browser_fallback` event for each Browser Run attempt (`outcome`, `status`, `page_status`, `browser_ms`, `blocks`, and a capped `error`). Successful previews, including AI- and Browser Run-converted ones, are edge-cached for 30 minutes via the Cache API; browser-facing responses are `private, no-store`. Transient upstream `429`/`5xx` responses are retried once with jitter. If bursts produce transient `503`s, check Workers Logs/CPU: the extractor runs in the request invocation, and sequential retries succeeding usually points at the Workers Free 10 ms CPU limit rather than a lasting upstream block. Lowering `DEFAULT_MAX_BYTES` in `core/preview.ts` (400 KB today) reduces the parse cost.
The Worker logs `preview_unavailable` and `preview_empty` events with the URL, reason, and upstream status; `preview_empty` also records `cache_ttl_seconds`. It logs a `preview_ai_fallback` event for each AI attempt (`kind`, `outcome` = `ok`/`empty`/`error`, `ms`, `blocks`, `tokens`, and a capped `error`), a `preview_browser_fallback` event for each Browser Run attempt (`outcome`, `status`, `page_status`, `browser_ms`, `blocks`, and a capped `error`), and `preview_rate_limited` when the local per-IP limiter rejects a cache miss. Successful previews, including AI- and Browser Run-converted ones, are edge-cached for 30 minutes; empty `no_content` results use a 5-minute TTL so a transient fallback failure can retry. Cache-miss previews pass a per-IP fixed-window limiter (Cache API, 20 per minute, per-colo, fail-open); cache hits bypass it. Because the in-Worker limiter is per-colo and fail-open, add a zone-level Cloudflare Rate Limiting rule for `/api/preview` if you need a global limit. Browser-facing responses are `private, no-store`. Transient upstream `429`/`5xx` responses are retried once with jitter. If bursts produce transient `503`s, check Workers Logs/CPU: the extractor runs in the request invocation, and sequential retries succeeding usually points at the Workers Free 10 ms CPU limit rather than a lasting upstream block. Lowering `DEFAULT_MAX_BYTES` in `core/preview.ts` (400 KB today) reduces the parse cost.

## Local SSR smoke procedure

Expand Down
Loading
Loading