Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ npx lighthouse https://reader.boringcode.dev --view
- **Scheduled refresh** every 1 hour on wall-clock boundaries
- **Manual refresh** from the header re-fetches the current feed view from backend stored items only; it does **not** re-fetch upstream sources
- **Persisted visited-link dimming** for feed card titles across reload/reopen using local storage
- **Optional article preview** (off by default, toggled in Reader settings) that opens a reader view beside the feed on wide screens or full-screen on phones, with inline links/emphasis, image captions, tables, publication date/reading time, source-specific Hacker News comments, and an **Open source** button
- **Optional article preview** (off by default, toggled in Reader settings) that opens a reader view beside the feed on wide screens or full-screen on phones, with inline links/emphasis, image captions, tables, publication date/reading time, a Workers AI Markdown fallback for PDFs, Office documents, and HTML pages with no readable blocks, source-specific Hacker News comments, and an **Open source** button
- **Saved articles** with an optional per-card bookmark toggle, a `Saved` filter, and a preview-panel toggle; bookmark controls are disabled by default and can be enabled in Reader settings, with saved items stored locally under `feedreader.bookmarks`
- **PWA-ready assets and offline caching** including manifest, service worker, touch icons, cached shell assets, cached `/api/items` responses for previously visited views, a standalone launch veil, and skeleton loading states
- **Reconnect list refresh** re-fetches the current view from backend stored items only; it does **not** refresh upstream sources
Expand Down Expand Up @@ -220,6 +220,7 @@ Runtime configuration:
| ----------------------------- | ----------------: | -------------------------------------------------------------- |
| `FEEDREADER_ITEMS_PER_SOURCE` | `20` | Per-source item count used in source dashboard/health contexts |
| `FEEDREADER_USER_AGENT` | `feedreader/0.1` | Outbound fetch user agent |
| `FEEDREADER_AI_PREVIEW_FALLBACK` | `true` | Workers AI Markdown fallback for PDFs/Office documents and empty previews; `false` disables it |
| `REFRESH_SECRET` | _required secret_ | Shared secret for internal per-source refresh fan-out routes |

---
Expand Down Expand Up @@ -265,7 +266,7 @@ Returns a reader view for a stored feed item: rich text segments, image blocks w

- `url` — required; must exactly match the URL of an item currently stored in D1

The Worker fetches the article with a timeout and 400 KB cap, rejects non-HTML responses and non-public hosts, and never returns upstream markup. Inline targets are restricted to validated public http(s) URLs. Responses are edge-cached for 30 minutes. Returns `404` for unknown URLs and `502` when the upstream page cannot be previewed.
The Worker fetches the article with a 10-second timeout and a 400 KB HTML cap, rejects non-public hosts, and never returns upstream markup. When the built-in extractor finds no readable blocks or the response is a document (PDF/Office), the Worker can convert it through Workers AI's Markdown conversion with a 5 MB cap; set `FEEDREADER_AI_PREVIEW_FALLBACK=false` to disable that fallback. Inline targets and images are restricted to validated public http(s) URLs. Responses are edge-cached for 30 minutes. Returns `404` for unknown URLs and `502` when the upstream page cannot be previewed.

### `POST /api/refresh`

Expand Down
181 changes: 170 additions & 11 deletions core/preview.ts
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,10 @@ import {
} from "./sources/dom.ts";

const DEFAULT_MAX_BYTES = 400_000;
// Non-HTML bodies are handed to the optional platform fallback (for example
// Workers AI's toMarkdown) instead of being parsed here. Keep the cap well
// below the Worker's memory budget while still covering typical papers.
const DEFAULT_MAX_FALLBACK_BYTES = 5_000_000;
const DEFAULT_TIMEOUT_MS = 10_000;
const MAX_REDIRECTS = 5;
const REDIRECT_STATUSES = new Set([301, 302, 303, 307, 308]);
Expand Down Expand Up @@ -118,6 +122,33 @@ export interface ArticlePreview {
blocks: PreviewBlock[];
}

/** Content the primary extractor cannot turn into blocks. A platform adapter
* may convert it another way (for example Workers AI's Markdown conversion);
* core only passes bytes/text and receives already-validated blocks back. */
export type PreviewFallbackKind =
| "no_content"
| "too_large"
| "unsupported_content_type";

export interface PreviewFallbackInput {
kind: PreviewFallbackKind;
url: string;
contentType: string;
/** Present for HTML fallbacks. */
html?: string;
/** Present for non-HTML fallbacks, capped by `maxFallbackBytes`. */
bytes?: Uint8Array;
}

/**
* Optional platform hook for previews the built-in extractor can't produce.
* It must return plain `PreviewBlock` data (never upstream HTML) or null to
* preserve the original failure. Core never imports a platform SDK.
*/
export type PreviewFallback = (
input: PreviewFallbackInput,
) => Promise<ArticlePreview | null>;

export type PreviewFailureReason =
| "invalid_url"
| "fetch_failed"
Expand Down Expand Up @@ -170,6 +201,11 @@ export interface FetchArticlePreviewOptions {
retryDelayMs?: number;
/** Random jitter added to the retry delay. */
retryJitterMs?: number;
/** Attempted only when the built-in extractor finds no readable blocks or
* the response is not HTML. */
fallback?: PreviewFallback;
/** Cap for non-HTML response bodies handed to `fallback`. */
maxFallbackBytes?: number;
}

/**
Expand Down Expand Up @@ -218,7 +254,11 @@ export async function fetchArticlePreview(
try {
lastResult = await fetchWithSafeRedirects(fetchImpl, target, {
headers: {
Accept: "text/html, application/xhtml+xml",
// PDF/octet-stream let document URLs negotiate their document
// instead of an HTML wrapper; HTML stays the preferred type for
// ordinary article URLs.
Accept:
"text/html, application/xhtml+xml, application/pdf;q=0.9, application/octet-stream;q=0.5",
"User-Agent": options.userAgent,
},
signal: timeoutSignal,
Expand Down Expand Up @@ -272,14 +312,56 @@ export async function fetchArticlePreview(
}

const contentType = (response.headers.get("content-type") ?? "").toLowerCase();
if (
contentType !== "" &&
!contentType.includes("text/html") &&
!contentType.includes("application/xhtml+xml")
) {
const isHtml =
contentType === "" ||
contentType.includes("text/html") ||
contentType.includes("application/xhtml+xml");
const fallback = options.fallback;

if (!isHtml) {
const maxFallbackBytes =
options.maxFallbackBytes ?? DEFAULT_MAX_FALLBACK_BYTES;
const declaredLength = Number.parseInt(
response.headers.get("content-length") ?? "",
10,
);
// Do not download a document that already declares itself too large: the
// fallback would reject the partial file anyway.
const declaredTooLarge =
Number.isFinite(declaredLength) && declaredLength > maxFallbackBytes;
let tooLarge = false;
if (fallback) {
if (declaredTooLarge) {
tooLarge = true;
} else {
try {
const { bytes, truncated } = await readBytesLimited(
response,
maxFallbackBytes,
);
// A partial PDF/Office file is unlikely to convert; report the size
// limit rather than spending an AI call on corrupt input.
if (truncated) {
tooLarge = true;
} else {
const converted = await tryPreviewFallback(fallback, {
kind: "unsupported_content_type",
url: finalUrl,
contentType,
bytes,
});
if (converted) return converted;
}
} catch {
// The fallback is best-effort; report the original reason below.
}
}
}
throw new PreviewUnavailableError(
"article is not HTML",
"unsupported_content_type",
tooLarge
? "article document is too large to preview"
: "article is not HTML",
tooLarge ? "too_large" : "unsupported_content_type",
);
}

Expand All @@ -292,6 +374,15 @@ export async function fetchArticlePreview(
}

const preview = extractArticlePreview(html, finalUrl);
if (preview.blocks.length === 0 && fallback) {
const converted = await tryPreviewFallback(fallback, {
kind: truncated ? "too_large" : "no_content",
url: finalUrl,
contentType,
html,
});
if (converted) return converted;
}
if (truncated && preview.blocks.length === 0) {
throw new PreviewUnavailableError(
"article body exceeded the preview limit before readable content",
Expand Down Expand Up @@ -689,8 +780,7 @@ function extractImageBlock(
}
}
if (resolved === "") return null;
if (isAvatarImageSrc(resolved)) return null;
if (isDecorativeImageSrc(resolved)) return null;
if (isDecorativePreviewImageSrc(resolved)) return null;

const caption = normalizeText(
image.closest("figure")?.querySelector("figcaption")?.textContent,
Expand Down Expand Up @@ -746,6 +836,15 @@ function resolveImageUrl(
}
}

/**
* True for interface chrome that should never become a reader-view figure.
* Shared with the Markdown fallback mapper so both extractors apply the same
* avatar/wordmark/badge policy.
*/
export function isDecorativePreviewImageSrc(rawUrl: string): boolean {
return isAvatarImageSrc(rawUrl) || isDecorativeImageSrc(rawUrl);
}

/**
* Avatar and profile images are interface chrome, not article figures. They
* often appear before the content and would otherwise consume the image cap
Expand Down Expand Up @@ -1058,7 +1157,7 @@ function extractPublishedAt(document: HtmlDocument): string | undefined {
return undefined;
}

function estimateReadingMinutes(
export function estimateReadingMinutes(
blocks: PreviewBlock[],
): number | undefined {
let words = 0;
Expand Down Expand Up @@ -1199,6 +1298,18 @@ function isPrivateIpv6(hostname: string): boolean {
return firstGroup.startsWith("ff");
}

async function tryPreviewFallback(
fallback: PreviewFallback,
input: PreviewFallbackInput,
): Promise<ArticlePreview | null> {
try {
const preview = await fallback(input);
return preview && preview.blocks.length > 0 ? preview : null;
} catch {
return null;
}
}

/**
* Follows redirects manually so every hop's Location is checked by
* normalizePreviewUrl before the request is issued. `redirect: "follow"`
Expand Down Expand Up @@ -1298,3 +1409,51 @@ async function readBodyLimited(
}
return { text: out, truncated };
}

async function readBytesLimited(
response: Response,
maxBytes: number,
): Promise<{ bytes: Uint8Array; truncated: boolean }> {
if (!response.body) {
const bytes = new Uint8Array(await response.arrayBuffer());
return {
bytes: bytes.slice(0, maxBytes),
truncated: bytes.byteLength > maxBytes,
};
}

const reader = response.body.getReader();
const chunks: Uint8Array[] = [];
let bytesRead = 0;
let truncated = false;
try {
while (bytesRead < maxBytes) {
const { done, value } = await reader.read();
if (done) break;
if (!value || value.byteLength === 0) continue;
const remaining = maxBytes - bytesRead;
if (value.byteLength > remaining) {
chunks.push(value.subarray(0, remaining));
bytesRead += remaining;
truncated = true;
break;
}
chunks.push(value);
bytesRead += value.byteLength;
}
if (!truncated && bytesRead >= maxBytes) {
const extra = await reader.read();
truncated = !extra.done;
}
} finally {
await reader.cancel().catch(() => {});
}

const bytes = new Uint8Array(bytesRead);
let offset = 0;
for (const chunk of chunks) {
bytes.set(chunk, offset);
offset += chunk.byteLength;
}
return { bytes, truncated };
}
Loading
Loading