Skip to content

Make the corpus figures cheap enough to ever appear - #165

Merged
ralyodio merged 1 commit into
mainfrom
fix-corpus-figures
Aug 29, 2026
Merged

Make the corpus figures cheap enough to ever appear#165
ralyodio merged 1 commit into
mainfrom
fix-corpus-figures

Conversation

@ralyodio

Copy link
Copy Markdown
Contributor

Found by looking at the live page after #163 deployed: /sales was showing three of its five figures. Full-text articles and Authors were missing, and would have stayed missing.

What happened

articleFigures sampled 20,000 rows to compute an average article length. text_length is covered by no index, so the cost is one row lookup per sampled row and nothing else. Measured against production, 2026-08-29:

query time result
count(*) where status='ok' 1,946ms 274,885
avg over 20,000 22,662ms 7,810
avg over 2,000 584ms 7,929
count(*) from authors 695ms 195,456

Twenty-three seconds against a caller timeout of eight. The read failed — and remember stores nothing on failure, which is exactly the trap cache.js was written to describe:

a "compute on miss, store on success" cache stores nothing, and every single request pays 30 seconds forever

So it does not fail once and recover. It fails for ever. The page rendered its graceful fallback (rows omitted rather than showing zeroes) permanently, which is why nothing looked broken — the fallback worked exactly as designed.

The fix

Sample 2,000 instead of 20,000. Ten times the sample bought a difference of 1.5% on a figure already labelled "sampled" on the page, against 39 times the cost and a page that could not show it at all.

Timeout also raised from 8s to 15s against ~3s of measured work. A timeout under the cost of the work it guards is not a safety margin, it is an off switch with a delay on it. Paid only on a genuinely cold cache; one success by anyone serves every reader for an hour.

If a defensible exact total is ever wanted, the answer is a sum(text_length) warmed by the poller on its patient connection — not a bigger sample on the request path.

Test

Bounds the sample (<= 5_000) rather than pinning it. The number is a judgement call and may reasonably move; the order of magnitude is not. Raising it past the bound does not make the page slow, it makes it empty — so the assertion message points at the measurement.

Full suite green on Node 22, pnpm build clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Pij2tFcRgqceMpheotFSoX

/sales shipped missing two of its five numbers. Full-text articles and
authors were absent from the live page, and would have stayed absent.

`articleFigures` sampled 20,000 rows for its average length. `text_length`
is covered by no index, so the cost is one row lookup per sampled row and
nothing else. Measured against production:

    count(*) where status='ok'      1,946ms   274,885
    avg over 20,000                22,662ms     7,810
    avg over 2,000                    584ms     7,929

Twenty-three seconds against a caller timeout of eight. The read failed,
and `remember` stores nothing on failure — which is precisely the trap
cache.js exists to describe. It does not fail once and recover; it fails
for ever, and the page renders its graceful fallback permanently. The
fallback worked exactly as designed, which is why nothing looked broken.

Ten times the sample bought 1.5% on a figure already labelled "sampled",
against 39 times the cost and a page that could not show it at all.

Also raises the timeout to 15s against ~3s of measured work. A timeout
under the cost of the work it guards is not a safety margin, it is an off
switch with a delay on it.

The test bounds the sample rather than pinning it: the number is a
judgement call, the order of magnitude is not, and raising it past the
bound does not make the page slow — it makes it empty.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pij2tFcRgqceMpheotFSoX
@ralyodio
ralyodio merged commit fd7b4d8 into main Aug 29, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant