fix(topics): label a topic by its most-used spelling, not min() - #162
Merged
Conversation
/topics was full of labels like `! news`, `["en"]`, `_posts` and `/life`. None of them were extraction bugs: `topicSlug` had normalised every one of them away correctly, and the grouping, URLs and counts were right the whole time. Only the display name was wrong. Five queries picked that name independently with `min(keyword)` — the rollup, the topic page, both alert queries and the followed-topics list. `min()` is a lexicographic minimum, and in ASCII `!` `"` `*` `.` `/` `[` and `_` all sort before lowercase letters, so a single publisher's malformed <category> tag won the label for everybody. Measured on prod: slug `news` showed as `! news` — 1 feed, against 10,310 spelling it `news` slug `en` showed as `["en"]` — 1 feed, against 13,130 slug `post` showed as `_posts` — 1 feed, against 11,774 slug `life` showed as `/life` — 4 feeds, against 8,985 12,318 of 152,814 topics carry such a label, concentrated at the top of the directory by feed count. The rule now lives once, in topicLabel.js, and ranks by count(distinct feed_id): the directory's own usage decides, which needs no stoplist and cannot be captured by one feed. Tie-breaks are shortest-first (so `ai` beats `ai,` and `ai:`) then lexicographic for determinism. The old comment on topicBySlug is why this stayed invisible — it reasoned that "any spelling will do … the rows under one slug differ only in ways the slug already erased". The slug strips punctuation and the keyword keeps it, so the surviving differences are exactly the ugly ones. Not addressed here: topics that are junk in themselves rather than mislabelled — `#39` and `rsquo` from undecoded HTML entities, and non-English grammar words from an English-only stoplist. Those are extraction-side and need a re-crawl, not a rollup. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TGaU2kAfCZSNxH6pvW5ZZQ
ralyodio
marked this pull request as ready for review
August 29, 2026 15:32
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
/topicswas full of labels like! news,["en"],_posts,/lifeand@work.None of these are extraction bugs.
topicSlughad already normalised every one of them away — the grouping, the URLs and the counts were correct the whole time. Only the display name was wrong.Cause
Five queries picked that name independently with
min(keyword): the rollup (refreshTopics), the topic page (topicBySlug), both alert queries, and the followed-topics list.min()is a lexicographic minimum, and in ASCII!(0x21)"(0x22)*(0x2A).(0x2E)/(0x2F)[(0x5B)_(0x5F) all sort before lowercase letters. So whenever one publisher wrote a malformed<category>tag, their spelling won the label for everyone.Measured on prod:
news! newsnewsen["en"]enai"ai"aipost_postspostlife/lifelife12,318 of 152,814 topics carry such a label, concentrated at the top of the directory by feed count.
Fix
The rule now lives once, in
packages/db/src/topicLabel.js, and ranks bycount(distinct feed_id)— the directory's own usage decides. That needs no stoplist and cannot be captured by a single feed. Tie-breaks are shortest-first (soaibeatsai,andai:at equal counts) then lexicographic for determinism.The old comment on
topicBySlugis why this stayed invisible: it reasoned that "any spelling will do … the rows under one slug differ only in ways the slug already erased". The slug strips punctuation and the keyword keeps it, so the surviving differences are exactly the ugly ones.Effect
topicBySlug, the alert queries and the followed-topics list correct themselves on deploy./topicsfollows within the hour, on the poller's existing hourlyrefreshTopics. No migration.Verified
packages/db/test/topic-label.test.js(fails onmin(), passes here)pnpm -r test: 1,305 tests across 10 packages, 0 failuresна) are left aloneNot addressed
Topics that are junk in themselves rather than mislabelled —
#39andrsquofrom undecoded HTML entities, and non-English grammar words (de,la,que) from an English-only stoplist. Those are extraction-side and need a re-crawl rather than a rollup, so they belong in their own change.🤖 Generated with Claude Code
https://claude.ai/code/session_01TGaU2kAfCZSNxH6pvW5ZZQ