Repository navigation
queue: reconcile LaTeX with rendered maths, and repair damaged lines - #63
Merged
Merged
Conversation
Two cleanups on the backfill's remaining misses.
**The matcher could not see past maths.** arXiv's metadata carries the LaTeX
source, a title pasted off the abstract page carries the rendered glyphs, and a
plain punctuation strip does not bring the two together: `$0.5\leq z<1.0$` and
`0.5≤z<1.0` reduced to different strings, as did `$\Lambda$CDM`/`ΛCDM`,
`$M_\star$`/`M⋆`, `$z\sim3$`/`z∼3`, `$2.2\times10^{-21}$`/`2.2×10−21`.
normalise_title now erases both spellings rather than translating one into the
other: LaTeX control sequences go, and so does every non-ASCII character left
after accent folding. `\leq` and `≤` both become nothing and the two sides meet
at `05z10`. Erasing beats a `\leq`→`≤` table, which would need every symbol
astronomers use and would silently mismatch the day it missed one; the
uniqueness rule in match_entries is what keeps the lossier reduction safe.
Accented Latin still survives — NFKD leaves an ASCII base (`Ekström` →
`ekstrom`), Greek has none.
**Five lines were damaged rather than unmatched**, and no matcher could have
helped:
* `Chandra Observations of Spikey: … Supermassive Black Hole` / `Binary System`
— one title that lost its wrap, rejoined.
* `…the local MBH-M⋆ relationProj C` — junk welded onto the end by whatever
pasted it, trimmed.
* `__SMBH Binaries_` — a sub-heading with one trailing underscore, so it read as
a paper.
* `Massive Black Holes summary: https://arxiv.org/pdf/…` — a ref written after a
colon instead of ` — `, so it was never recognised. Now a paper with a working
PDF button, keeping the human's own label as its title.
* Two paragraph fragments pasted out of a paper's introduction (13 lines) —
NOTE-marked by hand rather than spending 13 arXiv lookups discovering they are
prose.
Live queue: 232 → 208 papers waiting (the rest were never papers), PDF coverage
133 → 134 before the matcher improvement has even run against arXiv.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PjWdVFPn2WDzcqpMXTMp7G
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The two follow-ups from #62's remaining 90 search-only lines.
1. The matcher could not see past maths
arXiv's metadata carries the LaTeX source; a title pasted off the abstract
page carries the rendered glyphs. A plain punctuation strip never brought the
two together:
$0.5\leq z<1.0$0.5≤z<1.005leqz10≠05z1005z10$\Lambda$CDMΛCDMlambdacdm≠cdmcdm$M_\star$M⋆mstar≠mm$2.2\times10^{-21}$eV2.2×10−21eV22times1021ev≠221021ev221021evnormalise_titlenow erases both spellings rather than translating one intothe other: LaTeX control sequences go, and so does every non-ASCII character left
after accent folding. Erasing beats a
\leq→≤table, which would need everysymbol astronomers use and would silently mismatch the day it missed one — and
the uniqueness rule in
match_entriesis what keeps the lossier reduction safe.Accented Latin still survives, because NFKD leaves an ASCII base:
Ekström→ekstrom,Muñoz-Darias→munozdarias. Greek has none, and is erased on bothsides alike.
2. Five lines were damaged, not unmatched
No matcher could have rescued these:
Chandra Observations of Spikey: … Supermassive Black Hole+Binary System—one title that lost its wrap, rejoined
…the local MBH-M⋆ relationProj C— junk welded on by whatever pasted it__SMBH Binaries_— a sub-heading with one trailing underscore, so it read asa paper
Massive Black Holes summary: https://arxiv.org/pdf/…— a ref written after acolon instead of
—, so it was never recognised. Now a paper with a workingPDF button, keeping the human's own label as its title
NOTE-marked by hand rather than spending 13 arXiv lookups discovering theyare prose
Effect
The 208 is the honest number: the difference was never papers. The matcher
improvement is not yet reflected — it needs a backfill run against arXiv, which
I'll dispatch once this is on
main.5 new tests; 116 pass,
make validateclean.Generated by Claude Code