Skip to content

queue: reconcile LaTeX with rendered maths, and repair damaged lines - #63

Merged
Jammy2211 merged 1 commit into
mainfrom
claude/pyautomemory-queue-cleanup
Aug 25, 2026
Merged

Jammy2211 merged 1 commit into
mainfrom
claude/pyautomemory-queue-cleanup

Conversation

@Jammy2211

Copy link
Copy Markdown
Collaborator

The two follow-ups from #62's remaining 90 search-only lines.

1. The matcher could not see past maths

arXiv's metadata carries the LaTeX source; a title pasted off the abstract
page carries the rendered glyphs. A plain punctuation strip never brought the
two together:

arXiv queue before after
$0.5\leq z<1.0$ 0.5≤z<1.0 05leqz10 ≠ 05z10 both 05z10
$\Lambda$CDM ΛCDM lambdacdm ≠ cdm both cdm
$M_\star$ M⋆ mstar ≠ m both m
$2.2\times10^{-21}$eV 2.2×10−21eV 22times1021ev ≠ 221021ev both 221021ev

normalise_title now erases both spellings rather than translating one into
the other: LaTeX control sequences go, and so does every non-ASCII character left
after accent folding. Erasing beats a \leq→≤ table, which would need every
symbol astronomers use and would silently mismatch the day it missed one — and
the uniqueness rule in match_entries is what keeps the lossier reduction safe.

Accented Latin still survives, because NFKD leaves an ASCII base: Ekström →
ekstrom, Muñoz-Darias → munozdarias. Greek has none, and is erased on both
sides alike.

2. Five lines were damaged, not unmatched

No matcher could have rescued these:

  • Chandra Observations of Spikey: … Supermassive Black Hole + Binary System —
    one title that lost its wrap, rejoined
  • …the local MBH-M⋆ relationProj C — junk welded on by whatever pasted it
  • __SMBH Binaries_ — a sub-heading with one trailing underscore, so it read as
    a paper
  • Massive Black Holes summary: https://arxiv.org/pdf/… — a ref written after a
    colon instead of —, so it was never recognised. Now a paper with a working
    PDF button, keeping the human's own label as its title
  • Two paragraph fragments pasted out of a paper's introduction (13 lines) —
    NOTE-marked by hand rather than spending 13 arXiv lookups discovering they
    are prose

Effect

#61 #62 here
Papers waiting 232 223 208
With a 📄 PDF button 127 133 134
Search-only 105 90 74

The 208 is the honest number: the difference was never papers. The matcher
improvement is not yet reflected — it needs a backfill run against arXiv, which
I'll dispatch once this is on main.

5 new tests; 116 pass, make validate clean.


Generated by Claude Code

Two cleanups on the backfill's remaining misses.

**The matcher could not see past maths.** arXiv's metadata carries the LaTeX
source, a title pasted off the abstract page carries the rendered glyphs, and a
plain punctuation strip does not bring the two together: `$0.5\leq z<1.0$` and
`0.5≤z<1.0` reduced to different strings, as did `$\Lambda$CDM`/`ΛCDM`,
`$M_\star$`/`M⋆`, `$z\sim3$`/`z∼3`, `$2.2\times10^{-21}$`/`2.2×10−21`.

normalise_title now erases both spellings rather than translating one into the
other: LaTeX control sequences go, and so does every non-ASCII character left
after accent folding. `\leq` and `≤` both become nothing and the two sides meet
at `05z10`. Erasing beats a `\leq`→`≤` table, which would need every symbol
astronomers use and would silently mismatch the day it missed one; the
uniqueness rule in match_entries is what keeps the lossier reduction safe.
Accented Latin still survives — NFKD leaves an ASCII base (`Ekström` →
`ekstrom`), Greek has none.

**Five lines were damaged rather than unmatched**, and no matcher could have
helped:

* `Chandra Observations of Spikey: … Supermassive Black Hole` / `Binary System`
  — one title that lost its wrap, rejoined.
* `…the local MBH-M⋆ relationProj  C` — junk welded onto the end by whatever
  pasted it, trimmed.
* `__SMBH Binaries_` — a sub-heading with one trailing underscore, so it read as
  a paper.
* `Massive Black Holes summary: https://arxiv.org/pdf/…` — a ref written after a
  colon instead of ` — `, so it was never recognised. Now a paper with a working
  PDF button, keeping the human's own label as its title.
* Two paragraph fragments pasted out of a paper's introduction (13 lines) —
  NOTE-marked by hand rather than spending 13 arXiv lookups discovering they are
  prose.

Live queue: 232 → 208 papers waiting (the rest were never papers), PDF coverage
133 → 134 before the matcher improvement has even run against arXiv.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PjWdVFPn2WDzcqpMXTMp7G
@Jammy2211
Jammy2211 merged commit 09f858f into main Aug 25, 2026
1 check passed
@Jammy2211
Jammy2211 deleted the claude/pyautomemory-queue-cleanup branch August 26, 2026 20:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants