Repository navigation
arxiv: search on the words, not on the mangled maths - #64
Merged
Merged
Conversation
#63 taught normalise_title to reconcile `$0.5\leq z<1.0$` with `0.5≤z<1.0`, and the dry run that followed resolved 19 of 74 — none of them the maths titles it was written for. The fix was aimed at the wrong half of the problem: comparison was improved when retrieval was what failed. resolve_title built its phrase by replacing every non-word character with a space, so a title carrying maths went out flattened word-by-word: ti:"Photometric properties of … galaxies at 0 5 z 1 0" ti:"Dwarf galaxies imply dark matter is heavier than 2 2 10 21eV" ti:"Review of solutions to the Cusp core problem of the ΛCDM Model" arXiv indexes the LaTeX source and matches none of that, so nothing came back — and a better comparison has nothing to compare. (`Λ` survived into the query verbatim because Python's `\w` counts Greek as a word character.) query_phrase now drops the maths rather than mangling it, searching on the longest run of ordinary words the title contains — the part both sides spell the same way. `…galaxies at 0.5≤z<1.0` goes out as `ti:"…galaxies at"`. That is a broader search, not a looser match. The phrase only decides which candidates come back; every one still has to survive the exact, unique-title check in match_entries, and an ambiguous set still resolves to nothing. A test pins that distinction directly: a candidate sharing the searched phrase but not the full title is rejected. max_results rises 20 → 50 to suit the wider net, which costs nothing — the choice is made locally. Also on the workflow, so the next pass is usable and watchable: * `mark_unresolved` dispatch input, wired to --mark-unresolved. Only reachable from a deliberate dispatch: `inputs.*` is empty on a cron fire, so the nightly can never mark on its own. * `python -u`, so the log streams during the run instead of arriving in one block at the end. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PjWdVFPn2WDzcqpMXTMp7G
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The fix in #63 was aimed at the wrong half
#63 taught
normalise_titleto reconcile$0.5\leq z<1.0$with0.5≤z<1.0.The dry run that followed resolved 19 of 74 — none of them the maths titles it
was written for. Comparison was improved when retrieval was what failed.
resolve_titlebuilt its phrase by replacing every non-word character with aspace, so a title carrying maths went out flattened word-by-word:
arXiv indexes the LaTeX source and matches none of that, so no candidates came
back — and a better comparison has nothing to compare. (
Λsurvived into thequery verbatim, because Python's
\wcounts Greek as a word character.)The change
query_phrasedrops the maths rather than mangling it, searching on the longestrun of ordinary words the title contains — the part both sides spell the same
way:
…galaxies at 0.5≤z<1.0ti:"…galaxies at"…heavier than 2.2×10−21eVti:"Dwarf galaxies imply dark matter is heavier than"…problem of the ΛCDM Modelti:"…problem of the CDM Model"This is a broader search, not a looser match. The phrase only decides which
candidates come back; every one still has to survive the exact, unique-title
check in
match_entries, and an ambiguous set still resolves to nothing. A testpins that distinction directly — a candidate sharing the searched phrase but not
the full title is rejected.
max_resultsrises 20 → 50 to suit the wider net,which costs nothing since the choice is made locally.
Workflow, so the next pass is usable and watchable
mark_unresolveddispatch input, wired to--mark-unresolved(the markeradded in queue: line kinds — a heading is not a paper you can download #62). Only reachable from a deliberate dispatch:
inputs.*is emptyon a cron fire, so the nightly can never mark on its own.
python -u— the log streams during the run instead of arriving in oneblock at the end. This is why every run so far has been unreadable while in
flight.
5 new tests; 121 pass,
make validateclean.Generated by Claude Code