Skip to content

arxiv: search on the words, not on the mangled maths - #64

Merged
Jammy2211 merged 1 commit into
mainfrom
claude/pyautomemory-arxiv-query
Aug 25, 2026
Merged

Jammy2211 merged 1 commit into
mainfrom
claude/pyautomemory-arxiv-query

Conversation

@Jammy2211

Copy link
Copy Markdown
Collaborator

The fix in #63 was aimed at the wrong half

#63 taught normalise_title to reconcile $0.5\leq z<1.0$ with 0.5≤z<1.0.
The dry run that followed resolved 19 of 74 — none of them the maths titles it
was written for
. Comparison was improved when retrieval was what failed.

resolve_title built its phrase by replacing every non-word character with a
space, so a title carrying maths went out flattened word-by-word:

ti:"Photometric properties of … galaxies at 0 5 z 1 0"
ti:"Dwarf galaxies imply dark matter is heavier than 2 2 10 21eV"
ti:"Review of solutions to the Cusp core problem of the ΛCDM Model"

arXiv indexes the LaTeX source and matches none of that, so no candidates came
back — and a better comparison has nothing to compare. (Λ survived into the
query verbatim, because Python's \w counts Greek as a word character.)

The change

query_phrase drops the maths rather than mangling it, searching on the longest
run of ordinary words the title contains — the part both sides spell the same
way:

title query sent
…galaxies at 0.5≤z<1.0 ti:"…galaxies at"
…heavier than 2.2×10−21eV ti:"Dwarf galaxies imply dark matter is heavier than"
…problem of the ΛCDM Model ti:"…problem of the CDM Model"

This is a broader search, not a looser match. The phrase only decides which
candidates come back; every one still has to survive the exact, unique-title
check in match_entries, and an ambiguous set still resolves to nothing. A test
pins that distinction directly — a candidate sharing the searched phrase but not
the full title is rejected. max_results rises 20 → 50 to suit the wider net,
which costs nothing since the choice is made locally.

Workflow, so the next pass is usable and watchable

  • mark_unresolved dispatch input, wired to --mark-unresolved (the marker
    added in queue: line kinds — a heading is not a paper you can download #62). Only reachable from a deliberate dispatch: inputs.* is empty
    on a cron fire, so the nightly can never mark on its own.
  • python -u — the log streams during the run instead of arriving in one
    block at the end. This is why every run so far has been unreadable while in
    flight.

5 new tests; 121 pass, make validate clean.


Generated by Claude Code

#63 taught normalise_title to reconcile `$0.5\leq z<1.0$` with `0.5≤z<1.0`, and
the dry run that followed resolved 19 of 74 — none of them the maths titles it
was written for. The fix was aimed at the wrong half of the problem: comparison
was improved when retrieval was what failed.

resolve_title built its phrase by replacing every non-word character with a
space, so a title carrying maths went out flattened word-by-word:

    ti:"Photometric properties of … galaxies at 0 5 z 1 0"
    ti:"Dwarf galaxies imply dark matter is heavier than 2 2 10 21eV"
    ti:"Review of solutions to the Cusp core problem of the ΛCDM Model"

arXiv indexes the LaTeX source and matches none of that, so nothing came back —
and a better comparison has nothing to compare. (`Λ` survived into the query
verbatim because Python's `\w` counts Greek as a word character.)

query_phrase now drops the maths rather than mangling it, searching on the
longest run of ordinary words the title contains — the part both sides spell the
same way. `…galaxies at 0.5≤z<1.0` goes out as `ti:"…galaxies at"`.

That is a broader search, not a looser match. The phrase only decides which
candidates come back; every one still has to survive the exact, unique-title
check in match_entries, and an ambiguous set still resolves to nothing. A test
pins that distinction directly: a candidate sharing the searched phrase but not
the full title is rejected. max_results rises 20 → 50 to suit the wider net,
which costs nothing — the choice is made locally.

Also on the workflow, so the next pass is usable and watchable:

* `mark_unresolved` dispatch input, wired to --mark-unresolved. Only reachable
  from a deliberate dispatch: `inputs.*` is empty on a cron fire, so the nightly
  can never mark on its own.
* `python -u`, so the log streams during the run instead of arriving in one
  block at the end.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PjWdVFPn2WDzcqpMXTMp7G
@Jammy2211
Jammy2211 merged commit 1328476 into main Aug 25, 2026
1 check passed
@Jammy2211
Jammy2211 deleted the claude/pyautomemory-arxiv-query branch August 26, 2026 20:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants