Skip to content

fix: stop hiding arXiv API failures in paper ref backfill - #103

Merged
Jammy2211 merged 3 commits into
mainfrom
fix/arxiv-ref-transport-errors
Sep 30, 2026
Merged

Jammy2211 merged 3 commits into
mainfrom
fix/arxiv-ref-transport-errors

Conversation

@Jammy2211

Copy link
Copy Markdown
Collaborator

Problem

scripts/arxiv_refs.py currently catches every lookup exception and returns None. Downstream, None means a genuine empty/ambiguous arXiv result, so 406/429/DNS failures are logged as ordinary "no unique match" outcomes and the nightly Action stays green.

That is exactly what happened during the same arXiv outage that stopped the upstream paper feeds: PyAutoMemory's ref-backfill workflow looked healthy while resolving nothing.

Fix

  • Add the same 406/429/5xx retry shape used by the upstream digest.
  • If stdlib urllib receives the known Fastly 406, retry the request immediately through curl before spending the backoff ladder.
  • Introduce ArxivAPIError; None now only means arXiv answered and there was no unique exact-title match.
  • Track api_failed separately in the backfill.
  • Stop the run after an exhausted transport failure instead of hammering the same dead endpoint for every queue item.
  • Never NOTE-mark a line when its lookup failed because of transport.
  • Exit non-zero on an exhausted API failure so the scheduled workflow goes red.

Tests

  • injected dead API raises an explicit transport error
  • urllib 406 uses the curl fallback
  • transport failure is not counted/marked as a no-match
  • CLI exits non-zero when the API failed

This is the PyAutoMemory-side companion to PyAutoMind #432; it is not the cause of the stale daily inbox stamp, but it removes the silent failure mode exposed by the same incident.

@Jammy2211
Jammy2211 merged commit ae7dcfe into main Sep 30, 2026
1 check passed
@Jammy2211
Jammy2211 deleted the fix/arxiv-ref-transport-errors branch September 30, 2026 08:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant