From 0e9582da58734c2e44fdba5f85fc98f970c208fb Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 17 Sep 2026 20:01:26 +0000 Subject: [PATCH] memory: interests lapse, wikilink ratchet, seed split MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three changes from the 2026-09-17 ecosystem review (PyAutoMind docs/organism/ecosystem_review_2026-09-17.md): - arxiv-interests.md lapses like the inbox: `interests_actions.py sweep` drops whole day-batches older than the inbox window; the board shows the days left. Measured: 123 appended, 2 promoted, 114 sitting unread in 11 batches — a backlog nobody cleared is not knowledge. - scripts/validate_wikilinks.py in `make validate`: 2,108 links, 436 unresolved across 112 targets today, pinned in scripts/wikilink_baseline.txt; a NEW dangling target fails, the existing ones are a backlog `--list` prints. Cross-wiki links, the sources-/seed- aliases, anchors and code blocks follow wiki/CLAUDE.md. - The eight all-TODO import pages (1,747 lines, 100 unresolved entries) move to wiki//seed/, out of the indexes and out of the Brain's memory faculty, until a paper is verified and promoted into sources/. The citation validator now walks seed/ too; counts unchanged. - bibkey_aliases.yaml, retired in #34, is no longer documented as present. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_018AWgNfNwvJMi6s7MWBeVJ2 --- .github/workflows/validate.yml | 10 + AGENTS.md | 3 +- Makefile | 7 +- arxiv-interests.md | 28 ++- index.md | 22 +- scripts/board.py | 46 ++++- scripts/interests_actions.py | 83 +++++++- scripts/validate_literature_citations.py | 15 +- scripts/validate_wikilinks.py | 193 ++++++++++++++++++ scripts/wikilink_baseline.txt | 112 ++++++++++ tests/test_board.py | 14 ++ tests/test_interests_actions.py | 72 ++++++- tests/test_validate_literature_citations.py | 19 ++ tests/test_validate_wikilinks.py | 178 ++++++++++++++++ wiki/CLAUDE.md | 20 +- wiki/galaxies/CLAUDE.md | 3 +- wiki/galaxies/concepts/dm-halo-shapes.md | 4 +- wiki/galaxies/index.md | 12 +- wiki/galaxies/log.md | 19 ++ .../bulge-disk-decomposition.md | 0 .../dark-matter-geometry-galaxies.md | 0 .../galaxy-formation-misc.md | 2 +- .../halo-galaxy-connection.md | 2 +- .../{sources => seed}/stellar-halos.md | 2 +- .../galaxies/sources/light-profile-fitting.md | 2 +- wiki/lensing/index.md | 6 +- wiki/lensing/log.md | 19 ++ .../lensing/{sources => seed}/unclassified.md | 0 wiki/lensing/sources/reviews-and-summaries.md | 2 +- wiki/methods/CLAUDE.md | 3 +- wiki/methods/index.md | 7 +- wiki/methods/log.md | 19 ++ .../deep-learning-methods.md | 0 .../medical-imaging-adjacents.md | 0 .../sources/likelihood-free-inference.md | 2 +- 35 files changed, 855 insertions(+), 71 deletions(-) create mode 100644 scripts/validate_wikilinks.py create mode 100644 scripts/wikilink_baseline.txt create mode 100644 tests/test_validate_wikilinks.py rename wiki/galaxies/{sources => seed}/bulge-disk-decomposition.md (100%) rename wiki/galaxies/{sources => seed}/dark-matter-geometry-galaxies.md (100%) rename wiki/galaxies/{sources => seed}/galaxy-formation-misc.md (99%) rename wiki/galaxies/{sources => seed}/halo-galaxy-connection.md (99%) rename wiki/galaxies/{sources => seed}/stellar-halos.md (99%) rename wiki/lensing/{sources => seed}/unclassified.md (100%) rename wiki/methods/{sources => seed}/deep-learning-methods.md (100%) rename wiki/methods/{sources => seed}/medical-imaging-adjacents.md (100%) diff --git a/.github/workflows/validate.yml b/.github/workflows/validate.yml index 7cf7b86..65d2926 100644 --- a/.github/workflows/validate.yml +++ b/.github/workflows/validate.yml @@ -22,4 +22,14 @@ jobs: python-version: "3.12" - run: pip install pytest - run: make validate + # The wikilink leg of `make validate` is a ratchet: it fails only on an + # unresolved target that is not in scripts/wikilink_baseline.txt, because + # a link with no page yet is legal (wiki/CLAUDE.md, "Cross-references"). + # This step re-runs it with --list so the log always carries the current + # wanted-page backlog — and, when the ratchet trips, shows how often the + # offending slug is referenced. `if: always()` so a failure above still + # gets the listing. + - name: Wikilink report + if: always() + run: python scripts/validate_wikilinks.py --list - run: make test diff --git a/AGENTS.md b/AGENTS.md index cc0690b..9011774 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -31,8 +31,7 @@ The repo has exactly **two content homes**, enforced by [`wiki/CLAUDE.md`](wiki/CLAUDE.md). New sub-wikis are added beside the existing ones, never at the repo root. - **`bibliography/`** — the *only* place BibTeX lives. One canonical file - (`pyautomemory.bib`) plus `bibkey_aliases.yaml`; never add loose `.bib` - files anywhere else. + (`pyautomemory.bib`); never add loose `.bib` files anywhere else. **Source PDFs live off-repo.** Never commit a paper (PDF/HTML, with or without a file extension) — read it, stub it in the right diff --git a/Makefile b/Makefile index 9bc86ed..cd230e2 100644 --- a/Makefile +++ b/Makefile @@ -1,6 +1,6 @@ -.PHONY: validate validate-literature-citations validate-structure test board +.PHONY: validate validate-literature-citations validate-structure validate-wikilinks test board -validate: validate-literature-citations validate-structure +validate: validate-literature-citations validate-structure validate-wikilinks validate-literature-citations: python scripts/validate_literature_citations.py @@ -8,6 +8,9 @@ validate-literature-citations: validate-structure: python scripts/validate_structure.py +validate-wikilinks: + python scripts/validate_wikilinks.py + test: python -m pytest tests/ -q diff --git a/arxiv-interests.md b/arxiv-interests.md index ea3947a..7af9d4a 100644 --- a/arxiv-interests.md +++ b/arxiv-interests.md @@ -16,18 +16,28 @@ Format: into — these papers span domains, so unlike the strong-lensing inbox there is no single destination. A topic naming no real section falls back to `## Interests` (`FALLBACK_SECTION` in `scripts/interests_actions.py`). -- **A backlog, not a timer.** Nothing here lapses. The Dashboard shows the - **oldest un-cleared batch only**, and its 🧹 *clear* button drops that whole - day and reveals the next — so a fortnight away is a fortnight of batches to - cycle through, one tap each, not a fortnight of lost recommendations. This is - the one way this list differs from `arxiv-inbox.md`, which lapses after seven - days. +- **A backlog on the inbox's timer.** The Dashboard shows the **oldest + un-cleared batch only**, and its 🧹 *clear* button drops that whole day and + reveals the next — but a batch nobody clears **lapses after 7 days** + (`INBOX_WINDOW_DAYS` in `scripts/inbox_actions.py`, reused by + `interests_actions.py sweep`) and is swept by the nightly job. The batch is + the unit: the whole day goes, never part of one. + + This list was built the other way — "a backlog, not a timer; nothing here + lapses" — so that a fortnight away would be a fortnight of batches to cycle + through rather than a fortnight of lost recommendations. The measurement + said otherwise: of 123 papers appended, **2 reached the reading queue and 1 + batch was cleared** — a 1.6 % clearance rate, leaving 114 entries in 11 + day-batches unread. A backlog nobody walks forward through is not a backlog, + and it makes the oldest batch — the one the Dashboard shows — the least + relevant thing on the board. Seven days is how long a suggestion is worth + looking at; the same rule as `arxiv-inbox.md`, applied a day at a time. - Each paper carries the same one-tap actions as the inbox: 📄 straight to the PDF, ➕ add to the reading queue, 📥 intake into memory, 📑 make citeable, ✖️ dismiss. -- A cleared batch is not lost: git history holds it. Same reasoning as the - inbox sweep — the never-delete rule in `reading-queue.md` protects *reading - history*, and an un-acted suggestion is not history. +- A cleared or swept batch is not lost: git history holds it. Same reasoning as + the inbox sweep — the never-delete rule in `reading-queue.md` protects + *reading history*, and an un-acted suggestion is not history. - One `last digest: ` line, directly below the `---`, records the last run of the nightly digest — **papers or none** — so an empty list says *which* kind of empty it is: dated today it is a genuinely quiet day, four diff --git a/index.md b/index.md index 69bc6c4..73b0d4c 100644 --- a/index.md +++ b/index.md @@ -41,12 +41,15 @@ the schema in [`wiki/CLAUDE.md`](wiki/CLAUDE.md). that is **not** strong lensing: black holes, dark matter, galaxy formation, statistics. One line per paper as ` — [] [ — <ref>]`, where the topic names the reading-queue section ➕ files it into. It differs - from the inbox in one way: it is a **day-batched backlog, not a timer**. Each - run appends that day's ten, the dashboard shows the **oldest un-cleared batch - only**, and one 🧹 *clear* button drops that whole day and reveals the next — - so a fortnight away is a fortnight of batches to cycle through, not a - fortnight of lost recommendations. Same five per-paper actions as the inbox. - Format and transitions live in `scripts/interests_actions.py`. + from the inbox in one way: it is **day-batched**. Each run appends that day's + ten, the dashboard shows the **oldest un-cleared batch only**, and one 🧹 + *clear* button drops that whole day and reveals the next. The seven-day + window is the inbox's: a batch nobody clears lapses whole (`sweep` in + `scripts/interests_actions.py`, on `INBOX_WINDOW_DAYS`), because at a + measured 1.6 % clearance rate an unlapsing backlog just makes the oldest + batch — the one on the board — the least relevant thing on it. Same five + per-paper actions as the inbox. Format and transitions live in + `scripts/interests_actions.py`. - [`reading-queue.md`](reading-queue.md) — the paper queue AND the reading history: `## ` section headers group papers by domain, one title per line, and a read paper is never deleted — its line gains a `DONE <date> — ` @@ -68,7 +71,7 @@ reviewed`), and source-page layout are uniform across sub-wikis. ## Citation metadata The sub-wikis explain which claims papers support. Canonical BibTeX metadata, -key aliases, downstream-project resolution rules, and validation live in +downstream-project resolution rules, and validation live in [`bibliography/`](bibliography/README.md). ## Provenance @@ -77,6 +80,9 @@ The wikis were seeded in 2026-05 from a ~615-paper personal PDF library (the PDFs themselves were deleted from the repo the same day, backed up externally). Most legacy per-paper entries remain filename-inferred stubs: upgrade them to compact, claim-oriented `drafted` entries only after -verifying the paper, and log the change in the relevant sub-wiki. The live +verifying the paper, and log the change in the relevant sub-wiki. The eight +pages on which *every* entry is still such a stub were split out of `sources/` +into `wiki/<domain>/seed/` — same shape, nothing citeable, promoted a paper at +a time as each is verified. The live counts — pages, maturity, unresolved citation keys, reading queue — are on the [dashboard](https://pyautolabs.github.io/PyAutoMemory/). diff --git a/scripts/board.py b/scripts/board.py index ddc0c2b..07b1fff 100644 --- a/scripts/board.py +++ b/scripts/board.py @@ -28,9 +28,11 @@ the same overnight machinery pointed at everything that is *not* strong lensing — black holes, dark matter, galaxy formation, statistics — as one day-batch of ten. It renders the same five actions per paper, plus one 🧹 -*clear* button on the batch itself: the list is a backlog rather than a timer, -so the board shows the OLDEST un-cleared day only and clearing it reveals the -next. ``scripts/interests_actions.py`` owns that format and those transitions. +*clear* button on the batch itself: the board shows the OLDEST un-cleared day +only and clearing it reveals the next. That batch carries a days-left count of +its own, because it lapses on the inbox's window if nobody clears it — whole, +a day at a time. ``scripts/interests_actions.py`` owns that format and those +transitions. Both suggestion tiers also carry a **freshness line**: the date of the last digest run, so an empty inbox says *which* kind of empty it is — arXiv was quiet, or the @@ -217,6 +219,7 @@ def collect(root: Path | None = None) -> dict: # `interests_backlog` are how much is queued behind it. "interests": [], "interests_date": None, + "interests_days_left": None, "interests_batches": 0, "interests_backlog": 0, "interests_last_digest": None, @@ -233,7 +236,10 @@ def collect(root: Path | None = None) -> dict: pages = sorted(wdir.rglob("*.md")) stems |= {p.stem for p in pages} statuses: dict[str, int] = {} - kinds = {"concepts": 0, "entities": 0, "sources": 0} + # `seed/` is the 2026-05 import's unverified half, split out of + # `sources/`; counted separately so the sub-wiki row does not + # read as if those pages were verified claim support. + kinds = {"concepts": 0, "entities": 0, "sources": 0, "seed": 0} sections = 0 todo = 0 resolved: set[str] = set() @@ -288,9 +294,10 @@ def _flush(): papers.append(_parse_paper(line.strip())) _flush() + today = datetime.datetime.now(datetime.timezone.utc).date() + inbox = root / inbox_actions.INBOX_FILE if inbox.exists(): - today = datetime.datetime.now(datetime.timezone.utc).date() inbox_text = inbox.read_text(errors="replace") snapshot["inbox_last_digest"] = inbox_actions.last_digest(inbox_text) for entry in inbox_actions.papers(inbox_text): @@ -314,12 +321,19 @@ def _flush(): current = interests_actions.current_batch(text) if current: snapshot["interests_date"] = current[0] + # The batch is what lapses, so the days-left is the batch's, not + # each paper's — every line in it carries the same date. Kept on + # the papers too so an interests paper renders like an inbox one. + snapshot["interests_days_left"] = interests_actions.days_left( + current[0], today) for entry in current[1]: snapshot["interests"].append({ "title": entry["title"], "ref": entry["ref"], "line": entry["line"], "added": entry["added"], + "days_left": interests_actions.days_left(entry["added"], + today), "topic": entry["topic"], "section": interests_actions.section_for(entry), "done": False, @@ -763,8 +777,13 @@ def _render_md(snapshot: dict) -> str: if left else " · nothing behind it") clear_url = _interests_clear_url(snapshot, date, len(interests)) clear = f" · [clear this day \U0001f9f9]({clear_url})" if clear_url else "" + # The batch's days-left, beside the inbox's per-line one: a batch + # nobody clears is swept whole on the same window. + dleft = snapshot.get("interests_days_left") + left_bit = f" · {dleft}d left" if dleft is not None else "" lines += [f"_{date} · {len(interests)} " - f"paper{'s' if len(interests) != 1 else ''}{backlog}{clear}_", + f"paper{'s' if len(interests) != 1 else ''}{left_bit}" + f"{backlog}{clear}_", ""] for p_ in interests: label = p_["title"].replace("[", "\\[").replace("]", "\\]") @@ -1264,11 +1283,16 @@ def _fresh_span(fresh: dict, now_text: str, stale_text: str, # backlog depth matters most, and dropping it from the stale variant # hid it twice over: on a render that is already stale, and on a fresh # render the moment _EXTRA_JS swaps in `data-stale-text`. + # The batch's days-left rides on both texts too, for the same reason + # `behind` does: a late digest is exactly when it matters that the day + # on screen is about to be swept. + idleft = snapshot.get("interests_days_left") + ileft = f" · {idleft}d left" if idleft is not None else "" imeta = _fresh_span( ifresh, f"{len(interest_items)} paper" - f"{'s' if len(interest_items) != 1 else ''}{behind}", - f"{len(interest_items)} paper(s){behind} · {istale_tmpl}") + f"{'s' if len(interest_items) != 1 else ''}{ileft}{behind}", + f"{len(interest_items)} paper(s){ileft}{behind} · {istale_tmpl}") # The batch's date IS its name — this is a backlog of days, and which # day you are looking at is the first thing to know. interests_block = ( @@ -1321,7 +1345,8 @@ def _fresh_span(fresh: dict, now_text: str, stale_text: str, wiki_rows.append( f"<tr><td class='name'>{link}</td>" f"<td>{w['pages']} pages · {w.get('concepts', 0)}c/" - f"{w.get('entities', 0)}e/{w.get('sources', 0)}s</td>" + f"{w.get('entities', 0)}e/{w.get('sources', 0)}s" + f"{('/' + str(w['seed']) + ' seed') if w.get('seed') else ''}</td>" f"<td>{_bar(s)} <span class='meta'>{s.get('stub', 0)} stub · " f"{s.get('drafted', 0)} drafted</span> " f"{_copy_btn(_stub_prompt(snapshot, w['name']), 'copy: upgrade a stub')} " @@ -1392,7 +1417,8 @@ def _fresh_span(fresh: dict, now_text: str, stale_text: str, {inbox_block} <h2>arXiv interests <span class="muted">(everything that is not strong lensing — one day's ten at a time; \U0001f9f9 clears the day and shows the - next{interests_backlog_note})</span></h2> + next; un-cleared batches lapse after {inbox_actions.INBOX_WINDOW_DAYS} days +{interests_backlog_note})</span></h2> {interests_block} <h2>Reading queue <span class="muted">({trend_bits})</span>{spark}</h2> {filter_box} diff --git a/scripts/interests_actions.py b/scripts/interests_actions.py index d680969..5c04f48 100644 --- a/scripts/interests_actions.py +++ b/scripts/interests_actions.py @@ -1,29 +1,41 @@ -"""scripts/interests_actions.py — the arXiv interests list: append, stamp, clear, add, dismiss. +"""scripts/interests_actions.py — the arXiv interests list: append, stamp, sweep, clear, add, dismiss. The **second** suggestion tier, sibling of ``inbox_actions.py``. The arXiv inbox is strong lensing only; this one is everything else the human reads for — black holes, dark matter, galaxy formation, statistics — as the day's ten most relevant non-lensing papers, written by PyAutoMind's ``arxiv_interests.yml``. -It is a **day-batched backlog**, and that is the one real difference from the -inbox: +It is a **day-batched backlog**, and the batch is the one real difference from +the inbox — the granularity of everything, the window included: * The inbox is a flat list on a seven-day timer: act on a line or it lapses. -* This list never lapses. Each run appends one dated batch, the Dashboard shows - the **oldest un-cleared batch only**, and one 🧹 *clear* button drops that - whole day and reveals the next. So a week away is a week of batches to cycle - through, not a week of lost recommendations — the human asked to see every - day's ten, not the last seven days' worth of whatever survived. +* This list is the same timer applied to a **day at a time**. Each run appends + one dated batch, the Dashboard shows the **oldest un-cleared batch only**, and + one 🧹 *clear* button drops that whole day and reveals the next — but a batch + nobody clears lapses in its own right after :data:`inbox_actions.INBOX_WINDOW_DAYS` + days, whole, exactly as an un-acted inbox line does. + + It did not always. The list was built as "a backlog, not a timer — nothing + here lapses", on the reasoning that a fortnight away should be a fortnight of + batches to cycle through rather than a fortnight of lost recommendations. The + measurement said otherwise: of 123 papers appended, **2 were promoted to the + reading queue and 1 batch was cleared** — a 1.6 % clearance rate, with 114 + entries sitting unread in 11 day-batches. A backlog nobody walks forward + through is not a backlog, it is a pile that makes the oldest batch (the one + the Dashboard shows) the least relevant thing on the board. Seven days is how + long a suggestion is worth looking at; after that the recommendation has aged + out whether or not anyone tapped 🧹. Nothing else differs. The line format is the inbox's (``<date> — <text>``) with one addition — an optional ``[Topic]`` naming the reading-queue section the ➕ button files into, because these papers span domains where the inbox's all route to "Strong Lensing". The per-paper actions are the same four, and the freshness stamp is the same line, parsed by the same code: this module imports -``inbox_actions`` rather than restating any of it. +``inbox_actions`` rather than restating any of it — the window and ``days_left`` +included, so there is no second literal to drift. -A cleared batch is not lost — git history holds it, exactly as for a swept -inbox line, and for the same reason: a suggestion nobody acted on is not +A cleared or swept batch is not lost — git history holds it, exactly as for a +swept inbox line, and for the same reason: a suggestion nobody acted on is not reading history. Each subcommand prints exactly one status token on stdout. @@ -31,6 +43,7 @@ Usage: python scripts/interests_actions.py append --papers-file picks.json python scripts/interests_actions.py stamp + python scripts/interests_actions.py sweep python scripts/interests_actions.py clear --date 2026-08-25 python scripts/interests_actions.py add --body-file /tmp/issue_body.txt python scripts/interests_actions.py dismiss --body-file /tmp/issue_body.txt @@ -173,6 +186,40 @@ def append(interests_text: str, known_texts: list[str], entries: list[dict], return "\n".join(lines) + "\n", len(fresh) +def days_left(date: str, today: datetime.date) -> int: + """Days a batch has left before it lapses — the inbox's window, unchanged. + + A one-line passthrough on purpose: the window and the arithmetic have one + owner (``inbox_actions``), and callers that want a batch's days-left (the + board's column, the sweep below) should not have to know that. + """ + return inbox_actions.days_left(date, today) + + +def sweep(interests_text: str, + today: datetime.date) -> tuple[str, list[tuple[str, int]]]: + """Drop every batch past the window. Returns ``(new_text, lapsed)``. + + ``lapsed`` is ``[(date, papers_dropped)]``, oldest first. A **whole day** + goes or none of it does: the 🧹 button acts on a day, so the day is the unit + the human works in and the unit that ages out. Like the inbox's sweep this + is idempotent — a second run finds nothing left to drop — and what it drops + stays in git history. + """ + lapsed = [(date, len(batch)) for date, batch in batches(interests_text) + if days_left(date, today) == 0] + if not lapsed: + return interests_text, [] + dead = {date for date, _ in lapsed} + out = [] + for raw in interests_text.rstrip("\n").splitlines(): + parsed = parse_line(raw) + if parsed and parsed[0] in dead: + continue + out.append(raw) + return "\n".join(out) + "\n", lapsed + + def clear(interests_text: str, date: str) -> tuple[str, int]: """Drop a whole day's batch. Returns ``(new_text, dropped_count)``. @@ -267,6 +314,17 @@ def _cmd_stamp(ns) -> tuple[str, int]: return (f"stamped:{date}", 0) +def _cmd_sweep(ns) -> tuple[str, int]: + interests = Path(ns.interests) + text, lapsed = sweep(interests.read_text(errors="replace"), _today(ns)) + if lapsed: + interests.write_text(text) + for date, n in lapsed: + print(f"lapsed: {date} ({n} paper{'s' if n != 1 else ''})", + file=sys.stderr) + return (f"swept:{sum(n for _, n in lapsed)}", 0) + + def _cmd_clear(ns) -> tuple[str, int]: interests = Path(ns.interests) text = interests.read_text(errors="replace") @@ -341,6 +399,9 @@ def main(argv: list[str] | None = None) -> int: p = sub.add_parser("stamp", help="record that the digest ran today") p.set_defaults(fn=_cmd_stamp) + p = sub.add_parser("sweep", help="drop whole batches past the window") + p.set_defaults(fn=_cmd_sweep) + p = sub.add_parser("clear", help="drop a whole day's batch") p.add_argument("--date", dest="batch_date", default=None, help="the batch to clear (YYYY-MM-DD)") diff --git a/scripts/validate_literature_citations.py b/scripts/validate_literature_citations.py index befeaf8..7e872c1 100644 --- a/scripts/validate_literature_citations.py +++ b/scripts/validate_literature_citations.py @@ -77,8 +77,21 @@ def extract_canonical_keys( ) +#: Folders whose ``*.md`` carry per-paper sections. ``seed/`` is the 2026-05 +#: import's unverified half, split out of ``sources/`` so nothing cites it by +#: accident — but it is walked exactly like ``sources/``: a seed page is where +#: a TODO key is most likely to be, and a validator that skipped it would go +#: quiet on the pages that need it most. +SOURCE_FOLDERS = ("sources", "seed") + + def source_directories(root: Path) -> tuple[Path, ...]: - return tuple(sorted(path for path in root.glob("wiki/*/sources") if path.is_dir())) + return tuple(sorted( + path + for folder in SOURCE_FOLDERS + for path in root.glob(f"wiki/*/{folder}") + if path.is_dir() + )) def collect_source_citations(directories: tuple[Path, ...]) -> tuple[SourceCitation, ...]: diff --git a/scripts/validate_wikilinks.py b/scripts/validate_wikilinks.py new file mode 100644 index 0000000..2526eb1 --- /dev/null +++ b/scripts/validate_wikilinks.py @@ -0,0 +1,193 @@ +"""Resolve every ``[[wikilink]]`` in the wiki against the pages that exist. + +``[[slug]]`` is the wiki's cross-reference mechanism (``wiki/CLAUDE.md``, +"Cross-references"): a slug is a page's filename without ``.md``, slugs are +unique across sub-wikis so a link needs no path (``wiki/smbh/CLAUDE.md``), and +a link with no target yet is *legal* — it marks a page somebody meant to write. + +That last rule is why this lint is a **ratchet, not a gate**. The wiki carries +hundreds of deliberate wanted-page links; failing on them would only teach +everyone to ignore the lint. So it fails on exactly one thing: an unresolved +*target* that is not in ``scripts/wikilink_baseline.txt``. Adding a link to a +target already known-dangling is fine (it is the same wanted page, referenced +once more); inventing a new dangling slug — the typo case, and the case where a +page was renamed out from under its references — is not. + +Resolution, in full: + +* A link is ``[[target]]``, ``[[target#anchor]]`` or ``[[target|label]]`` + (and the two combined). The anchor is **not** validated — it points inside a + page, and the source pages' per-paper sections are not a structure this lint + should freeze. The label is display text. +* ``target`` resolves if some ``wiki/<domain>/**/*.md`` has it as its stem — + **any** sub-wiki, because slugs are unique across them and cross-wiki links + are documented. +* A source page carries a second slug: ``wiki/<d>/sources/<topic>.md`` resolves + both ``[[<topic>]]`` and ``[[sources-<topic>]]``, which is the form the + schema's ``[[sources-<topic>#author-year-slug]]`` uses. Seed pages + (``wiki/<d>/seed/<topic>.md``) resolve the same two ways. +* Fenced code blocks and inline code spans are skipped, so the schema pages can + show the syntax (``[[page-slug]]``, ``[[related-concept-1]]``) without those + examples counting as references. + +Nothing resolves against the bibliography: a citation is a source *section* +reached through its page, and no unresolved target in the wiki today is a +canonical key (five collide with software keys — ``emcee``, ``dynesty`` — and +those are wanted entity pages, not citations). + +Usage: + python scripts/validate_wikilinks.py # summary, exit 1 on new + python scripts/validate_wikilinks.py --list # + the dangling targets + python scripts/validate_wikilinks.py --write-baseline +""" + +from __future__ import annotations + +import argparse +import re +import sys +from collections import Counter +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] +DEFAULT_BASELINE = Path(__file__).resolve().parent / "wikilink_baseline.txt" + +#: ``[[target]]``, ``[[target#anchor]]``, ``[[target|label]]``. Non-greedy and +#: capped so an unclosed ``[[`` swallows a phrase, never half the page; DOTALL +#: because a link wrapped across two lines is still one link. +WIKILINK_RE = re.compile(r"\[\[([^\[\]]{1,200}?)\]\]", re.DOTALL) +FENCE_RE = re.compile(r"^\s*(```|~~~)") +INLINE_CODE_RE = re.compile(r"`[^`\n]*`") + +#: Page folders whose stems also answer to a ``<folder>-<stem>`` slug. +PREFIXED_FOLDERS = ("sources", "seed") + +#: The schema page itself is documentation *of* the syntax, not a page of +#: references: every example slug in it is illustrative. +SKIP_FILES = ("wiki/CLAUDE.md",) + + +def strip_code(text: str) -> str: + """Blank out fenced blocks and inline code, keeping line count intact.""" + out: list[str] = [] + fence: str | None = None + for line in text.splitlines(): + match = FENCE_RE.match(line) + if match: + token = match.group(1) + if fence is None: + fence = token + elif line.strip().startswith(fence): + fence = None + out.append("") + continue + out.append("" if fence else INLINE_CODE_RE.sub("", line)) + return "\n".join(out) + + +def wiki_files(root: Path) -> list[Path]: + skip = {(root / rel).resolve() for rel in SKIP_FILES} + return [p for p in sorted(root.glob("wiki/**/*.md")) + if p.resolve() not in skip] + + +def page_slugs(root: Path) -> set[str]: + """Every slug a link may resolve to, across all sub-wikis.""" + slugs: set[str] = set() + for path in sorted(root.glob("wiki/**/*.md")): + parts = path.relative_to(root).parts + slugs.add(path.stem) + if len(parts) >= 4 and parts[2] in PREFIXED_FOLDERS: + slugs.add(f"{parts[2]}-{path.stem}") + return slugs + + +def link_target(raw: str) -> str: + """The page slug a raw link body names — anchor and label removed.""" + target = raw.split("#", 1)[0].split("|", 1)[0] + return " ".join(target.split()).strip() + + +def collect_links(root: Path) -> list[tuple[Path, str]]: + """``(file, target)`` for every wikilink in the wiki, in file order.""" + found: list[tuple[Path, str]] = [] + for path in wiki_files(root): + body = strip_code(path.read_text(encoding="utf-8", errors="replace")) + for match in WIKILINK_RE.finditer(body): + target = link_target(match.group(1)) + if target: + found.append((path, target)) + return found + + +def unresolved(root: Path) -> tuple[int, Counter]: + """``(total_links, Counter(target -> occurrences))`` for dangling links.""" + slugs = page_slugs(root) + links = collect_links(root) + counts: Counter = Counter() + for _, target in links: + if target not in slugs: + counts[target] += 1 + return len(links), counts + + +def read_baseline(path: Path) -> set[str]: + if not path.exists(): + return set() + return {line.strip() for line in path.read_text(encoding="utf-8").splitlines() + if line.strip() and not line.startswith("#")} + + +def write_baseline(path: Path, targets: set[str]) -> None: + path.write_text("\n".join(sorted(targets)) + "\n", encoding="utf-8") + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--root", type=Path, default=ROOT) + parser.add_argument("--baseline", type=Path, default=DEFAULT_BASELINE) + parser.add_argument("--list", action="store_true", dest="show_list", + help="print the unresolved targets, most-referenced first") + parser.add_argument("--write-baseline", action="store_true", + help="rewrite the baseline from the current state") + args = parser.parse_args(argv) + + root = args.root.resolve() + total, counts = unresolved(root) + targets = set(counts) + print(f"{total} links, {sum(counts.values())} unresolved " + f"across {len(targets)} targets") + + if args.show_list: + for target, n in sorted(counts.items(), key=lambda kv: (-kv[1], kv[0])): + print(f" {n:4d} {target}") + + baseline = read_baseline(args.baseline) + + if args.write_baseline: + dropped = baseline - targets + added = targets - baseline + write_baseline(args.baseline, targets) + print(f"baseline rewritten: {len(targets)} targets " + f"(+{len(added)} new, -{len(dropped)} resolved since)") + return 0 + + fresh = sorted(targets - baseline) + if fresh: + print(f"ERROR: {len(fresh)} unresolved target(s) not in " + f"{args.baseline.name}:") + for target in fresh: + print(f" - {target} ({counts[target]} reference(s))") + print("A dangling link is legal — a NEW dangling target is not. Write " + "the page, fix the slug, or accept it with " + "`python scripts/validate_wikilinks.py --write-baseline`.") + return 1 + + gone = len(baseline - targets) + tail = f" ({gone} baseline target(s) now resolve)" if gone else "" + print(f"Wikilinks are valid — no new unresolved targets{tail}.") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/wikilink_baseline.txt b/scripts/wikilink_baseline.txt new file mode 100644 index 0000000..bc875d3 --- /dev/null +++ b/scripts/wikilink_baseline.txt @@ -0,0 +1,112 @@ +active-learning +arctic-algorithm +big-mac +blackjax +bulge-disk-decomposition-galaxies +bulges-vs-pseudobulges +califa-survey +ccd-trap-physics +ceers +charge-injection +chromatic-cmod +cosmos-web +cosmos-web-agn +cti-flat-field +cti-photometry-bias +cti-warm-pixels +deep-learning-astro +dm-rotation-curves +dragonfly +dual-agn +dwarf-galaxy-dynamics +dynesty +emcee +epta +equinox +eryn +etg-kinematics +etg-radial-gradients +euclid-vis-cti +final-parsec-problem +finufft +forward-model-cti +forward-modelling +galaxy-morphology +galaxy-quenching +gaussian-linear-models +gilman-pipeline +halo-mass-proxies +halo-occupation +hierarchical-population-models +high-redshift-galaxies +hsc-survey +hubble-acs +hubble-acs-cti +ipta +jades +jax-ecosystem +jax-finufft +jaxns +jaxopt +jwst-galaxies +lampe +lens-modeling-systematics +lensing-dynamics-joint +likelihood-emulation +linear-inversion +lisa-mbhb +little-red-dots +low-surface-brightness +ltu-ili +manga +manga-sami-califa +manga-survey +mcmc-samplers +message-passing +mge +model-comparison +multinest +nautilus +nbi +neural-emulators +neural-posterior-estimation +ngc-1277 +ngc-1380 +ngc-4736 +ngc-6861 +numpyro +optimistix +overmassive-undermassive-smbhs +pks-0745-191 +pks-2131-021 +plasma-lensing +polychord +primordial-bh-microlensing +psf-modelling +pulsar-timing-arrays +pylops +pymc +pynufft +recoil-host-signatures +refsdal-supernova +roman-cti +roman-wfi +sami-survey +scatter-in-smhm +sersic-profile +simulation-based-inference +smbh-accretion +smbh-recoil +smbh-seeds +sources-dual-agn +sources-kinematics-galaxies +sources-smbh-detection +sources-smbh-host-galaxies +stan +stellar-mass-halo-mass-relation +stellar-to-halo-mass +triple-smbh +ultranest +uvis-survey +vlbi-smbh +x-ray-recoiling diff --git a/tests/test_board.py b/tests/test_board.py index 41e86e9..82f4efe 100644 --- a/tests/test_board.py +++ b/tests/test_board.py @@ -80,6 +80,20 @@ def _snap_with_remote(tmp_path): return snap +def test_seed_pages_are_counted_apart_from_sources(tmp_path): + # `seed/` is the unverified import; a sub-wiki row that folded it into + # `sources` would read as if those pages were verified claim support. + root = _tree(tmp_path) + seed = root / "wiki" / "demo" / "seed" + seed.mkdir() + (seed / "unverified.md").write_text( + "---\ntitle: Seed\ntype: sources\nstatus: stub\n---\n\n" + "## One\n**Canonical BibTeX key:** TODO — no unique match found.\n") + (w,) = board.collect(root)["wikis"] + assert (w["sources"], w["seed"], w["pages"]) == (1, 1, 4) + assert "1 seed" in board.render(board.collect(root), "html") + + def test_counts_from_a_synthetic_tree(tmp_path): snap = board.collect(_tree(tmp_path)) (w,) = snap["wikis"] diff --git a/tests/test_interests_actions.py b/tests/test_interests_actions.py index dc7eb93..6de7498 100644 --- a/tests/test_interests_actions.py +++ b/tests/test_interests_actions.py @@ -1,15 +1,17 @@ """tests/test_interests_actions.py — the arXiv interests list. -The list's contract, and specifically the two things that make it NOT the +The list's contract, and specifically the thing that makes it NOT the strong-lensing inbox: it is grouped into day batches with the oldest one -current, and `clear` drops a whole day rather than a line lapsing on a timer. -Everything else — line format, dedup, the add/dismiss transitions — is the -inbox's contract, tested here because this module owns the `[Topic]` the inbox -lines do not carry. +current, so `clear` and `sweep` both act on a whole day where the inbox acts on +a line. The window itself is the inbox's — a batch nobody clears lapses after +`INBOX_WINDOW_DAYS`. Everything else — line format, dedup, the add/dismiss +transitions — is the inbox's contract, tested here because this module owns the +`[Topic]` the inbox lines do not carry. """ from __future__ import annotations +import datetime import json import sys from pathlib import Path @@ -18,6 +20,7 @@ sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "scripts")) +import inbox_actions as ib # noqa: E402 import interests_actions as ia # noqa: E402 LIST = """# arXiv interests @@ -197,6 +200,48 @@ def test_clear_keeps_the_prose_and_the_stamp(): assert "last digest: 2026-08-27" in text + +# --- sweep -------------------------------------------------------------------- +def _day(iso): + return datetime.date.fromisoformat(iso) + + +def test_days_left_is_the_inboxs_window_unchanged(): + # One owner for the number. If the inbox's window moves, this moves with it. + today = _day("2026-08-27") + assert ia.days_left("2026-08-25", today) == ib.days_left("2026-08-25", today) + assert ia.days_left("2026-08-25", today) == ib.INBOX_WINDOW_DAYS - 2 + + +def test_sweep_drops_only_batches_past_the_window(): + # 2026-08-25 is seven days old on 2026-09-01; 2026-08-26 has a day left. + text, lapsed = ia.sweep(LIST, _day("2026-09-01")) + assert lapsed == [("2026-08-25", 2)] + assert [p["added"] for p in ia.papers(text)] == ["2026-08-26"] + + +def test_sweep_takes_the_whole_batch_never_part_of_one(): + text, lapsed = ia.sweep(LIST, _day("2026-09-03")) + assert lapsed == [("2026-08-25", 2), ("2026-08-26", 1)] + assert ia.papers(text) == [] + # The prose header and the freshness stamp are not papers and must survive. + assert "Prose header, not papers." in text + assert "last digest: 2026-08-27" in text + + +def test_sweep_leaves_a_fresh_list_untouched(): + text, lapsed = ia.sweep(LIST, _day("2026-08-27")) + assert lapsed == [] + assert text == LIST + + +def test_sweep_is_idempotent(): + once, first = ia.sweep(LIST, _day("2026-09-01")) + twice, second = ia.sweep(once, _day("2026-09-01")) + assert second == [] + assert twice == once + + # --- remove / add_to_queue ---------------------------------------------------- def test_remove_takes_the_line_as_written(): line = "2026-08-25 — [SMBHs] A tidal disruption event — 2608.30001" @@ -283,6 +328,23 @@ def test_cli_append_and_stamp(repo, capsys): assert "2026-08-28 — [Stats] A new paper — 2608.50001" in text +def test_cli_sweep_drops_lapsed_batches(repo, capsys): + assert _run(repo, "--today", "2026-09-01", "sweep") == 0 + out = capsys.readouterr() + assert out.out.strip() == "swept:2" + assert "lapsed: 2026-08-25 (2 papers)" in out.err + text = (repo / "arxiv-interests.md").read_text() + assert "2608.30001" not in text + assert "2608.30004" in text + + +def test_cli_sweep_reports_a_quiet_run_without_touching_the_file(repo, capsys): + before = (repo / "arxiv-interests.md").read_text() + assert _run(repo, "--today", "2026-08-27", "sweep") == 0 + assert capsys.readouterr().out.strip() == "swept:0" + assert (repo / "arxiv-interests.md").read_text() == before + + def test_cli_clear_by_date_then_by_body(repo, capsys): assert _run(repo, "clear", "--date", "2026-08-25") == 0 assert capsys.readouterr().out.strip() == "cleared:2" diff --git a/tests/test_validate_literature_citations.py b/tests/test_validate_literature_citations.py index 74543ca..59f273f 100644 --- a/tests/test_validate_literature_citations.py +++ b/tests/test_validate_literature_citations.py @@ -75,3 +75,22 @@ def test_detect_claim_entry_without_key(tmp_path): ("Alpha 2024 — result", 1) ] assert not result.valid + + +def test_seed_pages_are_validated_like_sources(tmp_path): + # `seed/` holds the 2026-05 import's unverified source pages. It is where a + # broken canonical key is most likely to sit, so the validator must walk it + # rather than go quiet on the half of the wiki that needs it most. + bibliography, _ = _fixture_paths(tmp_path) + seed = tmp_path / "wiki" / "example" / "seed" + seed.mkdir(parents=True) + (seed / "topic.md").write_text( + "**Canonical BibTeX key:** `Missing2025`\n", encoding="utf-8" + ) + + result = validate_citations(bibliography, tmp_path) + + assert [citation.key for citation in result.missing_source_keys] == [ + "Missing2025" + ] + assert not result.valid diff --git a/tests/test_validate_wikilinks.py b/tests/test_validate_wikilinks.py new file mode 100644 index 0000000..16b99d6 --- /dev/null +++ b/tests/test_validate_wikilinks.py @@ -0,0 +1,178 @@ +"""tests/test_validate_wikilinks.py — the wikilink ratchet. + +What the lint must get right: the documented link forms all resolve (bare slug, +anchor, label, the `sources-<topic>` form, and across sub-wikis), a link with no +page is not an error *by itself*, and a dangling target that is not in the +baseline is. Plus the two things that make the baseline usable: `--list` and +`--write-baseline`. +""" + +from __future__ import annotations + +import sys +from pathlib import Path + +import pytest + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "scripts")) + +import validate_wikilinks as vw # noqa: E402 + + +def _page(root: Path, rel: str, body: str) -> Path: + path = root / rel + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(body, encoding="utf-8") + return path + + +@pytest.fixture +def wiki(tmp_path): + """A two-sub-wiki fixture exercising every documented link form.""" + _page(tmp_path, "wiki/lensing/index.md", + "# Lensing\n\n- [[mass-sheet-degeneracy]]\n- [[h0licow]]\n") + _page(tmp_path, "wiki/lensing/concepts/mass-sheet-degeneracy.md", + "# MSD\n\nSee [[h0licow]] and " + "[[sources-time-delay#suyu-2017-h0licow]].\n" + "Across sub-wikis: [[smbh-seeds]].\n" + "With a label: [[h0licow|the H0LiCOW collaboration]].\n") + _page(tmp_path, "wiki/lensing/entities/h0licow.md", "# H0LiCOW\n") + _page(tmp_path, "wiki/lensing/sources/time-delay.md", + "# Sources: time delay\n\n## Suyu 2017 — H0LiCOW\n" + "Also reachable as [[time-delay]].\n") + _page(tmp_path, "wiki/smbh/concepts/smbh-seeds.md", + "# Seeds\n\nBack to [[mass-sheet-degeneracy]].\n") + return tmp_path + + +def _run(root, *argv, baseline=None): + args = ["--root", str(root), "--baseline", + str(baseline if baseline else root / "baseline.txt"), *argv] + return vw.main(args) + + +# --- resolution --------------------------------------------------------------- +def test_every_documented_form_resolves(wiki, capsys): + total, counts = vw.unresolved(wiki) + assert counts == {} + assert total == 8 + + +def test_a_source_page_answers_to_both_of_its_slugs(wiki): + slugs = vw.page_slugs(wiki) + assert {"time-delay", "sources-time-delay"} <= slugs + + +def test_a_seed_page_answers_to_both_of_its_slugs(tmp_path): + _page(tmp_path, "wiki/galaxies/seed/stellar-halos.md", "# Seed\n") + assert {"stellar-halos", "seed-stellar-halos"} <= vw.page_slugs(tmp_path) + + +def test_the_anchor_is_not_validated(wiki): + _page(wiki, "wiki/lensing/concepts/anchored.md", + "# A\n\n[[h0licow#no-such-section]]\n") + assert vw.unresolved(wiki)[1] == {} + + +def test_code_is_not_a_reference(wiki): + _page(wiki, "wiki/lensing/concepts/schema-ish.md", + "# Schema\n\nUse `[[page-slug]]` like this:\n\n" + "```\n[[related-concept-1]]\n```\n\nReal one: [[h0licow]].\n") + total, counts = vw.unresolved(wiki) + assert counts == {} + assert total == 9 # only the real one was counted + + +def test_a_link_wrapped_across_two_lines_is_one_link(wiki): + _page(wiki, "wiki/lensing/concepts/wrapped.md", + "# W\n\nThe [[mass-sheet-degeneracy|mass-sheet\ndegeneracy]] again.\n") + assert vw.unresolved(wiki)[1] == {} + + +def test_unresolved_targets_are_counted_by_occurrence(wiki): + _page(wiki, "wiki/lensing/concepts/wanted.md", + "# W\n\n[[not-written-yet]] and [[not-written-yet]] and [[other]].\n") + _, counts = vw.unresolved(wiki) + assert counts == {"not-written-yet": 2, "other": 1} + + +# --- the ratchet -------------------------------------------------------------- +def test_a_clean_wiki_passes(wiki, capsys): + assert _run(wiki) == 0 + out = capsys.readouterr().out + assert "8 links, 0 unresolved across 0 targets" in out + + +def test_a_new_unresolved_target_fails(wiki, capsys): + _page(wiki, "wiki/lensing/concepts/typo.md", "# T\n\n[[h0licow-typo]]\n") + assert _run(wiki) == 1 + out = capsys.readouterr().out + assert "1 unresolved across 1 targets" in out + assert "- h0licow-typo (1 reference(s))" in out + + +def test_an_unresolved_target_already_in_the_baseline_passes(wiki, capsys): + _page(wiki, "wiki/lensing/concepts/wanted.md", "# W\n\n[[weak-lensing]]\n") + (wiki / "baseline.txt").write_text("weak-lensing\n", encoding="utf-8") + assert _run(wiki) == 0 + assert "no new unresolved targets" in capsys.readouterr().out + + +def test_another_reference_to_a_baselined_target_still_passes(wiki): + _page(wiki, "wiki/lensing/concepts/wanted.md", + "# W\n\n[[weak-lensing]] [[weak-lensing]] [[weak-lensing]]\n") + (wiki / "baseline.txt").write_text("weak-lensing\n", encoding="utf-8") + assert _run(wiki) == 0 + + +def test_one_new_target_fails_even_beside_a_baselined_one(wiki, capsys): + _page(wiki, "wiki/lensing/concepts/wanted.md", + "# W\n\n[[weak-lensing]] and [[brand-new]]\n") + (wiki / "baseline.txt").write_text("weak-lensing\n", encoding="utf-8") + assert _run(wiki) == 1 + assert "- brand-new" in capsys.readouterr().out + + +def test_a_resolved_baseline_target_is_reported_not_failed(wiki, capsys): + (wiki / "baseline.txt").write_text("weak-lensing\n", encoding="utf-8") + assert _run(wiki) == 0 + assert "1 baseline target(s) now resolve" in capsys.readouterr().out + + +def test_a_missing_baseline_is_an_empty_one(wiki, capsys): + _page(wiki, "wiki/lensing/concepts/wanted.md", "# W\n\n[[weak-lensing]]\n") + assert _run(wiki, baseline=wiki / "nope.txt") == 1 + + +# --- --list / --write-baseline ------------------------------------------------ +def test_list_prints_targets_most_referenced_first(wiki, capsys): + _page(wiki, "wiki/lensing/concepts/wanted.md", + "# W\n\n[[once]] [[twice]] [[twice]]\n") + (wiki / "baseline.txt").write_text("once\ntwice\n", encoding="utf-8") + assert _run(wiki, "--list") == 0 + listed = [line.split()[-1] for line in capsys.readouterr().out.splitlines() + if line.startswith(" ")] + assert listed == ["twice", "once"] + + +def test_write_baseline_records_the_current_state(wiki, capsys): + _page(wiki, "wiki/lensing/concepts/wanted.md", + "# W\n\n[[beta]] and [[alpha]]\n") + assert _run(wiki, "--write-baseline") == 0 + assert (wiki / "baseline.txt").read_text() == "alpha\nbeta\n" + assert "+2 new, -0 resolved" in capsys.readouterr().out + assert _run(wiki) == 0 + + +def test_write_baseline_reports_what_it_dropped(wiki, capsys): + (wiki / "baseline.txt").write_text("gone-now\nalso-gone\n", encoding="utf-8") + _page(wiki, "wiki/lensing/concepts/wanted.md", "# W\n\n[[still-wanted]]\n") + assert _run(wiki, "--write-baseline") == 0 + assert "+1 new, -2 resolved" in capsys.readouterr().out + assert (wiki / "baseline.txt").read_text() == "still-wanted\n" + + +# --- the real wiki ------------------------------------------------------------ +def test_the_checked_in_baseline_holds(): + """The repo's own state passes — the ratchet starts where the wiki is.""" + assert vw.main([]) == 0 diff --git a/wiki/CLAUDE.md b/wiki/CLAUDE.md index f15bea8..7982c74 100644 --- a/wiki/CLAUDE.md +++ b/wiki/CLAUDE.md @@ -19,8 +19,9 @@ PyAutoMemory/ # repo root │ ├── log.md # append-only compilation log │ ├── concepts/ # one topic per page — the science │ ├── entities/ # named things: surveys, instruments, software -│ └── sources/ # compact claim support (one paper = one section) -└── bibliography/ # canonical BibTeX, aliases, citation tooling +│ ├── sources/ # compact claim support (one paper = one section) +│ └── seed/ # unverified 2026-05 import, not citeable yet +└── bibliography/ # canonical BibTeX and citation tooling ``` Papers are the ground truth; wiki pages are syntheses. If they disagree, update @@ -29,6 +30,8 @@ the wiki and note the change in the sub-wiki's `log.md`. ## References and citation metadata - `sources/*.md` records compact guidance about what claims a paper supports. +- `seed/*.md` is the same shape, unverified: the 2026-05 import's stubs, held + apart so nothing cites them by accident. - `../bibliography/pyautomemory.bib` records canonical metadata and keys. Never record local PDF paths or fabricate metadata — source PDFs live @@ -48,6 +51,7 @@ before patching LaTeX. See | Concept | `concepts/` | One scientific concept (e.g. mass-sheet degeneracy) | | Entity | `entities/` | One named thing (survey, lens, code, collaboration) | | Sources | `sources/` | Claim support for one topic, one section per paper | +| Seed | `seed/` | Source pages from the 2026-05 import whose every entry is still a filename-inferred stub (`Canonical BibTeX key: TODO`) — never cite one until the paper is verified and the entry promoted into `sources/` | | Index/log | root | Navigation and provenance | ## Naming @@ -61,8 +65,11 @@ before patching LaTeX. See Use `[[page-slug]]` for wiki-internal links — for example `[[mass-sheet-degeneracy]]` or `[[h0licow]]`. Slugs match the filename without -`.md`. A `[[link]]` that has no target file yet is fine — it marks a future -page to write. +`.md`; a `sources/` page also answers to `[[sources-<topic>]]` and a `seed/` +page to `[[seed-<topic>]]`. A `[[link]]` that has no target file yet is fine — +it marks a future page to write, and `scripts/validate_wikilinks.py` ratchets +that backlog (it fails only on a *new* unresolved target, against +`scripts/wikilink_baseline.txt`). External references use verified DOI, arXiv, journal, or author/year/title metadata, never a local path. @@ -155,5 +162,6 @@ abstracts or infer claims from filenames. Add a TODO when support is unverified. ## Adding a sub-wiki Create `wiki/<domain>/` with its own `CLAUDE.md` (scope notes only — this -schema is inherited, not copied), `index.md`, `log.md`, and the three page -folders. Give it a row in the repo-root `index.md` and `README.md` tables. +schema is inherited, not copied), `index.md`, `log.md`, and the page folders it +needs (`seed/` only if it is taking an unverified import). Give it a row in the +repo-root `index.md` and `README.md` tables. diff --git a/wiki/galaxies/CLAUDE.md b/wiki/galaxies/CLAUDE.md index 800a055..8318434 100644 --- a/wiki/galaxies/CLAUDE.md +++ b/wiki/galaxies/CLAUDE.md @@ -18,7 +18,8 @@ galaxies/ ├── log.md # append-only compilation log ├── concepts/ # galaxy-formation & morphology topics ├── entities/ # surveys (MaNGA, SAMI, COSMOS-Web), notable galaxies -└── sources/ # per-topic claim support +├── sources/ # per-topic claim support +└── seed/ # unverified 2026-05 import — never cite from here ``` ## Schema diff --git a/wiki/galaxies/concepts/dm-halo-shapes.md b/wiki/galaxies/concepts/dm-halo-shapes.md index 8aeaa5e..4247f5e 100644 --- a/wiki/galaxies/concepts/dm-halo-shapes.md +++ b/wiki/galaxies/concepts/dm-halo-shapes.md @@ -45,9 +45,9 @@ parameters — a possible systematic on inferred shear / convergence. ## Key references -- Satellite alignment statistics: [[sources-dark-matter-geometry-galaxies]]. +- Satellite alignment statistics: [[seed-dark-matter-geometry-galaxies]]. - Intrinsic ETG shapes: [[sources-ifu-spectroscopy#weijmans-2014-intrinsic-shape-etg-kinematic-misalignments]]. -- Hydrodynamic comparison: [[sources-dark-matter-geometry-galaxies#tenneti-2015-intrinsic-alignment-disk-etg]]. +- Hydrodynamic comparison: [[seed-dark-matter-geometry-galaxies#tenneti-2015-intrinsic-alignment-disk-etg]]. ## See also diff --git a/wiki/galaxies/index.md b/wiki/galaxies/index.md index 90ef9ab..64dc6de 100644 --- a/wiki/galaxies/index.md +++ b/wiki/galaxies/index.md @@ -82,16 +82,18 @@ These pages describe claim support. Canonical citation metadata and key-manageme live in [`../../bibliography/`](../../bibliography/README.md). - [[sources-elliptical-galaxies]] -- [[sources-bulge-disk-decomposition]] - [[sources-light-profile-fitting]] - [[sources-ifu-spectroscopy]] - [[sources-kinematics-galaxies]] - [[sources-cosmos-survey]] -- [[sources-stellar-halos]] -- [[sources-halo-galaxy-connection]] - [[sources-high-redshift]] -- [[sources-dark-matter-geometry-galaxies]] -- [[sources-galaxy-formation-misc]] +- **Seed (unverified imports):** [[seed-bulge-disk-decomposition]], + [[seed-stellar-halos]], [[seed-halo-galaxy-connection]], + [[seed-dark-matter-geometry-galaxies]], [[seed-galaxy-formation-misc]] — + filename-inferred stubs from the 2026-05 PDF import, every entry still + reading `Canonical BibTeX key: TODO — no unique match found`. They live in + `seed/`, not `sources/`, and **nothing in them may be cited** until the paper + itself is verified and the entry promoted into the matching `sources/` page. ## Meta diff --git a/wiki/galaxies/log.md b/wiki/galaxies/log.md index 502bd0a..c69a139 100644 --- a/wiki/galaxies/log.md +++ b/wiki/galaxies/log.md @@ -50,3 +50,22 @@ sibling papers when reading the PDFs. Same Karpathy "LLM Wiki" pattern as the lensing sub-wiki. The largest sibling sub-wiki by paper count. + +--- + +## 2026-09-17 — Unverified import seed split out of `sources/` + +**By:** Claude (Opus 5, PyAutoMemory seed-split task). + +**What changed:** 5 pages (87 entries, 1,487 lines) moved from `sources/` to `seed/`: `bulge-disk-decomposition`, `stellar-halos`, `halo-galaxy-connection`, `dark-matter-geometry-galaxies`, `galaxy-formation-misc`. Every +entry on them is a filename-inferred stub from the 2026-05 PDF import whose +canonical key still reads `TODO — no unique match found`, so none of it is +citeable and none of it was ever claim-verified. `sources/` now holds only +pages with at least some verified support; `seed/` holds the raw import +waiting for it. + +**Nothing was deleted or rewritten** — the pages moved, their `[[sources-…]]` +references became `[[seed-…]]`, and each index gained one *Seed (unverified +imports)* line saying what they are. Promoting an entry means verifying the +paper, adding its canonical key to `../../bibliography/pyautomemory.bib`, and +moving the section into the matching `sources/` page. diff --git a/wiki/galaxies/sources/bulge-disk-decomposition.md b/wiki/galaxies/seed/bulge-disk-decomposition.md similarity index 100% rename from wiki/galaxies/sources/bulge-disk-decomposition.md rename to wiki/galaxies/seed/bulge-disk-decomposition.md diff --git a/wiki/galaxies/sources/dark-matter-geometry-galaxies.md b/wiki/galaxies/seed/dark-matter-geometry-galaxies.md similarity index 100% rename from wiki/galaxies/sources/dark-matter-geometry-galaxies.md rename to wiki/galaxies/seed/dark-matter-geometry-galaxies.md diff --git a/wiki/galaxies/sources/galaxy-formation-misc.md b/wiki/galaxies/seed/galaxy-formation-misc.md similarity index 99% rename from wiki/galaxies/sources/galaxy-formation-misc.md rename to wiki/galaxies/seed/galaxy-formation-misc.md index 698df03..879782f 100644 --- a/wiki/galaxies/sources/galaxy-formation-misc.md +++ b/wiki/galaxies/seed/galaxy-formation-misc.md @@ -498,4 +498,4 @@ review collection. - [[etg-structure]] - [[stellar-mass-halo-mass-relation]] - [[sources-massive-ellipticals]] -- [[sources-stellar-halos]] +- [[seed-stellar-halos]] diff --git a/wiki/galaxies/sources/halo-galaxy-connection.md b/wiki/galaxies/seed/halo-galaxy-connection.md similarity index 99% rename from wiki/galaxies/sources/halo-galaxy-connection.md rename to wiki/galaxies/seed/halo-galaxy-connection.md index c745cf6..7d484a6 100644 --- a/wiki/galaxies/sources/halo-galaxy-connection.md +++ b/wiki/galaxies/seed/halo-galaxy-connection.md @@ -126,5 +126,5 @@ occupation. - [[stellar-to-halo-mass]] - [[halo-occupation]] -- [[sources-stellar-halos]] +- [[seed-stellar-halos]] - [[sources-mass-models]] diff --git a/wiki/galaxies/sources/stellar-halos.md b/wiki/galaxies/seed/stellar-halos.md similarity index 99% rename from wiki/galaxies/sources/stellar-halos.md rename to wiki/galaxies/seed/stellar-halos.md index 34c1549..b690766 100644 --- a/wiki/galaxies/sources/stellar-halos.md +++ b/wiki/galaxies/seed/stellar-halos.md @@ -208,4 +208,4 @@ to the bulge-halo decompositions in PyAutoLens. - [[stellar-halos]] - [[halo-mass-proxies]] - [[low-surface-brightness]] -- [[sources-halo-galaxy-connection]] +- [[seed-halo-galaxy-connection]] diff --git a/wiki/galaxies/sources/light-profile-fitting.md b/wiki/galaxies/sources/light-profile-fitting.md index 36f5fcd..9ad2c5c 100644 --- a/wiki/galaxies/sources/light-profile-fitting.md +++ b/wiki/galaxies/sources/light-profile-fitting.md @@ -369,5 +369,5 @@ methodology and applications. - [[sersic-profile]] - [[mge]] - [[bulge-disk-decomposition-galaxies]] -- [[sources-bulge-disk-decomposition]] +- [[seed-bulge-disk-decomposition]] - [[sources-elliptical-galaxies]] diff --git a/wiki/lensing/index.md b/wiki/lensing/index.md index 67f3e61..0b1352b 100755 --- a/wiki/lensing/index.md +++ b/wiki/lensing/index.md @@ -109,7 +109,11 @@ live in [`../../bibliography/`](../../bibliography/README.md). - [[sources-gr-cosmology]] - [[sources-specific-lenses]] - [[sources-galaxy-formation]] -- [[sources-unclassified]] +- **Seed (unverified imports):** [[seed-unclassified]] — filename-inferred + stubs from the 2026-05 PDF import, every entry still reading `Canonical + BibTeX key: TODO — no unique match found`. They live in `seed/`, not + `sources/`, and **nothing in them may be cited** until the paper itself is + verified and the entry promoted into the matching `sources/` page. ## Meta diff --git a/wiki/lensing/log.md b/wiki/lensing/log.md index 4a59946..67cd2eb 100755 --- a/wiki/lensing/log.md +++ b/wiki/lensing/log.md @@ -215,3 +215,22 @@ PyAutoMemory#39). forward-modelling pipeline (arXiv:2503.22657) was added to the bibliography and `sources/lens-modeling-methods.md` during the #34/#36 wiki-hygiene pass but never recorded here. No content change; this line closes the log gap. + +--- + +## 2026-09-17 — Unverified import seed split out of `sources/` + +**By:** Claude (Opus 5, PyAutoMemory seed-split task). + +**What changed:** 1 page (5 entries, 97 lines) moved from `sources/` to `seed/`: `unclassified`. Every +entry on them is a filename-inferred stub from the 2026-05 PDF import whose +canonical key still reads `TODO — no unique match found`, so none of it is +citeable and none of it was ever claim-verified. `sources/` now holds only +pages with at least some verified support; `seed/` holds the raw import +waiting for it. + +**Nothing was deleted or rewritten** — the pages moved, their `[[sources-…]]` +references became `[[seed-…]]`, and each index gained one *Seed (unverified +imports)* line saying what they are. Promoting an entry means verifying the +paper, adding its canonical key to `../../bibliography/pyautomemory.bib`, and +moving the section into the matching `sources/` page. diff --git a/wiki/lensing/sources/unclassified.md b/wiki/lensing/seed/unclassified.md similarity index 100% rename from wiki/lensing/sources/unclassified.md rename to wiki/lensing/seed/unclassified.md diff --git a/wiki/lensing/sources/reviews-and-summaries.md b/wiki/lensing/sources/reviews-and-summaries.md index 4e26b80..095f443 100644 --- a/wiki/lensing/sources/reviews-and-summaries.md +++ b/wiki/lensing/sources/reviews-and-summaries.md @@ -175,6 +175,6 @@ introductions or for cross-referencing established field-knowledge. ## See also -- [[sources-galaxy-formation-misc]] +- [[seed-galaxy-formation-misc]] - [[sources-galaxy-formation]] (existing field-galaxy-formation bibliography) diff --git a/wiki/methods/CLAUDE.md b/wiki/methods/CLAUDE.md index 7efda6c..b76cde9 100644 --- a/wiki/methods/CLAUDE.md +++ b/wiki/methods/CLAUDE.md @@ -17,7 +17,8 @@ methods/ ├── log.md # append-only compilation log ├── concepts/ # Bayes basics, samplers, deep learning, etc. ├── entities/ # PyAutoFit, PolyChord, BlackJAX, jax-finufft, … -└── sources/ # per-topic claim support +├── sources/ # per-topic claim support +└── seed/ # unverified 2026-05 import — never cite from here ``` ## Schema diff --git a/wiki/methods/index.md b/wiki/methods/index.md index df5e319..0c8a8d3 100644 --- a/wiki/methods/index.md +++ b/wiki/methods/index.md @@ -87,11 +87,16 @@ live in [`../../bibliography/`](../../bibliography/README.md). - [[sources-samplers]] - [[sources-likelihood-free-inference]] - [[sources-probabilistic-programming]] -- [[sources-deep-learning-methods]] - [[sources-linear-algebra]] - [[sources-nufft]] - [[sources-scientific-software]] - [[sources-simulations]] +- **Seed (unverified imports):** [[seed-deep-learning-methods]], + [[seed-medical-imaging-adjacents]] — filename-inferred stubs from the 2026-05 + PDF import, every entry still reading `Canonical BibTeX key: TODO — no unique + match found`. They live in `seed/`, not `sources/`, and **nothing in them may + be cited** until the paper itself is verified and the entry promoted into the + matching `sources/` page. ## Meta diff --git a/wiki/methods/log.md b/wiki/methods/log.md index 984fadb..f9d0e84 100644 --- a/wiki/methods/log.md +++ b/wiki/methods/log.md @@ -104,3 +104,22 @@ three crash-level bugs found and fixed (from_dict 0.0-drop, AggregateFITS fd leak at ~500 results, database slicing inversion). Durable methodology lessons (fresh-Aggregator staging, idle-machine grids, interleaved A/B under load) recorded. Cross-linked from index.md (Software ecosystem). + +--- + +## 2026-09-17 — Unverified import seed split out of `sources/` + +**By:** Claude (Opus 5, PyAutoMemory seed-split task). + +**What changed:** 2 pages (8 entries, 163 lines) moved from `sources/` to `seed/`: `deep-learning-methods`, `medical-imaging-adjacents`. Every +entry on them is a filename-inferred stub from the 2026-05 PDF import whose +canonical key still reads `TODO — no unique match found`, so none of it is +citeable and none of it was ever claim-verified. `sources/` now holds only +pages with at least some verified support; `seed/` holds the raw import +waiting for it. + +**Nothing was deleted or rewritten** — the pages moved, their `[[sources-…]]` +references became `[[seed-…]]`, and each index gained one *Seed (unverified +imports)* line saying what they are. Promoting an entry means verifying the +paper, adding its canonical key to `../../bibliography/pyautomemory.bib`, and +moving the section into the matching `sources/` page. diff --git a/wiki/methods/sources/deep-learning-methods.md b/wiki/methods/seed/deep-learning-methods.md similarity index 100% rename from wiki/methods/sources/deep-learning-methods.md rename to wiki/methods/seed/deep-learning-methods.md diff --git a/wiki/methods/sources/medical-imaging-adjacents.md b/wiki/methods/seed/medical-imaging-adjacents.md similarity index 100% rename from wiki/methods/sources/medical-imaging-adjacents.md rename to wiki/methods/seed/medical-imaging-adjacents.md diff --git a/wiki/methods/sources/likelihood-free-inference.md b/wiki/methods/sources/likelihood-free-inference.md index 8d47292..308f7a2 100644 --- a/wiki/methods/sources/likelihood-free-inference.md +++ b/wiki/methods/sources/likelihood-free-inference.md @@ -45,5 +45,5 @@ status: stub - [[simulation-based-inference]] - [[neural-posterior-estimation]] -- [[sources-deep-learning-methods]] +- [[seed-deep-learning-methods]] - [[sources-deep-learning-lensing]]