Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .env.template
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,17 @@ PRISM_ASSETS="/flare/<project>/<your_username>/assets"
# the Stage 1 projector named by src/conf/training/bioreason_sft.yaml.
PRISM_OUTPUT_ROOT="/flare/<project>/<your_username>/outputs"

# Shell-script variables (optional)
# Read by the bioreason / GRPO submit scripts and scripts/smoke_test_grpo.sh,
# which fail with a "set <NAME>" message when they are unset. These are plain
# shell variables, not part of src/site_paths.py.

# Conda environment those scripts activate.
# PRISM_CONDA_ENV="/<filesystem>/<project>/<your_username>/conda_env/prism"

# Checkout the scripts cd into (scripts/smoke_test_grpo.sh).
# PRISM_REPO_ROOT="/flare/<project>/<your_username>/PRISM"

# Model & Dataset Path Overrides (optional)
# These override placeholder defaults baked into src/config.py and
# src/encoders/geometry.py. Leave unset to use the placeholder paths — this
Expand Down
5 changes: 3 additions & 2 deletions .github/pull_request_template.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,8 +64,9 @@ See *Guidelines for AI/LLM-Assisted Contributions* in
accountable for the review — a project norm, not yet enforced by branch
protection.

- [ ] AI/LLM tools were primarily used to generate code or artifacts in this PR
- [ ] I have reviewed all such output line-by-line and understand it
- **AI tools used:** None, or list the tools.
- **Scope of assistance:** What was generated, edited, or reviewed?
- **Human validation:** What did you review or test?

## Additional Context

Expand Down
37 changes: 37 additions & 0 deletions .github/workflows/pr_quality.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,12 @@ on:
push:
branches:
- main
workflow_dispatch:
inputs:
check_external_urls:
description: Check external documentation URLs (advisory only)
type: boolean
default: false

permissions:
contents: read
Expand Down Expand Up @@ -60,6 +66,37 @@ jobs:
python-version: "3.10"
- name: Check relative Markdown links
run: python tools/ci/check_doc_links.py
- name: List tracked Markdown files
id: markdown
run: |
git ls-files '*.md' > "$RUNNER_TEMP/prism-markdown.txt"
echo "files=$(paste -sd ' ' "$RUNNER_TEMP/prism-markdown.txt")" >> "$GITHUB_OUTPUT"
- name: Check spelling
uses: crate-ci/typos@4c8b7f51ca60df14e3c5ca82770dfbbe6c3dae85 # v1.52.0
with:
files: ${{ steps.markdown.outputs.files }}
- name: Check local Markdown links and anchors
uses: lycheeverse/lychee-action@e7477775783ea5526144ba13e8db5eec57747ce8 # v2
with:
args: --offline --include-fragments=anchor-only --no-progress --files-from ${{ runner.temp }}/prism-markdown.txt

external_urls:
if: github.event_name == 'workflow_dispatch' && inputs.check_external_urls
runs-on: ubuntu-latest
timeout-minutes: 15
continue-on-error: true
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
persist-credentials: false
- name: List tracked Markdown files
run: git ls-files '*.md' > "$RUNNER_TEMP/prism-markdown.txt"
- name: Check external documentation URLs
continue-on-error: true
uses: lycheeverse/lychee-action@e7477775783ea5526144ba13e8db5eec57747ce8 # v2
with:
fail: false
args: --exclude-all-private --timeout 20 --max-retries 2 --no-progress --files-from ${{ runner.temp }}/prism-markdown.txt

types:
runs-on: ubuntu-latest
Expand Down
9 changes: 9 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,12 @@ will make up v0.1.0.

### Changed

- Site-path gate now also flags internal identities (usernames, the old repo
name) and scans `experiments/`, `scaling-study/`,
`requirements/`, `.github/` and root-level files, and the `grand`, `gpfs`
and `lustre` roots; the bioreason/GRPO scripts and `experiments/prism_designs.yaml`
take locations from `PRISM_*` variables (`PRISM_CONDA_ENV` and `PRISM_REPO_ROOT`
are new, see `.env.template`)
- Corrected the `src/libs/walrus` submodule URL to the upstream
`PolymathicAI/walrus` organization (#186)
- `requirements/base.txt` now installs `h5py`, which `src/data/multimodal.py`
Expand All @@ -51,6 +57,9 @@ will make up v0.1.0.

### Removed

- `experiments/stage_a_round1_olmo3_1b.yaml`, `ablation_catalog.yaml` and
`qwen3_siglip_smokes.yaml` — study records tied to internal paths with no
code dependents. Remaining study files carry `TODO(release)` notes.
- `docs/plans/` (9 files) and `docs/historical/` (5 files) — dated design plans
and bring-up debug journals whose work has all landed. Plans are tracking
artifacts and belong in the issue tracker rather than the source tree.
Expand Down
9 changes: 9 additions & 0 deletions _typos.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
[default.extend-identifiers]
unparseable_response = "unparseable_response"
nature_comunication = "nature_comunication"
hsa_var = "hsa_var"
ABL = "ABL"
ABL1 = "ABL1"
arange = "arange"
HPE = "HPE"
PNGs = "PNGs"
14 changes: 7 additions & 7 deletions applications/bioreason/scripts/eval/submit_eval_kegg.sh
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,8 @@ PROJ_DIR="${PROJ_DIR:-$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../.." 2>/dev/n
if [ -n "${PBS_O_WORKDIR:-}" ] && [ ! -f "${PROJ_DIR}/src/train.py" ]; then
PROJ_DIR="${PBS_O_WORKDIR}"
fi
CHECKPOINT_DIR="/lus/eagle/projects/argonne_tpc/abalaji/modcon/genome/output/GENOME-SFT-1B/2026-06-29/07-41-10/checkpoints/step_8000"
# SFT with sample weigting and answer structure <answer>: /lus/eagle/projects/argonne_tpc/abalaji/modcon/genome/output/GENOME-SFT-1B/2026-06-29/07-41-10/checkpoints/step_3000
CHECKPOINT_DIR="${PRISM_OUTPUT_ROOT:?set PRISM_OUTPUT_ROOT}/GENOME-SFT-1B/<date>/<time>/checkpoints/step_8000"
# SFT with sample weigting and answer structure <answer>: ${PRISM_OUTPUT_ROOT:?set PRISM_OUTPUT_ROOT}/GENOME-SFT-1B/<date>/<time>/checkpoints/step_3000
# Set to "true" to run interleaved mode, "false" for prefix mode.
# Must be "true" for checkpoints trained with is_interleaved_qa=True (SFT/GRPO stages).
INTERLEAVE_QA="true"
Expand All @@ -34,7 +34,7 @@ ml --ignore_cache load cudatoolkit-standalone/12.6.1

# --- Conda env ---
ml --ignore_cache load conda
conda activate /lus/eagle/projects/argonne_tpc/abalaji/conda_env/prism
conda activate ${PRISM_CONDA_ENV:?set PRISM_CONDA_ENV}

# --- Proxy ---
export HTTP_PROXY="http://proxy.alcf.anl.gov:3128"
Expand All @@ -44,10 +44,10 @@ export https_proxy="http://proxy.alcf.anl.gov:3128"
export no_proxy="127.0.0.1,admin,polaris-adminvm-01,localhost,*.cm.polaris.alcf.anl.gov,polaris-*,*.polaris.alcf.anl.gov,*.alcf.anl.gov"

# --- HuggingFace cache ---
export HF_HOME="/lus/eagle/projects/argonne_tpc/abalaji/model_weights"
export HF_DATASETS_CACHE="/lus/eagle/projects/argonne_tpc/abalaji/model_weights"
export TRANSFORMERS_CACHE="/lus/eagle/projects/argonne_tpc/abalaji/model_weights/hub"
export HF_HUB_CACHE="/lus/eagle/projects/argonne_tpc/abalaji/model_weights/hub"
export HF_HOME="$(dirname "${PRISM_HF_HUB:?set PRISM_HF_HUB}")"
export HF_DATASETS_CACHE="$(dirname "${PRISM_HF_HUB:?set PRISM_HF_HUB}")"
export TRANSFORMERS_CACHE="${PRISM_HF_HUB:?set PRISM_HF_HUB}"
export HF_HUB_CACHE="${PRISM_HF_HUB:?set PRISM_HF_HUB}"

# --- General ---
export PYTHONUNBUFFERED=1
Expand Down
12 changes: 6 additions & 6 deletions applications/bioreason/scripts/test/verify_dna_dataset_mix.sh
Original file line number Diff line number Diff line change
Expand Up @@ -28,15 +28,15 @@ PROJ_DIR="${PROJ_DIR:-$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../.." 2>/dev/n
if [ -n "${PBS_O_WORKDIR:-}" ] && [ ! -f "${PROJ_DIR}/src/train.py" ]; then
PROJ_DIR="${PBS_O_WORKDIR}"
fi
STAGE1_CKPT="/lus/eagle/projects/argonne_tpc/abalaji/modcon/genome/output/GENOME-SFT-1B/2026-06-29/07-41-10/checkpoints/step_8000"
STAGE1_CKPT="${PRISM_OUTPUT_ROOT:?set PRISM_OUTPUT_ROOT}/GENOME-SFT-1B/<date>/<time>/checkpoints/step_8000"

cd "${PROJ_DIR}"

# --- Modules ---
ml use /soft/modulefiles
ml --ignore_cache load cudatoolkit-standalone/12.6.1
module --ignore_cache load conda
conda activate /lus/eagle/projects/argonne_tpc/abalaji/conda_env/prism
conda activate ${PRISM_CONDA_ENV:?set PRISM_CONDA_ENV}

# --- Proxy ---
export HTTP_PROXY="http://proxy.alcf.anl.gov:3128"
Expand All @@ -46,10 +46,10 @@ export https_proxy="http://proxy.alcf.anl.gov:3128"
export no_proxy="127.0.0.1,admin,polaris-adminvm-01,localhost,*.cm.polaris.alcf.anl.gov,polaris-*,*.polaris.alcf.anl.gov,*.alcf.anl.gov"

# --- HuggingFace cache ---
export HF_HOME="/lus/eagle/projects/argonne_tpc/abalaji/model_weights"
export HF_DATASETS_CACHE="/lus/eagle/projects/argonne_tpc/abalaji/model_weights"
export TRANSFORMERS_CACHE="/lus/eagle/projects/argonne_tpc/abalaji/model_weights/hub"
export HF_HUB_CACHE="/lus/eagle/projects/argonne_tpc/abalaji/model_weights/hub"
export HF_HOME="$(dirname "${PRISM_HF_HUB:?set PRISM_HF_HUB}")"
export HF_DATASETS_CACHE="$(dirname "${PRISM_HF_HUB:?set PRISM_HF_HUB}")"
export TRANSFORMERS_CACHE="${PRISM_HF_HUB:?set PRISM_HF_HUB}"
export HF_HUB_CACHE="${PRISM_HF_HUB:?set PRISM_HF_HUB}"

# --- W&B offline (no network needed) ---
export WANDB_MODE=offline
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,18 +16,18 @@ PROJ_DIR="${PROJ_DIR:-$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../.." 2>/dev/n
if [ -n "${PBS_O_WORKDIR:-}" ] && [ ! -f "${PROJ_DIR}/src/train.py" ]; then
PROJ_DIR="${PBS_O_WORKDIR}"
fi
OUTPUT_BASE="/lus/eagle/projects/argonne_tpc/abalaji/modcon/genome/output"
OUTPUT_BASE="${PRISM_OUTPUT_ROOT:?set PRISM_OUTPUT_ROOT}"
RUN_ID="GENOME-GRPO"
RUN_VARIANT="1B"
STAGE1_CKPT="/lus/eagle/projects/argonne_tpc/abalaji/modcon/genome/output/GENOME-SFT-1B/2026-06-29/07-41-10/checkpoints/step_8000"
STAGE1_CKPT="${PRISM_OUTPUT_ROOT:?set PRISM_OUTPUT_ROOT}/GENOME-SFT-1B/<date>/<time>/checkpoints/step_8000"

cd "${PROJ_DIR}"

# --- Modules ---
ml use /soft/modulefiles
ml --ignore_cache load cudatoolkit-standalone/12.6.1
module --ignore_cache load conda
conda activate /lus/eagle/projects/argonne_tpc/abalaji/conda_env/prism
conda activate ${PRISM_CONDA_ENV:?set PRISM_CONDA_ENV}

# --- Proxy ---
export HTTP_PROXY="http://proxy.alcf.anl.gov:3128"
Expand All @@ -37,10 +37,10 @@ export https_proxy="http://proxy.alcf.anl.gov:3128"
export no_proxy="127.0.0.1,admin,polaris-adminvm-01,localhost,*.cm.polaris.alcf.anl.gov,polaris-*,*.polaris.alcf.anl.gov,*.alcf.anl.gov"

# --- HuggingFace cache ---
export HF_HOME="/lus/eagle/projects/argonne_tpc/abalaji/model_weights"
export HF_DATASETS_CACHE="/lus/eagle/projects/argonne_tpc/abalaji/model_weights"
export TRANSFORMERS_CACHE="/lus/eagle/projects/argonne_tpc/abalaji/model_weights/hub"
export HF_HUB_CACHE="/lus/eagle/projects/argonne_tpc/abalaji/model_weights/hub"
export HF_HOME="$(dirname "${PRISM_HF_HUB:?set PRISM_HF_HUB}")"
export HF_DATASETS_CACHE="$(dirname "${PRISM_HF_HUB:?set PRISM_HF_HUB}")"
export TRANSFORMERS_CACHE="${PRISM_HF_HUB:?set PRISM_HF_HUB}"
export HF_HUB_CACHE="${PRISM_HF_HUB:?set PRISM_HF_HUB}"

# --- W&B ---
# No WANDB_API_KEY here -- a plaintext key was committed here previously
Expand Down
2 changes: 1 addition & 1 deletion docs/applications/bioreason_grpo.md
Original file line number Diff line number Diff line change
Expand Up @@ -706,4 +706,4 @@ particularly important for free-form pathway descriptions where lexical matching

---

*Generated from codebase analysis of `BaseMM_PRISM` as of 2026-06-26.*
*Generated from codebase analysis of `PRISM` as of 2026-06-26.*
12 changes: 6 additions & 6 deletions docs/applications/kegg_curation_implementation_status.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@ ground truth for "what was planned and why."

All code lives in `src/data/kegg_curation/`. All pipeline **output data**
(checkpoints + downloaded genome) lives outside the repo, under
`$KEGG_CURATION_DATA_ROOT` (env var, defaults to
`/lus/eagle/projects/argonne_tpc/abalaji/modcon/dataset/kegg_curation`) —
`$KEGG_CURATION_DATA_ROOT` (env var; the code default is a placeholder, so set it, e.g.
`${PRISM_DATA_ROOT}/kegg_curation`) —
never committed to git.

**Update (2026-07-30)**: the full 10-stage pipeline now runs end-to-end
Expand Down Expand Up @@ -842,13 +842,13 @@ the deferred regex-vs-LLM comparison for stage 03's fallback path.
## Environment / How to Run the Full Pipeline

```bash
export PATH=/lus/eagle/projects/argonne_tpc/abalaji/conda_env/prism/bin:$PATH
export PATH=${PRISM_CONDA_ENV}/bin:$PATH
export http_proxy=http://proxy.alcf.anl.gov:3128 # needed on ALCF compute nodes
export https_proxy=http://proxy.alcf.anl.gov:3128
export KEGG_CURATION_DATA_ROOT=/lus/eagle/projects/argonne_tpc/abalaji/modcon/dataset/kegg_curation
export KEGG_CURATION_DATA_ROOT=${PRISM_DATA_ROOT}/kegg_curation
export NCBI_ENTREZ_EMAIL=your-email@example.com # required for stages 04, 07
# export ANTHROPIC_API_KEY=... # only if reasoning_backend="claude"
cd /home/abalaji/projects/modcon/genome/BaseMM_PRISM
cd ${PRISM_REPO_ROOT}

python -m src.data.kegg_curation.01_fetch_kegg_networks
python -m src.data.kegg_curation.02_extract_variant_ids
Expand All @@ -863,7 +863,7 @@ python -m src.data.kegg_curation.10_assemble_dataset
```

Dependencies (already installed in the `prism` conda env at
`/lus/eagle/projects/argonne_tpc/abalaji/conda_env/prism`):
`${PRISM_CONDA_ENV}`):
`biopython`, `anthropic` (both added to `requirements.txt`; `kegg_pull`
was added then removed once found to be broken — see stage 01/`kegg_rest.py`
notes above).
Expand Down
6 changes: 3 additions & 3 deletions docs/applications/kegg_curation_pipeline.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
# KEGG Dataset Curation Pipeline — Design Doc

This document plans a from-scratch recreation, inside `BaseMM_PRISM`, of the
This document plans a from-scratch recreation, inside `PRISM`, of the
pipeline BioReason used to build the `wanglab/kegg` biological-reasoning
dataset (1,159 train rows: `question`, `answer`, `reasoning`,
`reference_sequence`, `variant_sequence`). `BaseMM_PRISM` currently only
`reference_sequence`, `variant_sequence`). `PRISM` currently only
*consumes* the published dataset (`src/data/multimodal.py`'s
`_process_dna_bioreason`, mirroring `BioReason/bioreason/dataset/kegg.py`'s
`_format_kegg`) — there is no curation/construction code anywhere in this
Expand Down Expand Up @@ -237,7 +237,7 @@ precedents to follow).
chromosome subsetting purely in Python via `Bio.SeqIO` (slower for a
~3GB genome FASTA but no extra system dependency)?
- **Target scale for a first run**: the original processed ~1,449 variants
end-to-end (Claude cost driver). Should the first `BaseMM_PRISM` run
end-to-end (Claude cost driver). Should the first `PRISM` run
target a similar scale, or start with a small smoke-scale subset (e.g.
20-50 variants) to validate the full pipeline mechanically before
spending on a larger Claude batch run?
Expand Down
23 changes: 23 additions & 0 deletions docs/development/ci_multiplatform.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,29 @@ out through the UI. Change the matrix and this list together.
- `launcher_contract`
- `coverage_gate`

## Documentation checks

The `doc_links` job checks every tracked Markdown file for relative links,
spelling (typos v1.52.0), and local links including heading anchors (lychee
v0.24.2). `_typos.toml` allows specific technical identifiers without renaming
APIs, configuration keys, or data paths.

With those checker binaries on `PATH`, run from the repository root:

```bash
python tools/ci/check_doc_links.py
git ls-files '*.md' | typos --file-list -
git ls-files '*.md' | lychee --offline --include-fragments=anchor-only --no-progress --files-from -
```

External URLs are not checked on pushes or pull requests. To run an advisory
check, select **Actions → CI → Run workflow** and enable **Check external
documentation URLs**. This input defaults to off. The separate `external_urls`
job is non-blocking, including checker or job failures, and must not be made a
required branch-protection check. HTTP rate limits and access restrictions
need human interpretation; a failed request does not necessarily mean a
broken URL.

## Test marker taxonomy

- `unit`: fast, local-only checks
Expand Down
29 changes: 0 additions & 29 deletions docs/modalities/timeseries.md
Original file line number Diff line number Diff line change
Expand Up @@ -261,32 +261,3 @@ Example command to verify a checkpoint on the first 50 validation samples:
```
export PYTHONNOUSERSITE=1 && unset PYTHONPATH && PRISM_VAL_SHARDS_DIR=/flare/ModCon/pemami/data/SciTS-processed/val_shards python tools/universal_evaluator.py --checkpoint outputs/PRISM-OLMO-1B-TIMEOMNI-TEST/2026-07-30/06-56-32/checkpoints/step_5000/model.safetensors --mode verify_timeseries_scits --validation --limit 50 --backbone allenai/OLMo-1B-0724-hf
```

### Local TS_QA Data Analysis

Use [scripts/perlmutter/analyze_ts_data.py](scripts/perlmutter/analyze_ts_data.py) to scan all JSONL files under a TS_QA root folder and report, per file:
- Longest time series length observed.
- Min/max number of `<ts></ts>` placeholders per sequence.

Example:

```bash
python3 scripts/perlmutter/analyze_ts_data.py \
--root /flare/ModCon/ngetty/data/zone_a/ts_qa
```

Optional JSON output:

```bash
python3 scripts/perlmutter/analyze_ts_data.py \
--root /flare/ModCon/ngetty/data/zone_a/ts_qa \
--output-json outputs/ts_qa_stats.json
```

Output columns:
- `file`: JSONL path
- `records`: number of non-empty lines parsed as records
- `bad_json`: malformed JSON lines
- `max_ts_len`: longest per-series length found in that file
- `min_ts_tags` / `max_ts_tags`: min/max `<ts></ts>` count per sequence in text fields

9 changes: 9 additions & 0 deletions docs/platforms/site_paths.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,15 @@ documented in [`.env.template`](../../.env.template) but are **not** part of
`SITE_VARIABLES`, so they get no `<unset:NAME>` placeholder and no
`require_resolved()` error — an unset one fails wherever it is used.

Two plain shell variables are read only by the bioreason and GRPO scripts
(`applications/bioreason/scripts/`, `scripts/smoke_test_grpo.sh`):
`PRISM_CONDA_ENV` (the conda environment to activate) and `PRISM_REPO_ROOT`
(the checkout to `cd` into, smoke test only). They are documented in
`.env.template`, are not resolved by `src/site_paths.py`, and the scripts exit
with a `set <NAME>` message when one is unset. The same scripts take their
checkpoint, output and Hugging Face cache locations from `PRISM_OUTPUT_ROOT`
and `PRISM_HF_HUB`.

## Where the values come from

Highest precedence first:
Expand Down
2 changes: 1 addition & 1 deletion docs/skills/prism-multi-agent-workflow/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ installed **`multi-agent-git-hygiene`** skill; this is the PRISM-specific drill.
- Call `EnterWorktree` at the start of any session that edits files or git state
→ creates `.claude/worktrees/<name>` on a fresh branch off `origin/main`.
- Spawning editing subagents? Pass `isolation: "worktree"` to the `Agent` tool.
- The bare clone (`/lus/flare/projects/ModCon/ngetty/BaseMM_PRISM`) is
- The bare clone (`<path-to-your-clone>`) is
**read-only** — never commit or branch-switch there.

## Mandatory: verify branch before you claim it
Expand Down
3 changes: 3 additions & 0 deletions experiments/LLM_timeseries_scaling.yaml
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# TODO(release): study record, kept for now because applications/timeseries/launch_capacity_intern_s2_397b.sh reads it. Decide before
# v0.1.0: keep and scrub, or remove with the other scaling-study files.
# Known leaks to resolve: that launcher script and its internal site paths (allowlisted).
# Experiment configuration for Qwen3/OLMo + TS encoder scaling studies
experiments:

Expand Down
Loading
Loading