Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# Data and configs
*.toml
!examples/slac_fel/slac-fel-tutorial.toml
!examples/slac_fel/test_tomls/dec2025.toml

# Logging files
*.db
Expand All @@ -19,6 +21,7 @@ data/*

# Files
*.png
!examples/slac_fel/figures/focused_drift_adaptation.png
!docs/images/*.png
*.txt
!docs/requirements.txt
Expand Down
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]

### Added
- Reproducible SLAC-FEL scientific example with a focused December 2025 drift-adaptation tutorial, pretrained model and scalers, paper-style replay figure, full-month configuration, documentation, and tests
- Community health files: `CONTRIBUTING.md` (including guidelines for AI/LLM-assisted
contributions), `CODE_OF_CONDUCT.md`, `SECURITY.md`, and this changelog
- GitHub issue templates for bug reports and feature requests
Expand Down Expand Up @@ -44,6 +45,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
`ruff format` — and it carried three advisories.

### Fixed
- The configured experiment seed is now applied at startup to Python, NumPy, Torch, and CUDA, enabling deterministic scientific metrics and adapted checkpoints for repeat runs
- `WandBLogger.url` called `Run.get_url()`, which wandb removed in the upgrade to 0.30; it
raised `AttributeError`. It now reads the `Run.url` property, which exists across the whole
supported range (`wandb>=0.22.0`).
Expand Down
7 changes: 4 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,7 @@ From the project root:
uv run python -m src.main --config examples/mnist/mnist.toml
uv run python -m src.main --config examples/cifar/cifar10_vit.toml
uv run python -m src.main --config examples/imagenet/imagenet_vit.toml # requires ImageNet data at data.path
uv run python -m src.main --config examples/slac_fel/slac-fel-tutorial.toml --set data.path=/path/to/fel-tutorial-data
```

## Metrics Logging
Expand All @@ -83,7 +84,7 @@ uv run python -m src.main --config examples/mnist/mnist.toml --set logging.backe
# To view results for MLflow, run `mlflow ui` in another terminal and navigate to http://localhost:5000
```

Currently the mnist example sets the logging to wandb in the toml `config` file. The other examples do not set any metric for the logging backend, which defaults to wandb.
Logging is configured in the TOML `[logging]` section. The default backend is `"none"`; examples may opt into WandB or MLflow and may write long-format metrics to a local CSV file.

## Configuration Overview

Expand All @@ -94,7 +95,7 @@ Primary sections in config TOML:
- `[train]`
- `[drift_detection]`
- `[continual_learning]` (optional but recommended)
- `[visualization]` (optional)
- `[logging]` (optional; experiment tracking and CSV output)

Top-level fields commonly used:

Expand Down Expand Up @@ -200,7 +201,7 @@ See [`docs/architecture.md`](docs/architecture.md) for the full pipeline, includ
- Update batch size and worker count with `[train] batch_size` and `[train] num_workers`.

## Output
The console logger reports per-stage metrics (`eval`, `drift`, `cl`). When `[visualization] input` is set, the run also writes a metrics CSV at that path for external plotting.
The console logger reports per-stage metrics (`eval`, `drift`, `cl`). Set `[logging] metrics_output_path` to write the same long-format metrics to a CSV file for external analysis and plotting.

## Deployment

Expand Down
34 changes: 15 additions & 19 deletions examples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,11 @@ uv run python -m src.main --config <path_to_toml>
| [`mnist/`](mnist/README.md) | `mnist` | 3-layer CNN (`Cnn`, ~1M params) | Simulated: cumulative random affine per stream window | Auto-downloaded to `./data` | Yes — CPU is fine |
| [`cifar/`](cifar/README.md) | `cifar10` | ViT-B/16 or VGG-11 (`VisionModelCifar`) | Simulated: random affine per stream window | Auto-downloaded to `./data` | GPU strongly recommended |
| [`imagenet/`](imagenet/README.md) | `imagenet` | ViT-B/16 (`VisionModelImageNet`) | Simulated: cumulative random affine per stream window | **You provide** ILSVRC-2012 in `ImageFolder` layout | No — multi-GPU scale |
| [`slac_fel/`](slac_fel/README.md) | `slac-fel` | Pretrained fully connected surrogate | Measured: chronological accelerator operating regimes | **You provide** the focused 14-file FEL subset | Yes — CPU is fine |

**Start with `mnist/`.** It is the only example that ships a pretrained
checkpoint, downloads its own data, and finishes in minutes on CPU.
**Start with `mnist/`** for the simplest self-contained workflow. It
downloads its own data and finishes in minutes on CPU. Use `slac_fel/`
for a measured scientific stream after obtaining the focused dataset.

## How an Example Is Wired Up

Expand All @@ -39,22 +41,16 @@ An example becomes runnable through three pieces:
To add your own, mirror the MNIST layout and add a branch to `get_example()`.
See [`docs/model_harness.md`](../docs/model_harness.md) for the full contract.

## How Drift Is Simulated
## Where Drift Comes From

None of these datasets drift on their own, so each harness manufactures drift the
same way: `update_data_stream()` draws a seeded random affine transform
(rotation / scale / shear / translation) and rebuilds the train, validation, and
stream loaders through it. Every time the stream is exhausted, another transform
is drawn, so the input distribution keeps moving away from what the model was
trained on and monitored accuracy degrades.
The computer-vision harnesses manufacture drift with seeded random affine
transformations such as rotation, scale, shear, and translation. MNIST and
ImageNet accumulate those transformations over time, while CIFAR applies a
fresh perturbation to each stream window.

MNIST and ImageNet **compose** the whole history of transforms (drift accumulates
and gets progressively worse); CIFAR applies only the most recent one (each
window is a fresh, independent perturbation). Historical replay loaders follow
the same convention — MNIST replays *every* prior regime, CIFAR replays only the
immediately preceding one.

The number of windows is capped by `[drift_detection] max_stream_updates`.
The SLAC-FEL example instead consumes chronological accelerator observations.
Its drift signals arise from changes in the measured surrogate error, not from
synthetic input transformations.

## What Every Run Produces

Expand All @@ -63,7 +59,7 @@ The number of windows is capped by `[drift_detection] max_stream_updates`.
confidence, then a `CL Updates (drift_event_id=N)` bar and before/after test
accuracy on both the current and historical distributions.
- **CSV** — long-format `step,metric,value` at the path in
`[visualization] input` (e.g. `output/mnist.csv`). Metric names are stage
`[logging] metrics_output_path` (e.g. `output/mnist.csv`). Metric names are stage
prefixed: `eval/*` (per-batch accuracy and loss, plus `test_curr_acc` and
`test_hist_acc` after each CL round), `drift/*` (score, regime, confidence,
the monitored metric), `cl/*` (per-iteration losses and FLOP/time counters).
Expand All @@ -72,8 +68,8 @@ The number of windows is capped by `[drift_detection] max_stream_updates`.
console + CSV. See [`docs/tracking.md`](../docs/tracking.md) for backend
setup, the full metric namespace, and annotated screenshots of a run.

The package does not bundle a plotting step: `[visualization] input` is written
for you to chart externally.
The package does not bundle a common plotting step. Long-format CSV metrics are
written to `[logging] metrics_output_path` for external analysis and plotting.

## Common Overrides

Expand Down
232 changes: 232 additions & 0 deletions examples/slac_fel/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,232 @@
# SLAC-FEL Scientific Example

This example applies Apeiron to measured data from the Linac Coherent Light
Source free-electron laser (LCLS FEL) at SLAC. It demonstrates how to monitor a
pretrained scientific surrogate, detect a change in its prediction error, and
adapt the model to the changed operating regime.

Unlike the computer-vision examples, this workflow does not manufacture drift
with image transformations. The stream consists of chronological accelerator
observations.

## What the Model Predicts

Each observation contains:

- 341 accelerator state and control readbacks used as model inputs.
- One measured FEL pulse-energy target,
`GDET:FEE1:241:ENRC`, expressed in millijoules (mJ).
- A timestamp that preserves the experimental ordering.

The pretrained neural-network surrogate maps the 341 accelerator variables to
the measured pulse energy. As accelerator conditions change, the relationship
learned by the surrogate may become less accurate.

Apeiron monitors the model's mean absolute error (MAE). The focused tutorial
uses Page-Hinkley to identify a large error transition and then performs a
short online-EWC adaptation.

## Tutorial Scope

The focused tutorial uses 14 chronological files:

```text
data_730.pkl
data_731.pkl
...
data_743.pkl
```

Together they contain 70,000 observations from December 2 through December 4,
2025. The major transition begins in `data_740.pkl`.

The focused subset was selected because it provides:

- a stable pre-transition period;
- one clear detector event;
- a substantial loss of accuracy under the pretrained model;
- measurable recovery after adaptation;
- a short CPU runtime;
- deterministic results under seed 42.

The full December paper-reproduction configuration remains available at
`test_tomls/dec2025.toml`, but it is not required for this tutorial.

## Repository Contents

```text
examples/slac_fel/
├── README.md
├── __init__.py
├── model.py
├── utils.py
├── slac-fel-tutorial.toml
├── model/
│ ├── feature_config.yml
│ ├── final_lcls_fel_model.pt
│ ├── lcls_fel_input_scaler.pt
│ └── lcls_fel_output_scaler.pt
└── test_tomls/
└── dec2025.toml
```

The pretrained model, feature definition, and scalers are included in the
repository.

## Data Availability

The permanent public hosting location for the focused dataset is still being
determined. Until that location is finalized, collaborators who already have
the data can place the 14 files in any local directory and provide that
directory at runtime.

The tutorial does not require a fixed filesystem layout or storage service.

## Run the Tutorial

From the repository root, first install the locked environment:

```bash
uv sync
```

Then run the tutorial:

```bash
uv run python -m src.main \
--config examples/slac_fel/slac-fel-tutorial.toml \
--set data.path=/path/to/fel-tutorial-data
```

Output paths may also be overridden:

```bash
uv run python -m src.main \
--config examples/slac_fel/slac-fel-tutorial.toml \
--set data.path=/path/to/fel-tutorial-data \
--set model.ckpts_path=/path/to/run/checkpoints \
--set logging.metrics_output_path=/path/to/run/metrics.csv
```
The supplied configuration runs on CPU and uses:

- seed 42;
- Page-Hinkley drift detection;
- a 10-observation detector warm-up;
- online EWC with historical replay;
- 200 adaptation updates;
- 13 stream updates after the initial window;
- detector reset after adaptation.

## Expected Result

A validated CPU run produces one drift event:

```text
DRIFT DETECTED (Event #1)
```

The event occurs while processing `data_740.pkl`. Apeiron then completes 200
continual-learning updates, saves `drift_adaptation_1.pt`, and resumes
monitoring through `data_743.pkl`.

The exact pipeline step is 31 with the supplied configuration. The complete
run should process all 14 files once, without wrapping back to the first file.

Typical artifacts are:

```text
output/slac-fel/tutorial/
├── metrics.csv
└── checkpoints/
├── drift_adaptation_1.pt
└── latest
```

Runtime depends on the machine. The validated CPU workflow finishes in well
under one minute.

## Reproduce the Figure

The focused paper-style replay is summarized below:

![Focused SLAC-FEL drift-adaptation replay](figures/focused_drift_adaptation.png)

The blue curve shows the original pretrained surrogate. The red curve uses
the pretrained model before the detected transition and the adapted checkpoint
from the beginning of the trigger window. The lower panel shows the
corresponding rolling absolute error.

This is an offline segmented replay aligned with the paper analysis. It is
not a strict causal replay because the adapted checkpoint becomes available only
after the trigger window has been processed.

Regenerate the figure after completing the tutorial run:

```bash
uv run python -m examples.slac_fel.plot_tutorial \
--data /path/to/fel-tutorial-data \
--checkpoint output/slac-fel/tutorial/checkpoints/drift_adaptation_1.pt \
--output examples/slac_fel/figures/focused_drift_adaptation.png
```
## Interpreting the Result

The tutorial demonstrates a real tradeoff rather than universal improvement.

Across the changed regime, the static pretrained model has approximately
12.33 mJ MAE, while the adapted checkpoint has approximately 0.61 mJ MAE.
Adaptation therefore provides a large improvement after the transition.

However, replaying the adapted checkpoint on the earlier stable regime
increases MAE from approximately 0.16 mJ to 0.55 mJ. The update improves the
new regime but does not preserve all historical behavior.

This distinction is central to trustworthy adaptation:

- a drift alert indicates statistical evidence of change;
- it does not establish the physical cause of that change;
- adaptation can improve current performance while degrading prior regimes;
- operational deployment may require additional scientific or safety checks
before an update is accepted.

The example labels the event only as a detected change. It does not classify
the accelerator condition as normal, anomalous, unsafe, or adversarial.

## How the Example Connects to Apeiron

`model.py` implements the `BaseModelHarness` interface for the FEL surrogate.
It is responsible for:

- loading the pretrained neural network and scalers;
- reading the chronological pickle files lazily;
- creating train, validation, stream, and replay loaders;
- converting scaled prediction error back to millijoules;
- constructing the optimizer used during adaptation.

`utils.py` handles feature configuration, scaler loading, numeric file
ordering, and time-window construction.

`slac-fel-tutorial.toml` contains the focused experiment parameters. The
external data location is supplied with `--set data.path=...`, so the tutorial
does not depend on a particular storage service or filesystem layout.

## Implementation Provenance

The FEL harness and December reproduction configuration were ported from
APEIRON commit
`5542efbb47fdfddbb0b3202beaa2ad4c68cc3bd2`, committed by Sara Miskovich
during the original SLAC-FEL paper-reproduction work.

The feature configuration, pretrained model, and input and output scalers are
retained unchanged from that source commit. The focused configuration,
current-main compatibility updates, tests, documentation, and replay figure
were added for this tutorial.

## Full December Reproduction

The full configuration is:

```text
examples/slac_fel/test_tomls/dec2025.toml
```

It expects 150 files, `data_699.pkl` through `data_848.pkl`.
Empty file added examples/slac_fel/__init__.py
Empty file.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Loading