Stratum is an ML system for efficiently executing large-scale agentic pipeline search. It integrates with MLE agents by representing batches of agent-generated pipelines as lazily evaluated DAGs, applying logical and runtime optimizations, and executing them across heterogeneous backends, including a Rust-based runtime. Stratum builds on skrub's operator abstraction and is under active development.
📄 Paper: The motivation and vision behind stratum are described in our VLDB 2026 paper.
- Provide seamless and unrestricted support for arbitrary ML libraries without operator porting.
- A semantic abstraction built on a minimal set of logical operators that enables rewrites and lazy evaluation with physical operator independence.
- Implement a runtime with efficient operator kernels (in Rust), scheduling across CPUs, GPUs, and distributed backends, plus runtime optimizations such as buffer pools, reuse of intermediates, and inter- and intra-operator parallelization.
Install the current development build from PyPI:
python -m pip install "stratum-ai==0.0.0.dev1"Pre-built Rust wheels are provided for the supported CPython 3.11+ platforms, so a Rust toolchain is not required for a normal installation. The source-build instructions below are for contributors developing Stratum itself.
For source development, you need Python 3.11+ and a Rust toolchain (nightly not required; stable is fine). Stratum uses maturin to build its Rust extension.
From the repository root, create and activate a virtual environment, then install the extension in editable (development) mode:
python -m venv .venv
source .venv/bin/activate # Windows PowerShell: .venv\Scripts\Activate.ps1
python -m pip install maturin
maturin develop --releaseFor more details (including building wheels), see the Developer Instructions section below.
To leverage stratum, agent prompts or pipelines need minor changes. Prompts should be modified to generate code following skrub DataOps syntax.
Stratum can also significantly speed up human-written skrub code.
The following flags enable different features of Stratum. These flags can be set via environment variables or directly in code:
import stratum
stratum.set_config(
scheduler=True,
implementation_selector="greedy",
explain=True,
stats=True,
)set_config and nested config(...) contexts change only the settings supplied;
None preserves the current value. Use explain=False to turn plan printing off.
To reuse eligible results across runs in the same process, add
reuse_cache=True to the configuration. reuse_cache_memory_fraction=0.1
sets its estimated memory budget to 10% of system memory, separate from the
BufferPool budget. Results are keyed by Merkle lineage, kept only in memory,
and evicted in least-recently-used order. Reuse covers shared preprocessing and
per-fold training and prediction. Fold keys include the selected rows; fitted
states are cached separately from outputs and retain their producer's size
estimate. A hit loads the required values into the buffer pool or state store
and prunes work needed only to produce them. File-backed inputs can be keyed,
while in-memory input data currently cannot. Disabling reuse bypasses the cache
and retains its entries for later enabled runs. Budget changes apply before the
next enabled run.
The scheduler can recompile the remaining plan between operator executions. Completed outputs and fitted states keep their existing runtime storage keys; recompilation probes the reuse cache and recomputes intermediate lifetimes while preserving the selected physical implementations.
import stratum as skrub #drop-in replacement
from sklearn.preprocessing import OneHotEncoder
from sklearn.linear_model import LinearRegression
def main():
# Collect and prepare datasets
dataset = skrub.datasets.fetch_employee_salaries()
df = skrub.as_data_op(dataset.employee_salaries).skb.subsample()
df_clean = df.dropna()
y = df_clean["current_annual_salary"].skb.mark_as_y()
X = df_clean.drop(columns=["current_annual_salary"]).skb.mark_as_X()
# Apply feature transformations
skrub.set_config(scheduler=True, implementation_selector="greedy", explain=True, stats=True)
tv = skrub.TableVectorizer()
X_enc = X.skb.apply(tv)
print(f"Encoded data shape: {X_enc.shape.skb.preview()}")
# Training and cross-validation
pred = X_enc.skb.apply(LinearRegression(), y=y)
search = pred.skb.make_grid_search(cv=3, fitted=True, scoring="r2", refit=False)
print(search.results_)
if __name__ == "__main__":
main()From the repository root, with your virtual environment activated, install the
server extra and start a process that shares its in-memory cache across submitted
scripts and DAGs:
maturin develop --release --extras server
python -m stratum serve --memory-fraction 0.1 --diagnostics-dir runsFrom another terminal with the same virtual environment activated, submit a script or inspect the cache:
python -m stratum submit pipeline.py
python -m stratum stats
python -m stratum clear
python -m stratum shutdownTo integrate a Machine Learning Engineering (MLE) agent, replace local execution
with a submission to the running server. Here, MLEAgent stands for your agent,
and draft_skrub_pipeline() returns Python source code.
Without Stratum:
pipe = MLEAgent.draft_skrub_pipeline()
exec(pipe)With Stratum: have the agent generate pipelines using import stratum as skrub,
as in the example above, and submit the generated script:
import stratum
with stratum.StratumClient() as client:
pipe = MLEAgent.draft_skrub_pipeline()
client.submit(pipe)submit() accepts Python source as a str or bytes, streams its output, and
returns its exit code. Bytes honor Python's source encoding declaration (UTF-8 by
default). To submit a file, use client.submit(Path("pipeline.py")) with
from pathlib import Path; strings are always treated as source code.
Keep the server running across drafts to reuse eligible preprocessing results.
For a DAG already built in Python, use the client:
import stratum
with stratum.StratumClient() as client:
results = client.grid_search(dag, cv=3, scoring="r2")
uncached = client.evaluate(dag, config={"reuse_cache": False})Jobs run sequentially in the server process. Local clients discover its address
and token automatically. Scripts and DAGs must be trusted, and their data must be
accessible to the server. DAG calls require matching Python, skrub, and Stratum
versions. A new server process starts with an empty cache. With diagnostics
enabled, session logs, live entry details and a memory-usage timeline are written
under runs/server/, and each job's run artifacts are written under runs/.
Entry details include operator labels, sizes, hits and the job that added them.
Cache events are sampled at most once a second, with every job boundary and cache
clear retained. Evicted entries leave no per-key history behind.
In your activated virtual environment, install the test dependencies and run the full test suite:
maturin develop --release --extras test,cpu,hardware
python -m pytest -v testsmaturin develop # Debug mode
maturin develop --release # Optimized dev buildThis produces redistributable .whl files under dist/.
# Linux / macOS
maturin build --release -o dist --interpreter python3.11 --compatibility linux
# Windows
maturin build --release -o distThen install with:
python -m pip install ./dist/stratum_ai-*.whlApache License 2.0. See LICENSE for details.