Skip to content
deem-dataPublic

About

An system for efficiently executing large-scale agentic pipeline search.

Resources

Contributing

Stars

41 stars

Watchers

4 watching

Forks

Repository files navigation

Stratum logo

Python CI Rust CI codecov License Python

Stratum is an ML system for efficiently executing large-scale agentic pipeline search. It integrates with MLE agents by representing batches of agent-generated pipelines as lazily evaluated DAGs, applying logical and runtime optimizations, and executing them across heterogeneous backends, including a Rust-based runtime. Stratum builds on skrub's operator abstraction and is under active development.

📄 Paper: The motivation and vision behind stratum are described in our VLDB 2026 paper.


Design Principles

  • Provide seamless and unrestricted support for arbitrary ML libraries without operator porting.
  • A semantic abstraction built on a minimal set of logical operators that enables rewrites and lazy evaluation with physical operator independence.
  • Implement a runtime with efficient operator kernels (in Rust), scheduling across CPUs, GPUs, and distributed backends, plus runtime optimizations such as buffer pools, reuse of intermediates, and inter- and intra-operator parallelization.

Installation

Install the current development build from PyPI:

python -m pip install "stratum-ai==0.0.0.dev1"

Pre-built Rust wheels are provided for the supported CPython 3.11+ platforms, so a Rust toolchain is not required for a normal installation. The source-build instructions below are for contributors developing Stratum itself.

For source development, you need Python 3.11+ and a Rust toolchain (nightly not required; stable is fine). Stratum uses maturin to build its Rust extension.

From the repository root, create and activate a virtual environment, then install the extension in editable (development) mode:

python -m venv .venv
source .venv/bin/activate  # Windows PowerShell: .venv\Scripts\Activate.ps1
python -m pip install maturin
maturin develop --release

For more details (including building wheels), see the Developer Instructions section below.


Usage

To leverage stratum, agent prompts or pipelines need minor changes. Prompts should be modified to generate code following skrub DataOps syntax.

Stratum can also significantly speed up human-written skrub code.

The following flags enable different features of Stratum. These flags can be set via environment variables or directly in code:

import stratum

stratum.set_config(
    scheduler=True,
    implementation_selector="greedy",
    explain=True,
    stats=True,
)

set_config and nested config(...) contexts change only the settings supplied; None preserves the current value. Use explain=False to turn plan printing off.

To reuse eligible results across runs in the same process, add reuse_cache=True to the configuration. reuse_cache_memory_fraction=0.1 sets its estimated memory budget to 10% of system memory, separate from the BufferPool budget. Results are keyed by Merkle lineage, kept only in memory, and evicted in least-recently-used order. Reuse covers shared preprocessing and per-fold training and prediction. Fold keys include the selected rows; fitted states are cached separately from outputs and retain their producer's size estimate. A hit loads the required values into the buffer pool or state store and prunes work needed only to produce them. File-backed inputs can be keyed, while in-memory input data currently cannot. Disabling reuse bypasses the cache and retains its entries for later enabled runs. Budget changes apply before the next enabled run.

The scheduler can recompile the remaining plan between operator executions. Completed outputs and fitted states keep their existing runtime storage keys; recompilation probes the reuse cache and recomputes intermediate lifetimes while preserving the selected physical implementations.

Example Code

import stratum as skrub #drop-in replacement
from sklearn.preprocessing import OneHotEncoder
from sklearn.linear_model import LinearRegression

def main():
    # Collect and prepare datasets
    dataset = skrub.datasets.fetch_employee_salaries()
    df = skrub.as_data_op(dataset.employee_salaries).skb.subsample()
    df_clean = df.dropna()
    y = df_clean["current_annual_salary"].skb.mark_as_y()
    X = df_clean.drop(columns=["current_annual_salary"]).skb.mark_as_X()

    # Apply feature transformations
    skrub.set_config(scheduler=True, implementation_selector="greedy", explain=True, stats=True)
    tv = skrub.TableVectorizer()
    X_enc = X.skb.apply(tv)
    print(f"Encoded data shape: {X_enc.shape.skb.preview()}")

    # Training and cross-validation
    pred = X_enc.skb.apply(LinearRegression(), y=y)
    search = pred.skb.make_grid_search(cv=3, fitted=True, scoring="r2", refit=False)
    print(search.results_)

if __name__ == "__main__":
    main()

Server

From the repository root, with your virtual environment activated, install the server extra and start a process that shares its in-memory cache across submitted scripts and DAGs:

maturin develop --release --extras server
python -m stratum serve --memory-fraction 0.1 --diagnostics-dir runs

From another terminal with the same virtual environment activated, submit a script or inspect the cache:

python -m stratum submit pipeline.py
python -m stratum stats
python -m stratum clear
python -m stratum shutdown

To integrate a Machine Learning Engineering (MLE) agent, replace local execution with a submission to the running server. Here, MLEAgent stands for your agent, and draft_skrub_pipeline() returns Python source code.

Without Stratum:

pipe = MLEAgent.draft_skrub_pipeline()
exec(pipe)

With Stratum: have the agent generate pipelines using import stratum as skrub, as in the example above, and submit the generated script:

import stratum

with stratum.StratumClient() as client:
    pipe = MLEAgent.draft_skrub_pipeline()
    client.submit(pipe)

submit() accepts Python source as a str or bytes, streams its output, and returns its exit code. Bytes honor Python's source encoding declaration (UTF-8 by default). To submit a file, use client.submit(Path("pipeline.py")) with from pathlib import Path; strings are always treated as source code. Keep the server running across drafts to reuse eligible preprocessing results.

For a DAG already built in Python, use the client:

import stratum

with stratum.StratumClient() as client:
    results = client.grid_search(dag, cv=3, scoring="r2")
    uncached = client.evaluate(dag, config={"reuse_cache": False})

Jobs run sequentially in the server process. Local clients discover its address and token automatically. Scripts and DAGs must be trusted, and their data must be accessible to the server. DAG calls require matching Python, skrub, and Stratum versions. A new server process starts with an empty cache. With diagnostics enabled, session logs, live entry details and a memory-usage timeline are written under runs/server/, and each job's run artifacts are written under runs/. Entry details include operator labels, sizes, hits and the job that added them. Cache events are sampled at most once a second, with every job boundary and cache clear retained. Evicted entries leave no per-key history behind.

Developer Instructions

Running the Tests

In your activated virtual environment, install the test dependencies and run the full test suite:

maturin develop --release --extras test,cpu,hardware
python -m pytest -v tests

Local Dev Install (Editable)

maturin develop				# Debug mode
maturin develop --release	# Optimized dev build

Building Wheels

This produces redistributable .whl files under dist/.

# Linux / macOS
maturin build --release -o dist --interpreter python3.11 --compatibility linux

# Windows
maturin build --release -o dist

Then install with:

python -m pip install ./dist/stratum_ai-*.whl

License

Apache License 2.0. See LICENSE for details.

About

An system for efficiently executing large-scale agentic pipeline search.

Resources

Contributing

Stars

41 stars

Watchers

4 watching

Forks

Releases

Packages

Used by

Contributors

Languages