ExaServe is an allocation-scoped Ray Serve control plane for OpenAI-compatible LLM inference on ALCF Aurora. ExaServe has been evaluated from 1 to 256 Aurora nodes, including HAProxy-fronted Llama-3-8B serving and multi-node, pipeline-parallel Llama-3.1-405B inference. The non-streaming study reached 27.1k requests/s at 256 nodes; streaming and gateway-specific limits are documented in the operations handbook. These measurements were made on earlier snapshots of this code; the current tree has not been re-measured at that scale.
The reference runtime is Aurora PBS, Intel PVC/XPU, vLLM, HAProxy, non-streaming
completions and a trusted allocation-internal network. Slurm, CUDA/ROCm, SGLang,
streaming, direct Serve exposure and other gateways are rejected or
validation-only under the default SiteProfile; other platforms need their own
runtime profiles and validation.
The original design and its measurements on Aurora are described in
ExaServe: Large-Scale LLM Serving on Exascale HPC Systems.
The 256-node evaluations establish the demonstrated scale of the system for
those workloads; they are not release evidence for the current tree. Focused
regression checks for later fixes have their own coverage, recorded in the
changelog; their allocation size does not
redefine that demonstrated scale. No artifact of the current tree is
production-qualified, and production submission fails closed; see
docs/hardening/STATUS.md.
Differences from the paper. The current tree differs from the design the paper describes in five ways:
- The deployment YAML is one flat mapping. The paper's nested
ray_cluster_config,model_deployment_configandproxy_configsections are rejected (canonical deployment YAML). - Compatibility patches are applied to hash-checked copies of the installed Ray and vLLM modules, which stay unmodified, and the compiled plan selects the patches. The paper's corrected Ray Serve file copies and its per-patch environment switches are gone.
- Readiness is a generation-bound predicate over the plan, not the stdout markers the paper describes (readiness and compatibility).
- A dense HAProxy deployment is one model application holding every replica, plus a proxy-anchor application on each node, instead of one Serve application per node.
- Only the allocation head reads or writes shared storage during a run. It broadcasts each pipeline stage's shards to the nodes that run that stage; in the paper, each node read its 405B stage from Lustre.
Start with the documentation index,
getting-started guide, usage examples,
and API reference. The unified docs/ tree also contains the
authoritative architecture plan, design notes, the hardening status and the
operations handbook, which collects measured
behaviour, limits, failures seen and their fixes. The
path migration guide explains relocated historical
references.
The repository is AI-ModCon/exaserve.
It is published as development and research source, not as a
production-qualified release; hardening status
records what each candidate has and has not qualified. The
AI-ModCon compliance checklist records the
BaseTemplate revision, implemented requirements, validation limits, and GitHub
administrator steps.
One Python composition root owns a deployment generation from allocation bind through bounded teardown:
PBS job
-> exaserve.launcher (one composition root)
-> immutable DeploymentPlan + SiteProfile + AllocationBinding
-> finite model/source staging
-> one supervised rank launcher
-> one NodeSupervisor per planned node
-> Ray daemon and rank-local receipt ingress
-> rank-zero deployment child (Ray Serve applications)
-> one head-owned HAProxy
-> generation-bound readiness predicate and status publication
Subprocesses remain only at real OS/process boundaries: PBS commands, MPI, Ray daemons, the isolated Serve deployment child, native staging, and HAProxy. They use argument vectors, finite deadlines, process-group ownership, and structured handshakes/status. No shell script or stdout marker owns lifecycle or readiness.
The default dense real-engine layout behind HAProxy uses one model application
with all replicas: one model, TP=1, PP=1, and exactly one replica on every GPU
of every serving node. Serve selects the replica; HAProxy distributes connections
among the node proxies. The plan derives this layout automatically. Per-node
proxy-anchor applications remain, so the total application count is
num_nodes + 1. Sparse, multi-device, pipeline-parallel and multi-model plans
keep one application per replica; dense null-compute keeps its node groups.
docs/design/call_graph.md walks the same path
function by function, worked against the canonical two-node HAProxy example.
Prepare the site runtime using the
deployment setup guide.
The current serving profile requires the qualified site-local interpreter, not
a home-directory virtualenv. In a validated compute session, load the profiled
stack as Prepare the runtime
shows (the default frameworks module fails ExaServe's compatibility check),
then check the site-provisioned environment:
PYTHONNOUSERSITE=1 PYTHONSAFEPATH=1 python3 -c 'import sys, exaserve; print(sys.executable); print(exaserve.__file__)'
command -v mpiexec
command -v haproxyRay, vLLM, MPI, and the XPU stack come from Aurora's frameworks/2025.3.1
module, loaded from the 26.26.0 image's module tree as that section shows. The
selected Python and package paths must match the intended approved runtime and
ExaServe artifact, not an incidental checkout. If the package is unavailable,
have the operator provision the exact wheel and control-plane dependencies for
that interpreter and batch bootstrap without replacing the site stack. Do not
modify the read-only module installation yourself or install ExaServe only with
--user: job isolation disables Python's user-site packages. A home-directory
venv is not a workaround for the qualified-interpreter requirement.
Submission captures its interpreter's executable; review the
current bootstrap limitations.
The wheel supplies the Python control plane plus the source for its small native MPI
broadcast helper. HAProxy must be available on PATH; the repository includes
scripts/build_haproxy.sh as an operator setup helper. Build it on compute with
an audited source hash, not on the login node.
A candidate is not a production release until the status document names its exact wheel and every required approval and qualification gate. Before that point, normal production submission fails closed; only authorized, predeclared work may set
validation_mode: true.
The current default SiteProfile intentionally has no approved, evidence-backed
production envelope, so submitting the canonical HAProxy example currently
fails with production execution is not qualified. That is a release gate, not
a setup error. After docs/hardening/STATUS.md records the required measurements
and approval, copy the canonical HAProxy example and change the shared model
path. Submission also needs the site variables described in
getting started:
cp examples/config.haproxy.yaml my_config.yaml
$EDITOR my_config.yaml
# The job sources this site environment script; submission requires it.
export EXASERVE_ENV_SCRIPT_AURORA=/absolute/path/to/environment.sh
# The site profile derives its model-storage root from this, even when the YAML sets model_storage_path.
export EXASERVE_PROJECT_ROOT=/lus/flare/projects/YOUR_PROJECT
exaserve-serve-submit my_config.yaml \
--project-account YOUR_PROJECT \
--waitThe command prints the exact scheduler job ID and, only after the matching
generation publishes canonical READY, its compiled advertised endpoint. It
does not infer readiness from PBS RUNNING, a listening proxy, a log line, or a
newest file.
Scheduler requests are validated against the selected SiteProfile. Aurora's
physical/validation ceiling is distinct from an approved production envelope;
neither a larger allocation limit nor a successful paper run automatically
widens production support. Queue and walltime can be overridden by CLI arguments or
EXASERVE_DEFAULT_QUEUE / EXASERVE_DEFAULT_WALLTIME only within an approved
or explicitly authorized validation envelope.
The submission protocol is idempotent. A durable SUBMITTING intent prevents a
blind retry when scheduler acceptance is ambiguous; use --new-generation
only when deliberately creating a new deployment generation.
Useful commands:
# For an authorized validation or qualified production config, render without submitting.
exaserve-serve-submit my_config.yaml --project-account YOUR_PROJECT --dry-run --log-dir ./submit-preview
# Resolve an already submitted job; add --wait to poll canonical status.
exaserve-serve-url JOB_ID --wait
# Inspect why a known run directory is or is not ready.
exaserve-status show --run-dir /path/to/run --jsonEnd the allocation with the scheduler (qdel JOB_ID). The composition root
handles drain, child/process-group cleanup, leases/listeners, and terminal
status publication.
There is one flat input language. Nested model_deployment_config,
proxy_config, and mutable Ray-runtime manifests were removed rather than kept
as parallel compatibility schemas.
num_nodes: 2
num_gpus_per_node: 12
node_cpus: 64
ray_port: 6379
model_storage_path: /lus/flare/projects/YOUR_PROJECT/YOUR_USER/models
local_stage_path: /tmp/hf_home
deployment_name: exaserve_serve
replica_max_ongoing_requests: 32
engine: vllm
vendor: xpu
models:
- model_id: meta-llama/Meta-Llama-3-8B-Instruct
tensor_parallel_size: 1
pipeline_parallel_size: 1
max_model_len: 4096
size: 8
gpu_memory_utilization: 0.90
enforce_eager: true
max_num_seqs: 64
num_cpus_per_replica: 4
exposure:
mode: PROXIED_INTERNAL
network_boundary: trusted_allocation
request_body_limit_bytes: 16777216
gateway:
kind: haproxy
port: 4001
backend_port: 8000
executable_ref: PATH:haproxy
worker_count: 1
options:
balance: leastconn
maxconn: 8000maxconn: 8000 is a conservative default, not a site ceiling. ExaServe
computes HAProxy's descriptor demand as 2 * maxconn + backend_count + 256,
checks it against the hard RLIMIT_NOFILE before launch, and raises the soft
limit when allowed. Aurora compute nodes allowed 1,048,575 descriptors when
measured in October 2026 (login nodes allow 16,384). Size maxconn to the
offered rate times end-to-end latency, plus headroom.
examples/config.reference.yaml is a
reference configuration of the commonly used fields; the accepted keys are
defined in src/exaserve/plan/compiler.py.
examples/config.direct.yaml
is explicitly DIRECT_VALIDATION, and config.litellm.yaml is a
validation-only alternate-gateway example. NGINX, Envoy, and Pingora adapters
exist for future qualification/benchmark work, but no shipped example or
production exposure claim currently advertises them.
READY is a continuously evaluated projection over the exact plan and current
generation. It requires:
- every planned node session and supervised component;
- exact Ray membership and resource totals;
- every planned Serve application/replica/route;
- engine-level readiness and exact compatibility receipts;
- the owned gateway process/listener; and
- a real canary through the compiled advertised endpoint.
Loss after READY revokes readiness and drives the declared terminal policy.
Compatibility is activated before Ray/vLLM imports and attested in each actual
process role; there is no full-file Ray overlay or source-tree replacement.
The eval control plane compiles an immutable RunPlan containing the exact
DeploymentPlan; its runtime manifest references the plan by path and SHA-256
instead of copying serving fields.
Set EXASERVE_PROJECT_ROOT, and for materialization and submission also
EXASERVE_PROJECT_ACCOUNT and EXASERVE_ENV_SCRIPT_AURORA, as in
getting started;
even spec validate needs the project root. Run from the repository root:
the eval tooling imports ExaServe from the checkout's src/.
# In a shell kept for eval and ClientLab, not the one you use for
# exaserve-serve-submit or the installed-wheel check:
export PYTHONPATH="$PWD:$PWD/src"
python3 -m eval.cli spec validate eval/specs/smoke_haproxy_1node.yaml
python3 -m eval.cli run materialize eval/specs/smoke_haproxy_1node.yaml
python3 -m eval.cli run submit-all smoke_haproxy_1node --run-group run0Use the exact runN group returned by materialization, not an assumed run0.
Execution occurs in the generated compute job, not on the login node.
ClientLab uses python3 -m clientlab. Synthetic studies are explicitly
diagnostic-only. A real ExaServe study must bind one RunPlan, trace, generation,
and deployment-status directory; it cannot accept a raw endpoint or maintain a
private readiness/scheduler control plane. The retired ad-hoc runners are
explained in clientlab/scripts/README.md.
For a portable Linux development environment, use the pinned uv version and the repository's locked tooling. No server/GPU extras are selected by default:
python3 -m pip install uv==0.13.4
make install-dev
make lock-check
make lint
make format-check
make type-checkThe uv resolution envelope is Linux with Python 3.11–3.13: the existing
optional vLLM version excludes Python 3.14, and Pillow 13.0.0, the first release
without its known advisories, requires Python 3.11. CI tests the portable core
on Python 3.12, matching the current Aurora Python 3.12.12 runtime. The
published core minimum remains 3.10, but Python 3.10 is outside the lock and
the current CI scope.
This is not a claim that a GPU stack or every interpreter is qualified.
requirements/aurora-frameworks-2025.3.1.lock remains the separate
site runtime record. Do not use uv sync --all-extras to replace the Aurora
frameworks environment.
On Aurora, load the stack in
Prepare the runtime, which
includes go, before brief static checks. That stack puts its own uv 0.10.1
first on PATH, which this project refuses, so install the pinned uv for your
user and pass it to make:
python3 -m pip install --user uv==0.13.4
make UV="$(python3 -m site --user-base)/bin/uv" install-dev lint format-check type-checkUse the same UV= for make test, test-cov and build. Full test
suites, package builds, MPI and GPU work require a validated compute session
and fresh environment setup. Once in that session
(or on a suitable non-HPC development machine), use:
make test
make test-cov
make buildCoverage is reported for the portable Python code, with XML artifacts retained by CI. It does not measure or certify real GPU, native MPI, scheduler, or scale behavior. The independently installed-wheel and randomized-order gates remain separate from the editable development environment.
Staging tests require /tmp to be a separate writable tmpfs from the declared
shared roots. GitHub-hosted CI provisions that filesystem explicitly; the
production locality guards remain active. Tests that import the optional
Ray/FastAPI serving API are skipped when those dependencies are absent; pure
deployment-plan and routing tests still run in the portable lane.
The release gate also builds an sdist and wheel, installs the wheel into a clean
Python environment outside the source tree, runs deterministic and randomized
hermetic suites, validates docs/hardening/FINDINGS.yaml, strictly compiles the
native MPI helper, and performs the required tests inside validated Aurora
compute sessions. Hardware/scale evidence that has not run on the final code is
reported as pending, never inferred from unit tests or historical artifacts.
The authoritative architecture and implementation contract is
docs/PRODUCTION_HARDENING_EXECUTION_PLAN.md.
Historical audits, KNOWN_ISSUES, and TODO material are context only and cannot
override it.
See CONTRIBUTING.md for the development workflow, required
human review, and AI-assisted contribution policy, and
CODE_OF_CONDUCT.md for community standards.
Use the repository's issue templates for ordinary bugs and feature requests.
Report vulnerabilities privately as described in SECURITY.md.
The current maintainer contact is Wenyi Wang, wenyiw@uchicago.edu.
ExaServe development has included assistance from coding agents, including OpenAI Codex and Claude Code, for implementation, audits, tests and documentation. Human contributors remain responsible for the work; see the AI-assistance and attribution policy.
Repository infrastructure and community guidance draw on AI-ModCon/BaseTemplate. We acknowledge the U.S. Department of Energy's Genesis Mission support for the ModCon Base collaboration. This statement does not assert a project-specific award.
ExaServe is licensed under Apache-2.0, except where individual files state otherwise. See NOTICE for attribution. External dependencies, vendored third-party code, and model weights retain their respective terms.