Skip to content
AI-ModConPublic

About

Exascale Inference

Resources

Code of conduct

Contributing

Security policy

Stars

1 star

Watchers

0 watching

Forks

ExaServe

ExaServe is an allocation-scoped Ray Serve control plane for OpenAI-compatible LLM inference on ALCF Aurora. ExaServe has been evaluated from 1 to 256 Aurora nodes, including HAProxy-fronted Llama-3-8B serving and multi-node, pipeline-parallel Llama-3.1-405B inference. The non-streaming study reached 27.1k requests/s at 256 nodes; streaming and gateway-specific limits are documented in the operations handbook. These measurements were made on earlier snapshots of this code; the current tree has not been re-measured at that scale.

The reference runtime is Aurora PBS, Intel PVC/XPU, vLLM, HAProxy, non-streaming completions and a trusted allocation-internal network. Slurm, CUDA/ROCm, SGLang, streaming, direct Serve exposure and other gateways are rejected or validation-only under the default SiteProfile; other platforms need their own runtime profiles and validation.

The original design and its measurements on Aurora are described in ExaServe: Large-Scale LLM Serving on Exascale HPC Systems. The 256-node evaluations establish the demonstrated scale of the system for those workloads; they are not release evidence for the current tree. Focused regression checks for later fixes have their own coverage, recorded in the changelog; their allocation size does not redefine that demonstrated scale. No artifact of the current tree is production-qualified, and production submission fails closed; see docs/hardening/STATUS.md.

Differences from the paper. The current tree differs from the design the paper describes in five ways:

  • The deployment YAML is one flat mapping. The paper's nested ray_cluster_config, model_deployment_config and proxy_config sections are rejected (canonical deployment YAML).
  • Compatibility patches are applied to hash-checked copies of the installed Ray and vLLM modules, which stay unmodified, and the compiled plan selects the patches. The paper's corrected Ray Serve file copies and its per-patch environment switches are gone.
  • Readiness is a generation-bound predicate over the plan, not the stdout markers the paper describes (readiness and compatibility).
  • A dense HAProxy deployment is one model application holding every replica, plus a proxy-anchor application on each node, instead of one Serve application per node.
  • Only the allocation head reads or writes shared storage during a run. It broadcasts each pipeline stage's shards to the nodes that run that stage; in the paper, each node read its 405B stage from Lustre.

Documentation and AI-ModCon preparation

Start with the documentation index, getting-started guide, usage examples, and API reference. The unified docs/ tree also contains the authoritative architecture plan, design notes, the hardening status and the operations handbook, which collects measured behaviour, limits, failures seen and their fixes. The path migration guide explains relocated historical references.

The repository is AI-ModCon/exaserve. It is published as development and research source, not as a production-qualified release; hardening status records what each candidate has and has not qualified. The AI-ModCon compliance checklist records the BaseTemplate revision, implemented requirements, validation limits, and GitHub administrator steps.

Architecture

One Python composition root owns a deployment generation from allocation bind through bounded teardown:

PBS job
  -> exaserve.launcher (one composition root)
       -> immutable DeploymentPlan + SiteProfile + AllocationBinding
       -> finite model/source staging
       -> one supervised rank launcher
            -> one NodeSupervisor per planned node
                 -> Ray daemon and rank-local receipt ingress
       -> rank-zero deployment child (Ray Serve applications)
       -> one head-owned HAProxy
       -> generation-bound readiness predicate and status publication

Subprocesses remain only at real OS/process boundaries: PBS commands, MPI, Ray daemons, the isolated Serve deployment child, native staging, and HAProxy. They use argument vectors, finite deadlines, process-group ownership, and structured handshakes/status. No shell script or stdout marker owns lifecycle or readiness.

The default dense real-engine layout behind HAProxy uses one model application with all replicas: one model, TP=1, PP=1, and exactly one replica on every GPU of every serving node. Serve selects the replica; HAProxy distributes connections among the node proxies. The plan derives this layout automatically. Per-node proxy-anchor applications remain, so the total application count is num_nodes + 1. Sparse, multi-device, pipeline-parallel and multi-model plans keep one application per replica; dense null-compute keeps its node groups.

docs/design/call_graph.md walks the same path function by function, worked against the canonical two-node HAProxy example.

Install on Aurora

Prepare the site runtime using the deployment setup guide. The current serving profile requires the qualified site-local interpreter, not a home-directory virtualenv. In a validated compute session, load the profiled stack as Prepare the runtime shows (the default frameworks module fails ExaServe's compatibility check), then check the site-provisioned environment:

PYTHONNOUSERSITE=1 PYTHONSAFEPATH=1 python3 -c 'import sys, exaserve; print(sys.executable); print(exaserve.__file__)'
command -v mpiexec
command -v haproxy

Ray, vLLM, MPI, and the XPU stack come from Aurora's frameworks/2025.3.1 module, loaded from the 26.26.0 image's module tree as that section shows. The selected Python and package paths must match the intended approved runtime and ExaServe artifact, not an incidental checkout. If the package is unavailable, have the operator provision the exact wheel and control-plane dependencies for that interpreter and batch bootstrap without replacing the site stack. Do not modify the read-only module installation yourself or install ExaServe only with --user: job isolation disables Python's user-site packages. A home-directory venv is not a workaround for the qualified-interpreter requirement. Submission captures its interpreter's executable; review the current bootstrap limitations.

The wheel supplies the Python control plane plus the source for its small native MPI broadcast helper. HAProxy must be available on PATH; the repository includes scripts/build_haproxy.sh as an operator setup helper. Build it on compute with an audited source hash, not on the login node.

Candidate inspection and eventual submission

A candidate is not a production release until the status document names its exact wheel and every required approval and qualification gate. Before that point, normal production submission fails closed; only authorized, predeclared work may set validation_mode: true.

The current default SiteProfile intentionally has no approved, evidence-backed production envelope, so submitting the canonical HAProxy example currently fails with production execution is not qualified. That is a release gate, not a setup error. After docs/hardening/STATUS.md records the required measurements and approval, copy the canonical HAProxy example and change the shared model path. Submission also needs the site variables described in getting started:

cp examples/config.haproxy.yaml my_config.yaml
$EDITOR my_config.yaml

# The job sources this site environment script; submission requires it.
export EXASERVE_ENV_SCRIPT_AURORA=/absolute/path/to/environment.sh
# The site profile derives its model-storage root from this, even when the YAML sets model_storage_path.
export EXASERVE_PROJECT_ROOT=/lus/flare/projects/YOUR_PROJECT
exaserve-serve-submit my_config.yaml \
  --project-account YOUR_PROJECT \
  --wait

The command prints the exact scheduler job ID and, only after the matching generation publishes canonical READY, its compiled advertised endpoint. It does not infer readiness from PBS RUNNING, a listening proxy, a log line, or a newest file.

Scheduler requests are validated against the selected SiteProfile. Aurora's physical/validation ceiling is distinct from an approved production envelope; neither a larger allocation limit nor a successful paper run automatically widens production support. Queue and walltime can be overridden by CLI arguments or EXASERVE_DEFAULT_QUEUE / EXASERVE_DEFAULT_WALLTIME only within an approved or explicitly authorized validation envelope.

The submission protocol is idempotent. A durable SUBMITTING intent prevents a blind retry when scheduler acceptance is ambiguous; use --new-generation only when deliberately creating a new deployment generation.

Useful commands:

# For an authorized validation or qualified production config, render without submitting.
exaserve-serve-submit my_config.yaml --project-account YOUR_PROJECT --dry-run --log-dir ./submit-preview

# Resolve an already submitted job; add --wait to poll canonical status.
exaserve-serve-url JOB_ID --wait

# Inspect why a known run directory is or is not ready.
exaserve-status show --run-dir /path/to/run --json

End the allocation with the scheduler (qdel JOB_ID). The composition root handles drain, child/process-group cleanup, leases/listeners, and terminal status publication.

Canonical deployment YAML

There is one flat input language. Nested model_deployment_config, proxy_config, and mutable Ray-runtime manifests were removed rather than kept as parallel compatibility schemas.

num_nodes: 2
num_gpus_per_node: 12
node_cpus: 64
ray_port: 6379
model_storage_path: /lus/flare/projects/YOUR_PROJECT/YOUR_USER/models
local_stage_path: /tmp/hf_home
deployment_name: exaserve_serve
replica_max_ongoing_requests: 32
engine: vllm
vendor: xpu

models:
  - model_id: meta-llama/Meta-Llama-3-8B-Instruct
    tensor_parallel_size: 1
    pipeline_parallel_size: 1
    max_model_len: 4096
    size: 8
    gpu_memory_utilization: 0.90
    enforce_eager: true
    max_num_seqs: 64
    num_cpus_per_replica: 4

exposure:
  mode: PROXIED_INTERNAL
  network_boundary: trusted_allocation
  request_body_limit_bytes: 16777216

gateway:
  kind: haproxy
  port: 4001
  backend_port: 8000
  executable_ref: PATH:haproxy
  worker_count: 1
  options:
    balance: leastconn
    maxconn: 8000

maxconn: 8000 is a conservative default, not a site ceiling. ExaServe computes HAProxy's descriptor demand as 2 * maxconn + backend_count + 256, checks it against the hard RLIMIT_NOFILE before launch, and raises the soft limit when allowed. Aurora compute nodes allowed 1,048,575 descriptors when measured in October 2026 (login nodes allow 16,384). Size maxconn to the offered rate times end-to-end latency, plus headroom.

examples/config.reference.yaml is a reference configuration of the commonly used fields; the accepted keys are defined in src/exaserve/plan/compiler.py. examples/config.direct.yaml is explicitly DIRECT_VALIDATION, and config.litellm.yaml is a validation-only alternate-gateway example. NGINX, Envoy, and Pingora adapters exist for future qualification/benchmark work, but no shipped example or production exposure claim currently advertises them.

Readiness and compatibility

READY is a continuously evaluated projection over the exact plan and current generation. It requires:

  • every planned node session and supervised component;
  • exact Ray membership and resource totals;
  • every planned Serve application/replica/route;
  • engine-level readiness and exact compatibility receipts;
  • the owned gateway process/listener; and
  • a real canary through the compiled advertised endpoint.

Loss after READY revokes readiness and drives the declared terminal policy. Compatibility is activated before Ray/vLLM imports and attested in each actual process role; there is no full-file Ray overlay or source-tree replacement.

Eval and ClientLab

The eval control plane compiles an immutable RunPlan containing the exact DeploymentPlan; its runtime manifest references the plan by path and SHA-256 instead of copying serving fields.

Set EXASERVE_PROJECT_ROOT, and for materialization and submission also EXASERVE_PROJECT_ACCOUNT and EXASERVE_ENV_SCRIPT_AURORA, as in getting started; even spec validate needs the project root. Run from the repository root: the eval tooling imports ExaServe from the checkout's src/.

# In a shell kept for eval and ClientLab, not the one you use for
# exaserve-serve-submit or the installed-wheel check:
export PYTHONPATH="$PWD:$PWD/src"
python3 -m eval.cli spec validate eval/specs/smoke_haproxy_1node.yaml
python3 -m eval.cli run materialize eval/specs/smoke_haproxy_1node.yaml
python3 -m eval.cli run submit-all smoke_haproxy_1node --run-group run0

Use the exact runN group returned by materialization, not an assumed run0. Execution occurs in the generated compute job, not on the login node.

ClientLab uses python3 -m clientlab. Synthetic studies are explicitly diagnostic-only. A real ExaServe study must bind one RunPlan, trace, generation, and deployment-status directory; it cannot accept a raw endpoint or maintain a private readiness/scheduler control plane. The retired ad-hoc runners are explained in clientlab/scripts/README.md.

Testing and release gates

For a portable Linux development environment, use the pinned uv version and the repository's locked tooling. No server/GPU extras are selected by default:

python3 -m pip install uv==0.13.4
make install-dev
make lock-check
make lint
make format-check
make type-check

The uv resolution envelope is Linux with Python 3.11–3.13: the existing optional vLLM version excludes Python 3.14, and Pillow 13.0.0, the first release without its known advisories, requires Python 3.11. CI tests the portable core on Python 3.12, matching the current Aurora Python 3.12.12 runtime. The published core minimum remains 3.10, but Python 3.10 is outside the lock and the current CI scope. This is not a claim that a GPU stack or every interpreter is qualified. requirements/aurora-frameworks-2025.3.1.lock remains the separate site runtime record. Do not use uv sync --all-extras to replace the Aurora frameworks environment.

On Aurora, load the stack in Prepare the runtime, which includes go, before brief static checks. That stack puts its own uv 0.10.1 first on PATH, which this project refuses, so install the pinned uv for your user and pass it to make:

python3 -m pip install --user uv==0.13.4
make UV="$(python3 -m site --user-base)/bin/uv" install-dev lint format-check type-check

Use the same UV= for make test, test-cov and build. Full test suites, package builds, MPI and GPU work require a validated compute session and fresh environment setup. Once in that session (or on a suitable non-HPC development machine), use:

make test
make test-cov
make build

Coverage is reported for the portable Python code, with XML artifacts retained by CI. It does not measure or certify real GPU, native MPI, scheduler, or scale behavior. The independently installed-wheel and randomized-order gates remain separate from the editable development environment.

Staging tests require /tmp to be a separate writable tmpfs from the declared shared roots. GitHub-hosted CI provisions that filesystem explicitly; the production locality guards remain active. Tests that import the optional Ray/FastAPI serving API are skipped when those dependencies are absent; pure deployment-plan and routing tests still run in the portable lane.

The release gate also builds an sdist and wheel, installs the wheel into a clean Python environment outside the source tree, runs deterministic and randomized hermetic suites, validates docs/hardening/FINDINGS.yaml, strictly compiles the native MPI helper, and performs the required tests inside validated Aurora compute sessions. Hardware/scale evidence that has not run on the final code is reported as pending, never inferred from unit tests or historical artifacts.

The authoritative architecture and implementation contract is docs/PRODUCTION_HARDENING_EXECUTION_PLAN.md. Historical audits, KNOWN_ISSUES, and TODO material are context only and cannot override it.

Contributing and reporting issues

See CONTRIBUTING.md for the development workflow, required human review, and AI-assisted contribution policy, and CODE_OF_CONDUCT.md for community standards. Use the repository's issue templates for ordinary bugs and feature requests. Report vulnerabilities privately as described in SECURITY.md. The current maintainer contact is Wenyi Wang, wenyiw@uchicago.edu.

Acknowledgments

ExaServe development has included assistance from coding agents, including OpenAI Codex and Claude Code, for implementation, audits, tests and documentation. Human contributors remain responsible for the work; see the AI-assistance and attribution policy.

Repository infrastructure and community guidance draw on AI-ModCon/BaseTemplate. We acknowledge the U.S. Department of Energy's Genesis Mission support for the ModCon Base collaboration. This statement does not assert a project-specific award.

License

ExaServe is licensed under Apache-2.0, except where individual files state otherwise. See NOTICE for attribution. External dependencies, vendored third-party code, and model weights retain their respective terms.

About

Exascale Inference

Resources

Code of conduct

Contributing

Security policy

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages