HyperParallel provides distributed parallel acceleration for Ascend SuperPoDs. Designed for resource pooling, peer-to-peer architectures, hierarchical network topologies, and FP8 low-precision formats, it spans cluster-level distributed parallelism and on-chip multicore parallelism. It supports pooled management of CPU DRAM and NPU HBM, topology-aware scheduling, communication path planning, and FP8 mixed-precision training.
-
Decoupling model code from system optimization: HyperParallel separates parallelism, recomputation, and offload from model code and injects them through declarative interfaces. Its parallel paradigms cover cluster SPMD, cluster MPMD, and on-chip multicore MPMD for large language and multimodal model training.
-
Hugging Face ecosystem compatibility: HyperParallel reads Hugging Face model configurations and pretrained weights directly and reuses existing Transformers model implementations. After model creation, HyperParallel declaratively injects distributed logic such as parallelism, recomputation, and operator replacement.
-
End-to-end determinism: HyperParallel supports determinism across computation, communication, data preprocessing, and random number generation, with bitwise alignment validation for supported models. Deterministic mode may incur a performance cost, but helps reproduce training results, detect silent data corruption (SDC), and diagnose issues.
-
Hybrid dynamic and static execution: Static graphs facilitate compute-communication overlap, memory analysis, and execution scheduling, but full dynamic-to-static conversion places substantial demands on model syntax and compiler capabilities. HyperParallel supports constrained, local dynamic-to-static conversion on top of eager execution.
- [Oct. 15, 2026]: DeepSeek-V4.1-Flash model support
- [Oct. 9, 2026]: moonshotai/Kimi-K2.6 model support
- [Aug. 25, 2026]: Qwen/Qwen3-30B-A3B model support
See the model roadmap for detailed capabilities and ongoing work.
| Model | Example configuration / documentation |
|---|---|
| Qwen3 | Model entry |
| Qwen3-MoE | Recipes |
| Qwen3-VL-MoE | Model adapter |
| Qwen3.5 Dense | Model entry |
| DeepSeek-V3 | Recipes |
| DeepSeek-V3.2 | Recipes |
| DeepSeek-V4.1 | Text configuration · Multimodal configuration |
| Kimi-K2.6 | Recipes |
| openPangu-2.0 | Repository |
See the Installation Guide for installation methods, source build options, and environment requirements.
Run the single-node, eight-device training example provided in hyper_parallel/models/qwen3_moe/recipes/train.yaml:
torchrun --standalone --nproc_per_node=8 \
scripts/train_lm.py hyper_parallel/models/qwen3_moe/recipes/train.yamlAdditional command-line arguments override configuration values by YAML field path, without modifying the configuration file:
torchrun --standalone --nproc_per_node=8 \
scripts/train_lm.py hyper_parallel/models/qwen3_moe/recipes/train.yaml \
--model.pretrained_model_name_or_path=/path/to/model \
--training.train_iters=10See Training Configuration and Components for field definitions and parsing rules.
The following example applies fully_shard on a data-parallel DeviceMesh to shard model parameters:
from hyper_parallel import fully_shard, init_device_mesh
dp_mesh = init_device_mesh("npu", (dp_size,), mesh_dim_names=("dp",))
model = fully_shard(model, mesh=dp_mesh)See the Feature Guides for TP, CP, EP, and PP combinations, and the API Reference for public interfaces.
- Documentation Center — Documentation index and navigation
- Training Configuration and Components — TrainerConfig fields, module structure, and component documentation
- Installation Guide — Source builds and dependencies
- Feature Guides — Parallelism and training capabilities
- API Reference — Interfaces organized by feature
- FAQ and Troubleshooting — Common issues and solutions
- AI-Assisted Development — Instructions for AI-assisted development
- Contributing — Contribution workflow, testing, review, and releases
- Release Notes — Version history
Issues and pull requests are welcome. Start with the Contribution Guide for development setup, testing, review, and the community release workflow.
If you are interested in HyperParallel's technical direction, join the Parallel Training System SIG.
Scan the QR code below to join the Parallel Training System SIG WeChat group and discuss usage, technical approaches, and project progress with community developers.
Scan the QR code below to follow HyperParallel on REDnote (account: hyperparallel) for technical articles and practical experience.
- HyperParallel-FSDP: Topology-Aware Fully Sharded Training with Layout-Driven Muon on Ascend SuperPods [paper]
- HyperParallel-Mpipe: A Composable Algebra System for Optimizing MLLM Training over Supernode Clusters [paper]
- HyperParallel-MoE: Multi-Core Interleaved Scheduling for Fast MoE Training on Ascend NPUs [paper]
- HyperParallel: A Supernode-Affinity AI Framework [paper]
- HyperOffload: Graph-Driven Hierarchical Memory Management for Large Language Models on SuperNode Architectures [paper]
We thank the following partners and projects for their work and support: