Skip to content

Add Megatron Core backends for Qwen3 Target and DSpark Draft - #53

Open
zhouaihui wants to merge 1 commit into
inclusionAI:mainfrom
zhouaihui:feat/megatron-backends
Open

zhouaihui wants to merge 1 commit into
inclusionAI:mainfrom
zhouaihui:feat/megatron-backends

Conversation

@zhouaihui

@zhouaihui zhouaihui commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

PR 1 of 4 — Target/Draft integration and tensor parallelism (TP). This PR implements the first part of the plan for #42. It covers all three new pairings for Qwen3-8B and DSpark (Megatron Target + Torch Draft, SGLang Target + Megatron Draft, and Megatron Target + Megatron Draft), including training, checkpoint recovery, export and strict reload. Implementation plan in #42

The remaining work is planned for three follow-up PRs.

  • PR 2 adds packed sequences (THD), context parallelism (CP) and sharded feature transfer for long contexts.
  • PR 3 adds pipeline parallelism (PP) for the frozen Target and streams features as stages finish, with bounded GPU buffers and safe draining.
  • PR 4 shares Target features across independently progressing Drafts and validates backpressure, failure isolation, recovery and performance across supported configurations.

Outcome

Adds separately selectable Target and Draft backends for dense Qwen3-8B and DSpark using upstream Megatron Core 0.20.0.

  • The frozen Target loads Qwen3 weights into a Core HybridModel. Each request emits five intermediate residuals and the final normalized state through ASpec's existing Mooncake feature contract.
  • The Draft reuses ASpec's DSpark objective, attention and loss kernels, FP32-master AdamW and schedule. Core supplies tensor-parallel projections and DDP gradient buffers. Megatron gradient accumulation and DP reduction can use FP32 (default) or BF16.
  • Either role can run in an isolated Megatron environment via the new ling-aspec-megatron profile. The default SGLang Target and Torch Draft remain independently selectable.

Numerical validation

The online comparison on SM90 used 200 Open PerfectBlend prompts with Qwen3-8B responses, 100 optimizer updates and two samples per update. Within each pair, both roles used the same TP size, with the indicated Draft DP size. The runs used matching initial weights, BF16 gradients and runtime-verified deterministic native cuDNN-FE attention backward on each Draft rank. At TP2, the Torch reference used a validation-only PyTorch DTensor adapter around the existing ASpec DSpark trainer.

The loss is the optimized DSpark objective (0.1 × CE + 0.9 × L1 + confidence). Each mean averages the 100 updates. Mean relative difference is (Megatron mean − baseline mean) / baseline mean.

Topology SGLang Target + Torch Draft mean loss Megatron Target + Megatron Draft mean loss Mean rel diff Max absolute per-step rel diff
TP1/DP1 2.83849453 2.83849453 0% 0%
TP2/DP1 2.83903998 2.83903998 0% 0%
TP2/DP2 2.85058107 2.85058107 0% 0%

All three pairs also matched on per-step gradient norms and final Draft weights, FP32 masters and Adam moments. The TP1 reference used single-request prefill to match the token-row dimension in the first MLP down projection. The separate eight-sample evaluation had a −0.0001653% CE difference at TP2/DP1.

Compatibility and verification

  • On the submitted commit, 1,327 regression tests and four subtests passed. Ruff and documentation link checks passed, and four wheels and four source distributions passed the release audit. A fresh TP2/DP1 fixed-feature run on the rebased code matched both Torch objectives and gradient norms over two updates, with 20 deterministic attention backward calls per rank on both backends.
  • Fixed-feature TP2/DP1 and TP2/DP2 runs restored each backend at update 50 and reproduced the remaining 50 updates and final state. Online checks exercised feature transport, fresh-process recovery, Draft export and strict native loading.
  • Exact resume requires the same Draft backend, topology and gradient precision. Checkpoints include TP shard metadata and Core RNG state. The integration uses the unmodified public Core API at revision 685fb8bd970185e9ac3283fe32a5d6667bf4b268.
  • Scope is single-node BF16, MBS1, CP1/PP1 and TP1/TP2. Megatron Draft packing, decode, throughput, peak memory and held-out quality are outside the validated scope.

@coderabbitai

coderabbitai Bot commented Nov 19, 2026

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 59 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

Commits

Reviewing files that changed from the base of the PR and between b7a4161 and 6856154.

Files selected for processing: 66

  • .dockerignore
  • aspec/_environment.py
  • aspec/_target_worker.py
  • aspec/cli.py
  • aspec/commands/_checkpoint_audit.py
  • aspec/commands/_recipe_smoke.py
  • aspec/commands/doctor.py
  • aspec/config/inference_config.py
  • aspec/config/megatron.py
  • aspec/config/train_config.py
  • aspec/controller/eval.py
  • aspec/controller/inference_manager.py
  • aspec/controller/loop.py
  • aspec/evidence.py
  • aspec/feature_contract.py
  • aspec/inference/context.py
  • aspec/inference/engine/megatron_engine.py
  • aspec/inference/engine/target_actor.py
  • aspec/inference/factory.py
  • aspec/inference/managed.py
  • aspec/inference/service.py
  • aspec/inference/service_protocol.py
  • aspec/inference/target_runtime.py
  • aspec/models/megatron.py
  • aspec/models/megatron_draft.py
  • aspec/models/megatron_target.py
  • aspec/planning.py
  • aspec/ray/placement_group.py
  • aspec/ray/train_group.py
  • aspec/runtime_identity.py
  • aspec/train_entry.py
  • aspec/training/checkpoint.py
  • aspec/training/data_fetcher.py
  • aspec/training/dflash_trainer.py
  • aspec/training/megatron_optimizer.py
  • aspec/training/megatron_trainer.py
  • aspec/training/optimizer.py
  • aspec/training/tp_source_prefetch.py
  • aspec/training/trainer.py
  • aspec/training/trainer_actor.py
  • aspec/utils/env.py
  • aspec/utils/train_dump.py
  • docs/README.md
  • docs/framework/REFERENCES.md
  • docs/releasing.md
  • docs/support.md
  • docs/user-guide/installation.md
  • docs/user-guide/megatron.md
  • requirements/megatron-build.txt
  • requirements/megatron-v0.20.0.txt
  • requirements/sglang-v0.5.21.txt
  • setup.py
  • setup.sh
  • tests/cpu/test_alignment_rel_diff.py
  • tests/cpu/test_megatron.py
  • tests/cpu/test_release_audit.py
  • tests/cpu/test_target_service.py
  • tests/cpu/test_torch_tp_validation.py
  • tests/test_optimizer_state_keys.py
  • tools/audit_release.py
  • tools/build_release.py
  • tools/install.sh
  • tools/validation/megatron_pipeline_acceptance.py
  • tools/validation/megatron_target_aspec_alignment.py
  • tools/validation/megatron_torch_alignment.py
  • tools/validation/torch_tp_dspark.py
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant