Repository navigation
Zerodot1/ming3 parallel disk reads - #1042
Merged
0xShug0 merged 3 commits intoOct 23, 2026
Merged
Conversation
I reviewed Issue 0xShug0#1033, which reports parallel disk expert reads and performance measurements on Apple Silicon/macOS. This commit adds a separate Linux implementation of that idea for Fish Audio Drama 3. It does not resolve or close the original issue; the reported macOS behavior still needs its own implementation and validation. On Linux, selected expert slices are copied from the memory-mapped GGUF into disjoint staging ranges with bounded parallel workers. Drama 3 defaults to four read workers on Linux and keeps the serial default on other platforms. The fish_drama3.disk_expert_read_threads session option lets users tune the worker count from 1 to 64. Backend uploads remain ordered and synchronized as before, and pinned expert loading is unchanged. Linux validation on an AMD Radeon RX 9070 XT (ROCm gfx1201, 16 GB VRAM): - Built audiocpp_cli and disk_expert_weight_store_test with HIP enabled. - disk_expert_weight_store_test passed before and after the implementation; synthetic resident/disk comparisons reported max_abs=0. - Ran Fish Audio Drama 3 Q3_K_S GGUF (20,325,196,992 bytes) with HIP, disk experts enabled, cache size 32, cpu_expert_layers=0, 8 backend threads, seed 123, the same 79-character prompt, and max_tokens=128. - Two serial runs measured disk_experts.read_ms at 2929.8 and 2923.6 ms and session.wall_ms at 10200.7 and 10044.5 ms. - Two four-worker runs measured disk_experts.read_ms at 2031.4 and 2147.0 ms and session.wall_ms at 8813.5 and 9045.1 ms. - Warm-cache medians improved from 2926.7 to 2089.2 ms for expert reads (28.6%) and from 10122.6 to 8929.3 ms end-to-end (11.8%). - All tested output WAV files were byte-identical to the serial baseline. These are warm Linux page-cache results on this machine; cold-storage and other Linux hardware performance may differ. Users can set the worker count to 1 to restore serial reads.
Ming 3 already supports keeping routed experts in a bounded GPU cache while loading cache misses from its GGUF. Its disk-backed path always used the shared store's serial reader, even though the Linux parallel reader is available. Pass a configurable worker count from Ming 3 session options into the shared store. Linux defaults to four workers; other platforms retain one. The setting is useful only with disk_experts enabled and pin_experts disabled, so pinned host-memory behavior and existing defaults are unchanged. This change extends the implementation in PR 0xShug0#1040 and currently depends on that PR's DiskExpertWeightStore API. Issue 0xShug0#1033 describes measurements on Apple Silicon/macOS. This Ming 3 Linux follow-up does not address that macOS implementation or close the issue. Tested on Linux with an AMD Radeon RX 9070 XT (ROCm gfx1201, 16 GB VRAM): - Configured and built audiocpp_cli for the ming3_audio model with HIP enabled. - Built and ran disk_expert_weight_store_test successfully; resident and disk-backed synthetic outputs matched exactly (max_abs=0). - Ran Ming 3.0 Audio Tiny Q4_K (9,966,699,744 bytes) ASR on the bundled sample WAV. Each run used HIP, 8 backend threads, cache size 8, max_tokens=256, disk_experts=true, and pin_experts=false. - Each serial run read 7,264,567,296 bytes across 5,474 expert loads. Three read_ms results were 816.5, 826.8, and 830.3 ms; wall times were 4196.5, 4247.6, and 4150.2 ms. - Each four-worker run read the same bytes and expert count. Three read_ms results were 502.3, 460.2, and 512.7 ms; wall times were 3889.3, 3727.6, and 3815.6 ms. - Warm-cache medians: expert reads 826.8 -> 502.3 ms (39.2% faster), total session time 4196.5 -> 3815.6 ms (9.1% faster). - The ASR transcription was byte-identical for serial, four-worker, and default-Linux-worker runs. The default was verified without explicitly setting ming3_audio.disk_expert_read_threads. These are warm-cache results for this machine, model, and sample. Results may vary with storage, backend, quantization, and input. The default pin_experts setting remains enabled except where the backend already selects otherwise; set pin_experts=false to use GGUF range reads and benefit from this option.
0xShug0
marked this pull request as ready for review
October 23, 2026 03:37
0xShug0
force-pushed
the
zerodot1/ming3-parallel-disk-reads
branch
from
October 23, 2026 04:05
0befb23 to
74927a8
Compare
Contributor
Author
|
@0xShug0 Thank you very much for your good work. 🥇 |
cunba-ai
pushed a commit
to cunba-ai/audio.cpp
that referenced
this pull request
Oct 28, 2026
…(Qwen3/YuE2/ACE-Step/MiniMax cuda_perf) + Metal IM2COL + Gradium Phonon + Drama3/Ming3-Tiny speedups Upstream commits 12cd5bb..eefd932 (17): - multilingual Granite TurboCTC checkpoint (0xShug0#1035) - Qwen3 TTS optimization controls + extended cuda_perf preset (0xShug0#1055); preset renamed perf_first - VoxCPM2 optimized preset renamed cuda_perf - YuE2 performance controls + cuda_perf preset (0xShug0#1056) - Zerodot1/ming3 parallel disk reads (0xShug0#1042) - Metal: 1D IM2COL cooperative tiling (0xShug0#1057) - Zerodot1/arch linux amd installer (0xShug0#1058) - opt-in plain RMS norm for timestep embeddings (0xShug0#1061) - Gradium Phonon TTS on PocketTTS runtime (0xShug0#1062) - Drama3 sampling optimized, tower decode on-device (0xShug0#1063) - parallel expert reads on Apple without OpenMP (0xShug0#1064) - UI models + LatentSync panel + models-folder history (0xShug0#1066) - Drama3 codec batching + overlap decode (0xShug0#1065); Gemma scales precomputed (0xShug0#1067) - Drama3 decode perf controls + F16 KV default (0xShug0#1068) - Ming3 Audio Tiny 2x faster, 20% less VRAM (0xShug0#1069) - ACE-Step perf controls + cuda_perf preset (0xShug0#1073) - MiniMax Music3 perf controls + compact output head (0xShug0#1077) # Conflicts: # src/models/qwen3_tts/talker.cpp # tests/unittests/test_decoder_packed_projections.cpp
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.