You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
- cli/api-reference: add q4_0 KV-cache dtype, --qwen-image-lora, missing
Qwen-Image-Edit and paged-kv-quant-bits flags; correct --cfg auto default
- api-reference/server: add --prefill-chunk-size, --kv-cache-dtype and
qwen-image server flags; new scheduler env vars (solo prefill chunks,
decode quantum) and Qwen-Image env vars (companions, LoRA, step cache)
- models/features: document Lightning LoRA and whole-step DiT cache
- advanced/features: TurboQuant tiers are Q2/Q4/Q8; describe solo-prefill
fresh/tail chunk split
- search indexes: add keywords for the new features (both languages)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFvrnoYEae44hnFrd5PPqa
<li><strong>Fused GPU decode</strong> (Gemma 4) — all transformer layers in a single GGML graph dispatch, cutting CPU↔GPU round-trips from hundreds per token to one (~2.6× over per-op dispatch).</li>
77
77
<li><strong>Fused GPU prefill</strong> (Gemma 4) — dense layers run the whole block (norms, QKV, RoPE, attention, FFN, residuals) as one dispatch per layer during prefill.</li>
78
-
<li><strong>Chunked prefill</strong> (Gemma 4) — long prompts are split into bounded chunks to avoid O(n²) attention score tensors for sliding-window layers.</li>
78
+
<li><strong>Chunked prefill</strong> (Gemma 4) — long prompts are split into bounded chunks to avoid O(n²) attention score tensors for sliding-window layers. A solo (uncontended) request gets a split chunk size: the fresh <code>start_pos = 0</code> chunk runs through the fused whole-graph path at <code>TS_SCHED_SOLO_PREFILL_CHUNK</code> (default 8192), the remaining tail at <code>TS_SCHED_SOLO_TAIL_PREFILL_CHUNK</code> (default 2048).</li>
<li><strong>Native quantized compute</strong> — Q4_K_M, Q6_K, Q8_0, IQ2_XXS, MXFP4 used directly in matmul without expanding to FP32; a batched <code>AddmmQuantBatch</code> handles multiple sub-weight matmuls in one dispatch.</li>
81
81
<li><strong>Batched GPU MoE</strong> — all selected experts (plus the optional shared expert and residual add) collapse into a single GGML graph dispatch per MoE layer.</li>
<li><strong>Zero-copy file-mapped weights</strong> — the GGUF is memory-mapped and quantized tensors bind directly into native ops, removing a per-tensor copy that roughly doubled the resident set. Example: <code>Qwen3.5-35B-A3B-IQ2_XXS</code> (~10 GB GGUF) runs at ~7 GB peak under Metal instead of ~17 GB.</li>
89
89
<li><strong>Best-fit memory pool</strong> with bounded retention (blocks capped at 64 MB, pool at 32 blocks) keeps the working set tight across long runs.</li>
90
90
<li><strong>Paged KV block pool with optional SSD spillover</strong> — RAM-capped, LRU-evicted, with content-hash prefix reuse across sessions.</li>
91
-
<li><strong>KV block codecs</strong> — optional in-place compression with <code>TurboQuantKvCodec</code> (Q4 / Q8) via <code>--paged-kv-quant-bits</code>, trading a small accuracy cost for half/quarter the per-block footprint.</li>
91
+
<li><strong>KV block codecs</strong> — optional in-place compression with <code>TurboQuantKvCodec</code> (Q2 / Q4 / Q8) via <code>--paged-kv-quant-bits</code>, trading a small accuracy cost for half/quarter the per-block footprint.</li>
<tr><td><code>--kv-cache-dtype <type></code></td><td>KV cache precision: <code>f32</code> (default), <code>f16</code>, <code>q8_0</code>, <code>q4_0</code> (~1/7 of f32, for very long 128K–256K contexts; the quantized tiers need the native GGML flash-attention path).</td></tr>
51
51
<tr><td><code>--interactive</code> / <code>-i</code></td><td>Start the interactive REPL.</td></tr>
52
52
<tr><td><code>--system <text></code> / <code>--system-file <path></code></td><td>Seed the system prompt.</td></tr>
<tr><td><code>--prompt <text></code></td><td>Qwen-Image-Edit edit instruction (with <code>--image</code>; the edited PNG goes to <code>--output</code>).</td></tr>
61
+
<tr><td><code>--cfg <F></code></td><td>Qwen-Image-Edit true-CFG guidance scale (auto: 2.5, or 1.0 with a Lightning LoRA; <code><= 1</code> disables the negative pass).</td></tr>
62
+
<tr><td><code>--qwen-image-vae / --qwen-image-vl / --qwen-image-mmproj <path></code></td><td>Override the resolved Qwen-Image-Edit companion GGUFs (VAE / Qwen2.5-VL text encoder / mmproj).</td></tr>
63
+
<tr><td><code>--qwen-image-lora <path></code></td><td>Lightning distillation LoRA (<code>.safetensors</code>) merged into the DiT; auto-derives the step count (e.g. 4 or 8) and switches CFG to 1.0.</td></tr>
<tr><td><code>--qwen-image-lora <path></code></td><td>Qwen-Image-Edit Lightning LoRA merged into the DiT (fewer denoise steps, CFG 1.0).</td></tr>
<tr><td><code>TS_SCHED_PREFILL_CHUNK</code></td><td>Max prefill tokens per step (1024).</td></tr>
118
+
<tr><td><code>TS_SCHED_SOLO_PREFILL_CHUNK</code></td><td>Fresh <code>start_pos = 0</code> prefill chunk for a solo / uncontended request (8192).</td></tr>
119
+
<tr><td><code>TS_SCHED_SOLO_TAIL_PREFILL_CHUNK</code></td><td>Tail chunks of a solo prompt (2048).</td></tr>
120
+
<tr><td><code>TS_SCHED_DECODE_QUANTUM</code></td><td>Decode tokens before a sequence switch (256 = block size).</td></tr>
<tr><td><code>TS_<FAMILY>_BATCHED</code></td><td><code>0</code> forces a family onto the per-sequence path (e.g. <code>TS_GEMMA4_BATCHED</code>, <code>TS_QWEN35_BATCHED</code>).</td></tr>
<tr><td><code>TS_QWEN_IMAGE_LORA / TS_QWEN_IMAGE_LORA_SCALE</code></td><td>Lightning LoRA <code>.safetensors</code> merged into the DiT / scale override.</td></tr>
0 commit comments