Skip to content

Commit bfaa4d3

Browse files
claudezhongkaifu
authored andcommitted
docs: align website pages with current code
- cli/api-reference: add q4_0 KV-cache dtype, --qwen-image-lora, missing Qwen-Image-Edit and paged-kv-quant-bits flags; correct --cfg auto default - api-reference/server: add --prefill-chunk-size, --kv-cache-dtype and qwen-image server flags; new scheduler env vars (solo prefill chunks, decode quantum) and Qwen-Image env vars (companions, LoRA, step cache) - models/features: document Lightning LoRA and whole-step DiT cache - advanced/features: TurboQuant tiers are Q2/Q4/Q8; describe solo-prefill fresh/tail chunk split - search indexes: add keywords for the new features (both languages) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MFvrnoYEae44hnFrd5PPqa
1 parent db57107 commit bfaa4d3

14 files changed

Lines changed: 88 additions & 38 deletions

website/advanced.html

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -75,7 +75,7 @@ <h2 id="perf">Performance optimizations</h2>
7575
<ul>
7676
<li><strong>Fused GPU decode</strong> (Gemma 4) — all transformer layers in a single GGML graph dispatch, cutting CPU↔GPU round-trips from hundreds per token to one (~2.6× over per-op dispatch).</li>
7777
<li><strong>Fused GPU prefill</strong> (Gemma 4) — dense layers run the whole block (norms, QKV, RoPE, attention, FFN, residuals) as one dispatch per layer during prefill.</li>
78-
<li><strong>Chunked prefill</strong> (Gemma 4) — long prompts are split into bounded chunks to avoid O(n²) attention score tensors for sliding-window layers.</li>
78+
<li><strong>Chunked prefill</strong> (Gemma 4) — long prompts are split into bounded chunks to avoid O(n²) attention score tensors for sliding-window layers. A solo (uncontended) request gets a split chunk size: the fresh <code>start_pos = 0</code> chunk runs through the fused whole-graph path at <code>TS_SCHED_SOLO_PREFILL_CHUNK</code> (default 8192), the remaining tail at <code>TS_SCHED_SOLO_TAIL_PREFILL_CHUNK</code> (default 2048).</li>
7979
<li><strong>Fused Qwen 3.5/3.6 attention &amp; FFN</strong> — single-graph fused attention-layer decode, fused prefill attention, fused out-proj + FFN, and fused vision encoder blocks (~15 ops → 2).</li>
8080
<li><strong>Native quantized compute</strong> — Q4_K_M, Q6_K, Q8_0, IQ2_XXS, MXFP4 used directly in matmul without expanding to FP32; a batched <code>AddmmQuantBatch</code> handles multiple sub-weight matmuls in one dispatch.</li>
8181
<li><strong>Batched GPU MoE</strong> — all selected experts (plus the optional shared expert and residual add) collapse into a single GGML graph dispatch per MoE layer.</li>
@@ -88,7 +88,7 @@ <h2 id="memory">Memory optimizations</h2>
8888
<li><strong>Zero-copy file-mapped weights</strong> — the GGUF is memory-mapped and quantized tensors bind directly into native ops, removing a per-tensor copy that roughly doubled the resident set. Example: <code>Qwen3.5-35B-A3B-IQ2_XXS</code> (~10 GB GGUF) runs at ~7 GB peak under Metal instead of ~17 GB.</li>
8989
<li><strong>Best-fit memory pool</strong> with bounded retention (blocks capped at 64 MB, pool at 32 blocks) keeps the working set tight across long runs.</li>
9090
<li><strong>Paged KV block pool with optional SSD spillover</strong> — RAM-capped, LRU-evicted, with content-hash prefix reuse across sessions.</li>
91-
<li><strong>KV block codecs</strong> — optional in-place compression with <code>TurboQuantKvCodec</code> (Q4 / Q8) via <code>--paged-kv-quant-bits</code>, trading a small accuracy cost for half/quarter the per-block footprint.</li>
91+
<li><strong>KV block codecs</strong> — optional in-place compression with <code>TurboQuantKvCodec</code> (Q2 / Q4 / Q8) via <code>--paged-kv-quant-bits</code>, trading a small accuracy cost for half/quarter the per-block footprint.</li>
9292
</ul>
9393

9494
<div class="content-footer">

website/advanced_zh-cn.html

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -75,7 +75,7 @@ <h2 id="perf">性能优化</h2>
7575
<ul>
7676
<li><strong>融合 GPU decode</strong>(Gemma 4)—— 所有 transformer 层在一次 GGML 图分派中完成,把每 token 的 CPU↔GPU 往返从数百次降到一次(相对逐 op 分派约 2.6×)。</li>
7777
<li><strong>融合 GPU prefill</strong>(Gemma 4)—— 在 prefill 期间,dense 层每层一次分派完成整块(归一化、QKV、RoPE、注意力、FFN、残差)。</li>
78-
<li><strong>分块 prefill</strong>(Gemma 4)—— 长提示被切成有界块,避免滑动窗口层产生 O(n²) 的注意力分数张量。</li>
78+
<li><strong>分块 prefill</strong>(Gemma 4)—— 长提示被切成有界块,避免滑动窗口层产生 O(n²) 的注意力分数张量。独占(无竞争)请求采用拆分的分块大小:全新的 <code>start_pos = 0</code> 分块以 <code>TS_SCHED_SOLO_PREFILL_CHUNK</code>(默认 8192)走融合整图路径,其余尾部按 <code>TS_SCHED_SOLO_TAIL_PREFILL_CHUNK</code>(默认 2048)处理。</li>
7979
<li><strong>融合 Qwen 3.5/3.6 注意力与 FFN</strong> —— 单图融合注意力层 decode、融合 prefill 注意力、融合 out-proj + FFN,以及融合视觉编码器块(约 15 op → 2)。</li>
8080
<li><strong>原生量化计算</strong> —— Q4_K_M、Q6_K、Q8_0、IQ2_XXS、MXFP4 直接用于 matmul 而不展开为 FP32;批量 <code>AddmmQuantBatch</code> 在一次分派中处理多个子权重 matmul。</li>
8181
<li><strong>批量 GPU MoE</strong> —— 所有被选专家(加上可选的共享专家与残差相加)在每个 MoE 层折叠为一次 GGML 图分派。</li>
@@ -88,7 +88,7 @@ <h2 id="memory">内存优化</h2>
8888
<li><strong>零拷贝文件映射权重</strong> —— GGUF 被内存映射,量化张量直接绑定进原生运算,去掉了大致使常驻内存翻倍的逐张量拷贝。例如 <code>Qwen3.5-35B-A3B-IQ2_XXS</code>(约 10 GB GGUF)在 Metal 下峰值约 7 GB,而非约 17 GB。</li>
8989
<li><strong>最佳匹配内存池</strong>,带有界保留(块上限 64 MB,池上限 32 块),让长时间运行中工作集保持紧凑。</li>
9090
<li><strong>带可选 SSD 溢出的分页 KV 块池</strong> —— RAM 上限、LRU 淘汰,并跨会话做内容哈希前缀复用。</li>
91-
<li><strong>KV 块编解码器</strong> —— 通过 <code>--paged-kv-quant-bits</code><code>TurboQuantKvCodec</code>(Q4 / Q8)做可选的原地压缩,以很小的精度代价换取每块占用减半 / 减至四分之一。</li>
91+
<li><strong>KV 块编解码器</strong> —— 通过 <code>--paged-kv-quant-bits</code><code>TurboQuantKvCodec</code>Q2 / Q4 / Q8)做可选的原地压缩,以很小的精度代价换取每块占用减半 / 减至四分之一。</li>
9292
</ul>
9393

9494
<div class="content-footer">

website/api-reference.html

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,7 @@ <h2 id="cli-flags">CLI flags — <code>TensorSharp.Cli</code></h2>
4747
<tr><td><code>--mmproj &lt;path&gt;</code></td><td>Multimodal projector GGUF (auto-detected beside the model).</td></tr>
4848
<tr><td><code>--max-tokens &lt;N&gt;</code></td><td>Maximum tokens to generate (default 100).</td></tr>
4949
<tr><td><code>--backend &lt;type&gt;</code></td><td><code>cpu</code>, <code>cuda</code>, <code>mlx</code>, <code>ggml_cpu</code>, <code>ggml_metal</code>, <code>ggml_cuda</code>.</td></tr>
50-
<tr><td><code>--kv-cache-dtype &lt;type&gt;</code></td><td>KV cache precision: <code>f32</code> (default), <code>f16</code>, <code>q8_0</code>.</td></tr>
50+
<tr><td><code>--kv-cache-dtype &lt;type&gt;</code></td><td>KV cache precision: <code>f32</code> (default), <code>f16</code>, <code>q8_0</code>, <code>q4_0</code> (~1/7 of f32, for very long 128K–256K contexts; the quantized tiers need the native GGML flash-attention path).</td></tr>
5151
<tr><td><code>--interactive</code> / <code>-i</code></td><td>Start the interactive REPL.</td></tr>
5252
<tr><td><code>--system &lt;text&gt;</code> / <code>--system-file &lt;path&gt;</code></td><td>Seed the system prompt.</td></tr>
5353
<tr><td><code>--think</code></td><td>Enable thinking / reasoning mode.</td></tr>
@@ -57,6 +57,11 @@ <h2 id="cli-flags">CLI flags — <code>TensorSharp.Cli</code></h2>
5757
<tr><td><code>--seed &lt;N&gt;</code> / <code>--stop &lt;string&gt;</code></td><td>Random seed (-1 = random) / stop sequence (repeatable).</td></tr>
5858
<tr><td><code>--dump-prompt</code></td><td>Render prompt + tokenization and exit.</td></tr>
5959
<tr><td><code>--diffusion-steps / --diffusion-seed / --diffusion-blocks &lt;N&gt;</code></td><td>DiffusionGemma generation controls.</td></tr>
60+
<tr><td><code>--prompt &lt;text&gt;</code></td><td>Qwen-Image-Edit edit instruction (with <code>--image</code>; the edited PNG goes to <code>--output</code>).</td></tr>
61+
<tr><td><code>--cfg &lt;F&gt;</code></td><td>Qwen-Image-Edit true-CFG guidance scale (auto: 2.5, or 1.0 with a Lightning LoRA; <code>&lt;= 1</code> disables the negative pass).</td></tr>
62+
<tr><td><code>--qwen-image-vae / --qwen-image-vl / --qwen-image-mmproj &lt;path&gt;</code></td><td>Override the resolved Qwen-Image-Edit companion GGUFs (VAE / Qwen2.5-VL text encoder / mmproj).</td></tr>
63+
<tr><td><code>--qwen-image-lora &lt;path&gt;</code></td><td>Lightning distillation LoRA (<code>.safetensors</code>) merged into the DiT; auto-derives the step count (e.g. 4 or 8) and switches CFG to 1.0.</td></tr>
64+
<tr><td><code>--paged-kv-quant-bits &lt;0|2|4|8&gt;</code></td><td>TurboQuant paged-KV block compression (0 = off).</td></tr>
6065
<tr><td><code>--benchmark / --bench-prefill / --bench-decode / --bench-runs</code></td><td>Synthetic throughput benchmark.</td></tr>
6166
<tr><td><code>--bench-kvcache / --bench-kv-turns &lt;N&gt;</code></td><td>Multi-turn KV-cache reuse benchmark.</td></tr>
6267
<tr><td><code>--warmup-runs &lt;N&gt;</code></td><td>Throw-away forward passes before timing (default 0).</td></tr>
@@ -83,6 +88,10 @@ <h2 id="server-flags">Server flags — <code>TensorSharp.Server</code></h2>
8388
<tr><td><code>--mtp-draft &lt;N&gt;</code></td><td>Max tokens drafted per speculative step (default 8).</td></tr>
8489
<tr><td><code>--mtp-pmin &lt;f&gt;</code></td><td>Minimum draft-head confidence to keep a token (default 0.75).</td></tr>
8590
<tr><td><code>--mtp-draft-model &lt;path&gt;</code></td><td>Separate MTP draft GGUF (Gemma 4 <code>gemma4-assistant</code>).</td></tr>
91+
<tr><td><code>--prefill-chunk-size &lt;N&gt;</code></td><td>Maximum prefill tokens per scheduler step (sets <code>TS_SCHED_PREFILL_CHUNK</code>).</td></tr>
92+
<tr><td><code>--kv-cache-dtype &lt;type&gt;</code></td><td>KV cache precision: <code>f32</code> (default), <code>f16</code>, <code>q8_0</code>, <code>q4_0</code> (env <code>KV_CACHE_DTYPE</code>).</td></tr>
93+
<tr><td><code>--qwen-image-vae / --qwen-image-vl / --qwen-image-mmproj &lt;path&gt;</code></td><td>Override the resolved Qwen-Image-Edit companion GGUFs.</td></tr>
94+
<tr><td><code>--qwen-image-lora &lt;path&gt;</code></td><td>Qwen-Image-Edit Lightning LoRA merged into the DiT (fewer denoise steps, CFG 1.0).</td></tr>
8695
<tr><td><code>--paged-kv* / --paged-kv-quant-bits</code></td><td>Legacy standalone paged-KV flags (engine now owns KV state).</td></tr>
8796
</tbody>
8897
</table>
@@ -101,15 +110,22 @@ <h2 id="env-vars">Environment variables</h2>
101110
<tr><td><code>TENSORSHARP_REPEAT_PENALTY / _PRESENCE_PENALTY / _FREQUENCY_PENALTY / _SEED</code></td><td>Default penalties and seed.</td></tr>
102111
<tr><td><code>TENSORSHARP_LOG_LEVEL / _LOG_DIR / _LOG_FILE</code></td><td>Logging level, directory, file toggle (CLI + server).</td></tr>
103112
<tr><td><code>DIFFUSION_STEPS / DIFFUSION_MAX_BATCH</code></td><td>DiffusionGemma steps per block / max batched requests.</td></tr>
113+
<tr><td><code>KV_CACHE_DTYPE</code></td><td>Server KV cache precision: <code>f32</code> (default), <code>f16</code>, <code>q8_0</code>, <code>q4_0</code> (= <code>--kv-cache-dtype</code>).</td></tr>
104114
<tr><td><code>TS_SCHED_DISABLE_BATCHED</code></td><td><code>1</code> forces per-sequence KV-swap (= <code>--no-continuous-batching</code>).</td></tr>
105115
<tr><td><code>TS_SCHED_MAX_BATCHED_TOKENS</code></td><td>Per-step token budget (4096).</td></tr>
106116
<tr><td><code>TS_SCHED_MAX_RUNNING_SEQS</code></td><td>Max in-flight sequences (16).</td></tr>
107117
<tr><td><code>TS_SCHED_PREFILL_CHUNK</code></td><td>Max prefill tokens per step (1024).</td></tr>
118+
<tr><td><code>TS_SCHED_SOLO_PREFILL_CHUNK</code></td><td>Fresh <code>start_pos = 0</code> prefill chunk for a solo / uncontended request (8192).</td></tr>
119+
<tr><td><code>TS_SCHED_SOLO_TAIL_PREFILL_CHUNK</code></td><td>Tail chunks of a solo prompt (2048).</td></tr>
120+
<tr><td><code>TS_SCHED_DECODE_QUANTUM</code></td><td>Decode tokens before a sequence switch (256 = block size).</td></tr>
108121
<tr><td><code>TS_SCHED_NUM_BLOCKS / TS_SCHED_BLOCK_SIZE</code></td><td>Engine block-pool size (256) / tokens per block (256).</td></tr>
109122
<tr><td><code>TS_SCHED_PREFIX_CACHE</code></td><td><code>0</code> disables block-hash prefix sharing.</td></tr>
110123
<tr><td><code>TS_&lt;FAMILY&gt;_BATCHED</code></td><td><code>0</code> forces a family onto the per-sequence path (e.g. <code>TS_GEMMA4_BATCHED</code>, <code>TS_QWEN35_BATCHED</code>).</td></tr>
111124
<tr><td><code>TS_MTP_SPEC / TS_MTP_DRAFT / TS_MTP_PMIN / TS_MTP_DRAFT_MODEL</code></td><td>MTP speculative-decoding knobs (mirror the <code>--mtp-*</code> flags).</td></tr>
112125
<tr><td><code>TS_GMTP_NO_FUSED / TS_GMTP_NO_FAST_ROLLBACK / TS_GMTP_BATCHED_TRUNK</code></td><td>Gemma 4 MTP draft-path A/B switches.</td></tr>
126+
<tr><td><code>TS_QWEN_IMAGE_VAE / TS_QWEN_IMAGE_TE / TS_QWEN_IMAGE_MMPROJ</code></td><td>Qwen-Image-Edit companion paths (VAE / Qwen2.5-VL text encoder / mmproj).</td></tr>
127+
<tr><td><code>TS_QWEN_IMAGE_LORA / TS_QWEN_IMAGE_LORA_SCALE</code></td><td>Lightning LoRA <code>.safetensors</code> merged into the DiT / scale override.</td></tr>
128+
<tr><td><code>TS_QWEN_DIT_CACHE / TS_QWEN_DIT_CACHE_MODE</code></td><td>Whole-DiT-step cache: <code>0</code> disables / mode <code>easycache</code>, <code>fbc</code>, <code>both</code>, <code>off</code>.</td></tr>
113129
<tr><td><code>TS_MLX_* </code></td><td>MLX backend tuning: pipelined decode, mlock GGUF, fused KV write, batched MoE decode, memory caps.</td></tr>
114130
<tr><td><code>TENSORSHARP_MLX_LIBRARY / _LIBRARY_DIR</code></td><td>Override the search path for <code>libmlxc</code>.</td></tr>
115131
<tr><td><code>TENSORSHARP_GGML_NO_UPDATE / _GGML_GIT_REF</code></td><td>Skip / pin the ggml source clone on native builds.</td></tr>

0 commit comments

Comments
 (0)