纯 CUDA 实现 GPT2 推理,逐步优化算子与架构,最终在单请求下推理速度领先 vLLM 约 1.5 倍。逐算子、逐迭代的进行:NCU 视察、性能解析与记录报告。
-
Updated
Nov 13, 2026 - Cuda
纯 CUDA 实现 GPT2 推理,逐步优化算子与架构,最终在单请求下推理速度领先 vLLM 约 1.5 倍。逐算子、逐迭代的进行:NCU 视察、性能解析与记录报告。
Ultra-High-Throughput Transformers Engine: NVIDIA FP8/TMA, Zero-Allocation Slotted Static KV Cache, Sub-5µs CUDA Graphs, Fused Logits Sampling, Tree Speculative Decoding, Split-KV Flash-Decoding & Apple Silicon Metal (MSL) Simdgroup Kernels
CuTileForge v3.0.0: Native tile-fusion compiler & universal quantization engine on NVIDIA CuTile. Zero-ATen CUDA-graph decode at 172 tok/s, 2,377 tok/s batch serving, 13 model families, 70B hybrid on a single 32 GB GPU (sm_89 ↔ sm_120).
From-scratch C++/CUDA LLM inference engine (SmolLM2-360M) on a GTX 1650 Ti: custom GEMV at 97% of bandwidth, split-KV attention, INT8 weights, cuBLAS autotuning, verified against HuggingFace
Benchmark comparing Flash-Decoding-style chunked KV-parallel attention against standard SDPA at decode time (q_len=1) on RTX 2070, measuring the crossover point where KV-parallelism outperforms fused attention kernels.
C++/CUDA inference engine for Qwen2.5-Coder-0.5B, written from scratch: INT4 GEMV/GEMM kernels, Flash-Decoding, a 24-layer decode engine and lossless speculative decoding. Verified against Hugging Face; benchmarked against bandwidth ceilings, PyTorch and llama.cpp.
To associate your repository with the flash-decoding topic, visit your repo's landing page and select "manage topics."