You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
GPU RepeatInterleaveF32 kernel launch segfaults (SIGSEGV, null-pointer CUDA call) -- crashes any GQA model on GPU #180
compute.(*GPUEngine).RepeatInterleave -> gpuapi.(*CUDAKernels).RepeatInterleaveF32 -> the purego/cgo kernel call crashes the whole process with SIGSEGV: segmentation violation, PC=0x0 -- a null-pointer CUDA kernel launch, not a recoverable Go error. Reproduced on ztensor v1.19.2 on an NVIDIA GB10 (aarch64).
This is reachable from zerfoo's layers/attention/grouped_query_attention.go via its "fused RepeatInterleave" decode optimization (added 2026-03-30, zerfoo@641a119b), whenever numQueryHeads != numKeyValueHeads -- true for essentially every modern GQA architecture (Llama, Mistral, Qwen, Gemma) -- and SDPA runs without a flash-decode shortcut. Because it's a hard SIGSEGV rather than an error return, zerfoo's own fallback path (Reshape -> Repeat -> Reshape) is never reached; the type-assertion gate around the fused call cannot protect against a crash inside the call itself.
Repro
zerfoo tests/parity.TestGPUParity_GQA reproduces it directly:
dModel, nQHeads, nKVHeads:=32, 4, 2// ... construct GroupedQueryAttention with these head counts, WithNoRoPE ...gpuOut, err:=gpuGQA.Forward(ctx, gpuInput) // <-- SIGSEGV here
The first argument to the underlying ccall (0x0) looks like a null kernel handle or device pointer -- consistent with the call site at gpu_engine_memory.go:641/666 passing an empty/nil dst (the RepeatInterleave signature accepts a variadic optional dst ...*tensor.TensorNumeric[U]), suggesting the kernel path may not correctly allocate/handle the no-dst-provided case before launching.
Environment
ztensor v1.19.2
NVIDIA GB10, aarch64, CUDA 13.0, driver 580.159.03
purego/dlopen kernel binding (no explicit -tags cuda build)
Current mitigation (in zerfoo, not a real fix)
zerfoo has disabled the fused path unconditionally (fusedRepeatInterleaveEnabled = false in layers/attention/grouped_query_attention.go) and falls back to the slower Reshape -> Repeat -> Reshape sequence, which does not crash and produces numerically correct output (verified maxDiff=7.6e-6 against CPU). This trades away the fused-kernel perf win until this is fixed upstream.
Ask
Root-cause the null-pointer launch in RepeatInterleaveF32's purego/cgo path (internal/cuda/kernels/fused_repeat_interleave_purego.go:23 and internal/gpuapi/cuda_kernels.go:218), likely around how the dst argument is handled when the caller doesn't pre-allocate one. Once fixed and released, zerfoo can bump its ztensor dependency and re-enable the fused path.
the 0x0 is Ccall's fn argument. The input and output device pointers
one frame up are both non-null, so the variadic-dst path is not involved and compute/gpu_engine_memory.go allocates devOut correctly.
What actually happens:
launch_repeat_interleave_f32 is in optionalSyms
(internal/cuda/kernels/purego.go:459). When dlsym misses, openKernelLib
leaves the pointer at 0 and continues — the documented contract is "callers check before use" (purego.go:496-497).
internal/cuda/kernels/fused_repeat_interleave_purego.go:19-26 checked only k == nil, never k.launchRepeatInterleaveF32 == 0, so it called cuda.Ccall(0, ...) → jump to address 0 → SIGSEGV PC=0x0 inside cgo. A
process kill, not an error, which is why the caller's fallback never ran.
dlsym misses because the GB10 loads a prebuilt library, not a build of
the tree. The gate pod sets LD_LIBRARY_PATH=...:/opt/zerfoo/lib and kernelLibPaths[0] resolves there:
That library was built 2026-07-03 and exports 79 symbols; current main builds
to 130. The .cu, the header, and the Makefile SRCS entry were all correct —
only the deployed binary was stale.
It was a class, not a one-off. An AST gate over the package found 34 cuda.Ccall sites launching optional symbols with no zero-check across 14
wrappers; 22 of those symbols are absent from the deployed library today,
so they were live process kills, not latent ones. #183 guards all 34 and adds TestOptionalKernelSymbolsAreGuarded so a new wrapper cannot reintroduce it.
The regression test reproduces this exact trace with no GPU required, and
crashes rather than fails against the unfixed wrapper.
Two things this fix does not do, both worth tracking separately:
The guard makes a stale library safe (clean error → caller falls back). The
fused kernel still cannot run until /opt/zerfoo/lib/libkernels.so is
refreshed. I refreshed it on the DGX for the zerfoo T148.2 proof
(backup: libkernels.so.bak-20260823-pre-t148-ztensor180), but that refresh
re-activates ~50 currently-dormant optional kernels at once and deserves its
own decision rather than riding along on this one.
nsa_attention_f32, kv_dequant_*, iq_dequant_* are declared and resolved
in purego.go but have no .cu in SRCS and no Go caller — dead entries,
left alone.
Correction to my comment above: the count of unguarded optional launches whose symbol was absent from the pre-refresh /opt/zerfoo/lib/libkernels.so is 19, not 22. I had included kv_dequant_q4_f32, kv_dequant_q3_f32, iq_dequant_nl_f32, iq_dequant_3s_f32 and iq_dequant_2xxs_f32, which are declared and resolved in purego.go but have no Go caller at all, so they could never have reached a launch. The other numbers stand: 34 unguarded call sites total, of which 19 were live and 15 latent (that library happened to export them). #183 guards all 34.
Activity
fix(attention): disable crash-inducing fused RepeatInterleave GPU pat…
dndungu commentedon Aug 24, 2026
Root-caused; fix in #183.
Correction to the diagnosis in the issue body: the null pointer is the
function pointer, not
dst. Inthe
0x0isCcall'sfnargument. Theinputandoutputdevice pointersone frame up are both non-null, so the variadic-
dstpath is not involved andcompute/gpu_engine_memory.goallocatesdevOutcorrectly.What actually happens:
launch_repeat_interleave_f32is inoptionalSyms(
internal/cuda/kernels/purego.go:459). Whendlsymmisses,openKernelLibleaves the pointer at
0and continues — the documented contract is"callers check before use" (
purego.go:496-497).internal/cuda/kernels/fused_repeat_interleave_purego.go:19-26checked onlyk == nil, neverk.launchRepeatInterleaveF32 == 0, so it calledcuda.Ccall(0, ...)→ jump to address 0 →SIGSEGV PC=0x0inside cgo. Aprocess kill, not an error, which is why the caller's fallback never ran.
dlsymmisses because the GB10 loads a prebuilt library, not a build ofthe tree. The gate pod sets
LD_LIBRARY_PATH=...:/opt/zerfoo/libandkernelLibPaths[0]resolves there:That library was built 2026-07-03 and exports 79 symbols; current
mainbuildsto 130. The
.cu, the header, and the MakefileSRCSentry were all correct —only the deployed binary was stale.
It was a class, not a one-off. An AST gate over the package found 34
cuda.Ccallsites launching optional symbols with no zero-check across 14wrappers; 22 of those symbols are absent from the deployed library today,
so they were live process kills, not latent ones. #183 guards all 34 and adds
TestOptionalKernelSymbolsAreGuardedso a new wrapper cannot reintroduce it.The regression test reproduces this exact trace with no GPU required, and
crashes rather than fails against the unfixed wrapper.
Two things this fix does not do, both worth tracking separately:
fused kernel still cannot run until
/opt/zerfoo/lib/libkernels.soisrefreshed. I refreshed it on the DGX for the zerfoo T148.2 proof
(backup:
libkernels.so.bak-20260823-pre-t148-ztensor180), but that refreshre-activates ~50 currently-dormant optional kernels at once and deserves its
own decision rather than riding along on this one.
nsa_attention_f32,kv_dequant_*,iq_dequant_*are declared and resolvedin
purego.gobut have no.cuinSRCSand no Go caller — dead entries,left alone.
dndungu commentedon Aug 24, 2026
Correction to my comment above: the count of unguarded optional launches whose symbol was absent from the pre-refresh
/opt/zerfoo/lib/libkernels.sois 19, not 22. I had includedkv_dequant_q4_f32,kv_dequant_q3_f32,iq_dequant_nl_f32,iq_dequant_3s_f32andiq_dequant_2xxs_f32, which are declared and resolved inpurego.gobut have no Go caller at all, so they could never have reached a launch. The other numbers stand: 34 unguarded call sites total, of which 19 were live and 15 latent (that library happened to export them). #183 guards all 34.