Skip to content

GPU RepeatInterleaveF32 kernel launch segfaults (SIGSEGV, null-pointer CUDA call) -- crashes any GQA model on GPU #180

Closed

Description

@dndungu

Summary

compute.(*GPUEngine).RepeatInterleave -> gpuapi.(*CUDAKernels).RepeatInterleaveF32 -> the purego/cgo kernel call crashes the whole process with SIGSEGV: segmentation violation, PC=0x0 -- a null-pointer CUDA kernel launch, not a recoverable Go error. Reproduced on ztensor v1.19.2 on an NVIDIA GB10 (aarch64).

This is reachable from zerfoo's layers/attention/grouped_query_attention.go via its "fused RepeatInterleave" decode optimization (added 2026-03-30, zerfoo@641a119b), whenever numQueryHeads != numKeyValueHeads -- true for essentially every modern GQA architecture (Llama, Mistral, Qwen, Gemma) -- and SDPA runs without a flash-decode shortcut. Because it's a hard SIGSEGV rather than an error return, zerfoo's own fallback path (Reshape -> Repeat -> Reshape) is never reached; the type-assertion gate around the fused call cannot protect against a crash inside the call itself.

Repro

zerfoo tests/parity.TestGPUParity_GQA reproduces it directly:

dModel, nQHeads, nKVHeads := 32, 4, 2
// ... construct GroupedQueryAttention with these head counts, WithNoRoPE ...
gpuOut, err := gpuGQA.Forward(ctx, gpuInput) // <-- SIGSEGV here

Full stack trace (zerfoo main, GB10):

SIGSEGV: segmentation violation
PC=0x0 m=0 sigcode=1 addr=0x0
signal arrived during cgo execution

goroutine ... [syscall]:
runtime.cgocall(0x91e130, ...)
	runtime/cgocall.go:167
github.com/zerfoo/ztensor/internal/cuda._Cfunc_ccall_wrapper(0x0, ...)
	_cgo_gotypes.go:93
github.com/zerfoo/ztensor/internal/cuda.ccall(...)
	internal/cuda/purego_linux_arm64_cgo.go:64
github.com/zerfoo/ztensor/internal/cuda.Ccall(...)
	internal/cuda/purego.go:238
github.com/zerfoo/ztensor/internal/cuda/kernels.RepeatInterleaveF32(0xf06420000900, 0xf06420000a00, 0x1, 0x2, 0x2, 0x8, 0x2, 0x21abf240)
	internal/cuda/kernels/fused_repeat_interleave_purego.go:23
github.com/zerfoo/ztensor/internal/gpuapi.(*CUDAKernels).RepeatInterleaveF32(...)
	internal/gpuapi/cuda_kernels.go:218
github.com/zerfoo/ztensor/compute.(*GPUEngine[...]).RepeatInterleave(...)
	compute/gpu_engine_memory.go:666

The first argument to the underlying ccall (0x0) looks like a null kernel handle or device pointer -- consistent with the call site at gpu_engine_memory.go:641/666 passing an empty/nil dst (the RepeatInterleave signature accepts a variadic optional dst ...*tensor.TensorNumeric[U]), suggesting the kernel path may not correctly allocate/handle the no-dst-provided case before launching.

Environment

  • ztensor v1.19.2
  • NVIDIA GB10, aarch64, CUDA 13.0, driver 580.159.03
  • purego/dlopen kernel binding (no explicit -tags cuda build)

Current mitigation (in zerfoo, not a real fix)

zerfoo has disabled the fused path unconditionally (fusedRepeatInterleaveEnabled = false in layers/attention/grouped_query_attention.go) and falls back to the slower Reshape -> Repeat -> Reshape sequence, which does not crash and produces numerically correct output (verified maxDiff=7.6e-6 against CPU). This trades away the fused-kernel perf win until this is fixed upstream.

Ask

Root-cause the null-pointer launch in RepeatInterleaveF32's purego/cgo path (internal/cuda/kernels/fused_repeat_interleave_purego.go:23 and internal/gpuapi/cuda_kernels.go:218), likely around how the dst argument is handled when the caller doesn't pre-allocate one. Once fixed and released, zerfoo can bump its ztensor dependency and re-enable the fused path.

Activity

  1. dndungu commented on Aug 24, 2026

    @dndungu
    ContributorAuthor

    Root-caused; fix in #183.

    Correction to the diagnosis in the issue body: the null pointer is the
    function pointer, not dst. In

    cuda._Cfunc_ccall_wrapper(0x0, ...)
    kernels.RepeatInterleaveF32(0xf06420000900, 0xf06420000a00, 0x1, 0x2, 0x2, 0x8, 0x2, 0x21abf240)
    

    the 0x0 is Ccall's fn argument. The input and output device pointers
    one frame up are both non-null, so the variadic-dst path is not involved and
    compute/gpu_engine_memory.go allocates devOut correctly.

    What actually happens:

    1. launch_repeat_interleave_f32 is in optionalSyms
      (internal/cuda/kernels/purego.go:459). When dlsym misses, openKernelLib
      leaves the pointer at 0 and continues — the documented contract is
      "callers check before use" (purego.go:496-497).
    2. internal/cuda/kernels/fused_repeat_interleave_purego.go:19-26 checked only
      k == nil, never k.launchRepeatInterleaveF32 == 0, so it called
      cuda.Ccall(0, ...) → jump to address 0 → SIGSEGV PC=0x0 inside cgo. A
      process kill, not an error, which is why the caller's fallback never ran.
    3. dlsym misses because the GB10 loads a prebuilt library, not a build of
      the tree. The gate pod sets LD_LIBRARY_PATH=...:/opt/zerfoo/lib and
      kernelLibPaths[0] resolves there:
    $ nm -D --defined-only /opt/zerfoo/lib/libkernels.so | grep -i repeat
    0000000000015f50 T launch_repeat
    # launch_repeat_interleave_f32 absent
    

    That library was built 2026-07-03 and exports 79 symbols; current main builds
    to 130. The .cu, the header, and the Makefile SRCS entry were all correct —
    only the deployed binary was stale.

    It was a class, not a one-off. An AST gate over the package found 34
    cuda.Ccall sites launching optional symbols with no zero-check across 14
    wrappers; 22 of those symbols are absent from the deployed library today,
    so they were live process kills, not latent ones. #183 guards all 34 and adds
    TestOptionalKernelSymbolsAreGuarded so a new wrapper cannot reintroduce it.

    The regression test reproduces this exact trace with no GPU required, and
    crashes rather than fails against the unfixed wrapper.

    Two things this fix does not do, both worth tracking separately:

    • The guard makes a stale library safe (clean error → caller falls back). The
      fused kernel still cannot run until /opt/zerfoo/lib/libkernels.so is
      refreshed. I refreshed it on the DGX for the zerfoo T148.2 proof
      (backup: libkernels.so.bak-20260823-pre-t148-ztensor180), but that refresh
      re-activates ~50 currently-dormant optional kernels at once and deserves its
      own decision rather than riding along on this one.
    • nsa_attention_f32, kv_dequant_*, iq_dequant_* are declared and resolved
      in purego.go but have no .cu in SRCS and no Go caller — dead entries,
      left alone.
  2. dndungu commented on Aug 24, 2026

    @dndungu
    ContributorAuthor

    Correction to my comment above: the count of unguarded optional launches whose symbol was absent from the pre-refresh /opt/zerfoo/lib/libkernels.so is 19, not 22. I had included kv_dequant_q4_f32, kv_dequant_q3_f32, iq_dequant_nl_f32, iq_dequant_3s_f32 and iq_dequant_2xxs_f32, which are declared and resolved in purego.go but have no Go caller at all, so they could never have reached a launch. The other numbers stand: 34 unguarded call sites total, of which 19 were live and 15 latent (that library happened to export them). #183 guards all 34.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions