perf(kernels): fused softmax, norms and banded SDPA kernels (wasm sidecar + JVM FFM)

CHANGELOG.md

@@ -11,6 +11,16 @@
 - **Explicit SIMD128 bodies for every sidecar matmul kernel**: Q4_K, Q5_K, Q6_K, Q5_0 and Q5_1 join Q8_0 and Q4_0 with hand-written `__wasm_simd128__` paths (the K-quant int8 dots on `i32x4.dot_i16x8_s`, the Q5 fifth-bit plane via a byte swizzle); the dense FP32 GEMM gets a 4×8 register-blocked, k-blocked micro-kernel; BF16 and FP16 widen eight weights per step. BF16 and FP16 also gain NEON bodies (none existed on any tier). Results stay bit-identical in accumulation order to the portable bodies.
 - **Threaded dense kernels**: `skainet_fp32_matmul`, `skainet_bf16_matmul` and `skainet_fp16_matmul` now split the output over the shared row-thread pool like the packed kernels do (the GEMM over the longer side of `C`, the 16-bit kernels over columns) — the mt sidecar's pthreads in a browser, the native pool on the JVM.
 - **`WasmCpuOps.matmul`**: a dense 2-D `A · B` on heap FP32 operands runs the sidecar GEMM; the common `[k, n]` loop no longer serves that shape on wasm (it was the only dense path that never reached the sidecar).
+- **Ternary kernels on the wasm tier**: the sidecar now exports `skainet_ternary_f32_gemv`, `skainet_ternary_lmhead_stage1` and `skainet_bitnet_gemv_tq2_0`, and `WasmTernaryKernelPack` (installed by `WasmSimd.install()`) registers them under the same `KernelDispatch` keys the JVM FFM and Android JNI ternary packs serve — exact FP32 × `BitNet-b1.58`, the fused multi-plane `BitNet-planes` lm_head and int8 × `TQ2_0`. A BitNet weight in a browser previously went to the decoding reference kernel. The portable branch of `skainet_ternary_f32.c` (MSVC and Emscripten) now holds the vendored kernel's 256-entry decode table as a compile-time constant with a SIMD128 body and threads over output rows through the shared row pool; `bitnet_gemv` gains a SIMD128 block body and the same row threading on every tier.
+- **Direct `x · Wᵀ` on the JVM**: `Fp32MatmulWtKernel` / `KernelProvider.matmulFp32Wt()` (and the JVM-only `Fp32MemSegMatmulWtKernel` / `MemSegKernelProvider.matmulFp32WtMemSeg()`) expose `skainet_fp32_matmul_wt`, which the wasm sidecar has served since 0.61.0 and the JVM never bound. `NativeKernelProvider` carries it; `DefaultCpuOpsJvm.matmulWeightTransposed` multiplies a dense `[out, in]` FP32 weight in place through it — a segment-backed weight where it lies, a heap weight after one resident off-heap copy — instead of transposing the weight once into `[k, n]` and caching the copy. The wasm provider exposes the same accessor (`WasmFp32MatmulWtKernel`), and `KernelProvider.supports("matmul_wt", …)` reports it.
+- **Kernel support matrix, fourth table** *Kernel packs (view dispatch)*: the ternary formats and the direct `x · Wᵀ` per platform, which the `matmul` accessors the first table introspects do not cover.
+- **`engine-fp32-linear`** in `jvm-cpu-publish` and `wasm-bench`: a `Linear` layer's `x · Wᵀ` (16 tokens, 768 → 3072) through `ctx.ops.matmulWeightTransposed`; on the JVM `--provider panama` measures the cached-transpose path and `--provider ffm` the direct kernel.
+- **Resident activations on the wasm tier**: `PlatformStorage.offHeapAllocator` (Kotlin/Wasm) binds `MemoryDomain.HOST_OFFHEAP` to a backend's memory; `WasmSimd.install()` binds it to the SIMD128 sidecar's linear memory (`WasmLinearStorage : Storage.OffHeap`, `WasmLinearTensorData : FloatArrayTensorData`), so `StorageCapabilities.current()` reports off-heap support in a browser and a `ForwardScope` frees sidecar activations at `reset()`. `WasmCpuOps` now keeps the outputs of `matmul`/`matmulWeightTransposed` resident and runs `add`/`subtract`/`multiply`/`divide`, `relu`/`sigmoid`/`silu`/`gelu`/`erf`/`tanh` and `sum`/`mean` in the sidecar on resident operands (transcendentals also on heap ones), reading a tensor back to the Kotlin heap only when something outside the sidecar needs it. Ambient outputs are freed when their tensor is collected (`FinalizationRegistry`); `WasmSimd.residentBudgetBytes` bounds what stays resident and `WasmCpuOps.toResident()` uploads an input once. New shared C kernels `skainet_f32_binary` / `skainet_f32_unary` / `skainet_f32_reduce_all` / `skainet_f32_reduce_axis` (SIMD128 and NEON bodies, row-pool threading) are in every native build as well.
+- **`wasm-bench`**: a `laya.chain` case (`x · Wᵀ`, GELU, multiply, bias add, sum, end to end) and `operands=resident` variants of `engine-elementwise-add` and `engine-reductions-sum`.
+- **Fused softmax, normalisations and attention on both tiers**: new shared C kernels `skainet_f32_softmax_rows`, `skainet_f32_layer_norm`, `skainet_f32_rms_norm` and `skainet_f32_sdpa` (`f32_fused.c`, SIMD128 and NEON bodies, row-pool threading) — one pass each where the ops layer ran several allocating scalar passes. `skainet_f32_sdpa` carries the full `scaledDotProductAttention` contract: grouped-query heads, `causal`, the sliding window with `rightContext`, per-batch `keyLengths` and the dense additive mask with its trailing-aligned broadcast. New SPI faces `Fp32SoftmaxKernel` / `Fp32NormKernel` / `Fp32AttentionKernel` behind `KernelProvider.softmaxFp32()` / `normFp32()` / `attentionFp32()` (reported by `supports("softmax" | "layer_norm" | "rms_norm" | "sdpa", ["Float32"])`), served by `NativeKernelProvider` (FFM) and `WasmKernelProvider`. On the JVM, `DefaultCpuOpsJvm` now routes `softmax`/`logSoftmax` over the last dimension (from 1024 elements) and `scaledDotProductAttention` to the native kernels when the FFM provider is loaded, replacing the scalar per-head loop; on wasmJs `WasmCpuOps` runs them in the sidecar with the outputs resident.
+- **`FusedNormOps`** (`skainet-lang-core`): an optional interface an op set may implement beside `TensorOps` — `layerNorm` / `rmsNorm` over the trailing dimensions — which `LayerNormalization` consults for FP32 input before its decomposed chain. `DefaultCpuOpsJvm` (native kernel loaded) and `WasmCpuOps` implement it; recording and differentiating op sets never see it, so recorded graphs and gradients are unchanged.
+- **`engine-softmax`, `engine-layer-norm`, `engine-sdpa`** in `jvm-cpu-publish` and `wasm-bench` (`--provider ffm` measures the fused kernels on the JVM, `panama`/`scalar` the common paths); the kernel support matrix's *Kernel packs* table gains rows for the three fused ops.
+- **Native library no longer references libmvec**: GCC with `-ffast-math` vectorised the `expf`/`tanhf`/`erf` loops of `f32_ops.c` (and the new softmax) into glibc libmvec calls the shared library is not linked against — the symbol resolves lazily, so the first JVM call into such a kernel terminated the process (exit 127, no crash log). The transcendentals in element loops now go through opaque scalar wrappers in `skainet_simd.h`; `libskainet_kernels.so` has no `_ZGV*` reference left and stays loadable on glibc builds without libmvec.
 - **`wasm-bench` runs the `engine-*` scenarios** of the JVM publish harness (`engine-fp32-gemm`, `engine-kernel-matmul`, `engine-bf16-matmul`, `engine-q8-matmul`, `engine-q4-gemm`, `engine-ternary-f32-gemv`, `engine-elementwise-add`, `engine-reductions-sum/mean`) with identical shapes and metrics on every tier it measures, so the JVM and wasm numbers sit in one table.
 
 ## [0.61.0] - 2026-10-16

build-logic/convention/src/main/kotlin/GenerateKernelMatrixTask.kt

@@ -128,6 +128,37 @@ abstract class GenerateKernelMatrixTask : DefaultTask() {
                 appendLine("|===")
                 appendLine("")
             }
+            if (module.packs.isNotEmpty()) {
+                appendLine("== Kernel packs (view dispatch)")
+                appendLine("")
+                appendLine(
+                    "Kernels that are not reached through the `matmul` accessors above: the ternary formats " +
+                        "are served by `ViewKernelPack`s keyed in `KernelDispatch` (`BitNet-b1.58` with exact " +
+                        "FP32 activations, the fused multi-plane `BitNet-planes` lm_head, `TQ2_0` with int8 " +
+                        "activations), and the dense `x · Wᵀ` of a `Linear` layer by the `matmulFp32Wt` " +
+                        "accessor, which multiplies a row-major `[out, in]` weight in place instead of " +
+                        "transposing and caching a copy. The fused row kernels — softmax / log-softmax, layer " +
+                        "norm / RMS norm and scaled dot-product attention with its structural masks and " +
+                        "grouped-query heads (`softmaxFp32`, `normFp32`, `attentionFp32`) — run those ops in one " +
+                        "pass where the ops layer otherwise runs its scalar loops or the normalisation's " +
+                        "decomposition. The scalar reference kernels serve every cell that is empty here (`—`), " +
+                        "at reference speed; the common row-dot serves `x · Wᵀ` there.",
+                )
+                appendLine("")
+                appendLine("[cols=\"$colSpec\", options=\"header\"]")
+                appendLine("|===")
+                append("| Kernel ")
+                module.platforms.forEach { append("| $it ") }
+                appendLine("")
+                appendLine("")
+                module.packs.forEach { fmt ->
+                    append("| `${fmt.name}` ")
+                    module.platforms.forEach { p -> append("| ${fmt.byPlatform[p] ?: "—"} ") }
+                    appendLine("")
+                }
+                appendLine("|===")
+                appendLine("")
+            }
             appendLine(
                 "See also the eager backends & kernels mindmap " +
                     "(xref:explanation/eager-execution.adoc[]) for the narrative overview and gaps.",

build-logic/convention/src/main/kotlin/models/KernelSupportModels.kt

@@ -23,6 +23,12 @@ data class KernelSupportModule(
      */
     val mapped: List<KernelFormatSupport> = emptyList(),
     val resident: List<KernelFormatSupport> = emptyList(),
+    /**
+     * Kernel packs (view dispatch): the ternary formats and the direct dense `x · Wᵀ`, served by
+     * `ViewKernelPack`s / the `matmulFp32Wt` accessor rather than the `matmul` accessors. Absent in
+     * older JSON.
+     */
+    val packs: List<KernelFormatSupport> = emptyList(),
 )
 
 @Serializable

docs/modules/ROOT/pages/contributing/benchmarks.adoc

@@ -33,13 +33,17 @@ microbench tests under `skainet-backends/skainet-backend-native-cpu`:
 |===
 | Scenario | What it exercises | Unit | Direction
 | engine-fp32-gemm        | `ctx.ops.matmul` end-to-end FP32 square matmul              | GFLOPS        | higher is better
+| engine-fp32-linear      | `ctx.ops.matmulWeightTransposed` — a `Linear` layer's `x · Wᵀ` with the weight stored `[out, in]` (panama: cached transpose + GEMM; ffm: native direct kernel, no transposed copy) | GFLOPS | higher is better
 | engine-q4-gemm          | `PanamaVectorQ4KMatmulKernel` F32 × Q4_K matmul at LLM shapes | GOP/s        | higher is better
 | engine-kernel-matmul    | Direct `Fp32MatmulKernel` (scalar vs panama-vector)         | GFLOPS        | higher is better
 | engine-bf16-matmul      | `Bf16MatmulKernel` F32 × BF16-packed matmul (scalar vs panama-vector) | GFLOPS | higher is better
 | engine-q8-matmul        | `Q8_0MatmulKernel` F32 × Q8_0-packed matvec (scalar vs panama-vector) | GOP/s | higher is better
 | engine-elementwise-add  | `ctx.ops.add` on 1M FP32 elements                           | M elements/s  | higher is better
 | engine-reductions-sum   | `ctx.ops.sum` on 1M FP32 elements                           | M elements/s  | higher is better
 | engine-reductions-mean  | `ctx.ops.mean` on 1M FP32 elements                          | M elements/s  | higher is better
+| engine-softmax          | `ctx.ops.softmax` over the last dimension of `[1024, 1024]` FP32 (ffm: fused native rows; panama/scalar: the common loop) | M elements/s  | higher is better
+| engine-layer-norm       | a `LayerNormalization` layer with affine over `[1024, 768]` FP32 (ffm: one fused native pass; panama/scalar: the decomposed elementwise chain) | M elements/s  | higher is better
+| engine-sdpa             | `ctx.ops.scaledDotProductAttention`, causal, 12 heads × 150 × 150, head dim 64 (ffm: fused native kernel; panama/scalar: the common per-head kernel under the context's schedule) | GFLOPS (`4·h·s²·d`) | higher is better
 |===
 
 Each scenario is wired up as a Phoronix test profile under

docs/modules/ROOT/pages/explanation/eager-execution.adoc

@@ -29,7 +29,8 @@ mindmap
       Native FFM
         JVM only — C kernels via CMake
         Q4_K MemSeg zero-copy
-        Q6_K missing
+        direct x·Wᵀ, no transposed copy
+        ternary LUT gemv and planes lm_head
       Native JNI
         Android only — same C kernels via JNI
         two .so tiers, cpuinfo-gated armv8-a vs armv8.2+dotprod
@@ -47,7 +48,7 @@ mindmap
       Android — scalar, Native-JNI
       Native linux — scalar, Native-cinterop
       Native apple — scalar, Accelerate, Native-cinterop
-      JS and WASM — scalar; packed weights relaid once into kernel feed order (SKEEP-007)
+      JS and WASM — scalar; wasmJs plus the SIMD128 sidecar pack: every matmul format, ternary, direct x·Wᵀ, resident activations, fused softmax / norms / attention (SKEEP-007)
 ----
 
 == Kernel × provider (matmul, FP32 activations)
@@ -56,6 +57,7 @@ mindmap
 |===
 | Weight format | Scalar (all targets) | Panama (JVM) | Native FFM (JVM) | Native JNI (Android) | Native cinterop (K/N) | Wasm SIMD128 (wasmJs)
 | FP32 | ✅ | ✅ | ✅ | ❌ | ❌ | ✅
+| FP32 `x · Wᵀ` direct (weight `[out, in]`, no transposed copy) | row-dot | ❌ (cached transpose) | ✅ | ❌ | ❌ | ✅
 | BF16 | ✅ | ✅ | ✅ | ❌ | ❌ | ✅
 | FP16 | ✅ | ✅ | ✅ | ❌ | ❌ | ✅
 | Q8_0 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅
@@ -65,9 +67,9 @@ mindmap
 | Q5_K | ✅ | ✅ | ✅ | ✅ | ✅ | ✅
 | Q5_1 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅
 | Q5_0 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅
-| TQ2_0 / BitNet b1.58 (int8 activations) | ✅ | — | — | ✅ NEON | — | —
-| BitNet b1.58 (exact FP32 activations, vendored NeoGPU kernel) | ✅ | ✅ FFM | — | ✅ JNI + fused lm_head | ✅ K/N cinterop | ❌ (reference)
-| BITNET_PLANES (multi-plane ternary packing) | ✅ | via kernel pack | — | via kernel pack | via kernel pack | ❌ (reference)
+| TQ2_0 / BitNet b1.58 (int8 activations) | ✅ | — | — | ✅ NEON | — | ✅ SIMD128
+| BitNet b1.58 (exact FP32 activations, LUT gemv) | ✅ | — | ✅ FFM (vendored NeoGPU) | ✅ JNI | ✅ K/N cinterop | ✅ SIMD128 LUT
+| BITNET_PLANES (multi-plane ternary packing, fused lm_head) | ✅ | — | ✅ FFM (vendored NeoGPU) | ✅ JNI | ✅ K/N cinterop | ✅ SIMD128 LUT
 | Q2_K / Q3_K / Q8_K / IQ4 | ❌ (dequant to FP32 only) | ❌ | ❌ | ❌ | ❌ | ❌
 |===
 
@@ -78,7 +80,24 @@ columns are mutually exclusive per platform rather than stacked. The Wasm SIMD12
 sidecar pack (`skainet-backend-wasm-simd`), present only after `WasmSimd.install()`; every matmul
 kernel in it has an explicit SIMD128 body (the K-quant int8 dots on `i32x4.dot_i16x8_s`, a 4×8
 register-blocked dense GEMM, eight-wide BF16/FP16 widening), and the dense kernels thread over the
-sidecar's pthread pool like the packed ones when the page is cross-origin isolated.
+sidecar's pthread pool like the packed ones when the page is cross-origin isolated. The ternary
+kernels in the sidecar are the portable mirror of the vendored NeoGPU kernel — the same 256-entry
+decode table as a compile-time constant, read four codes at a time in SIMD128 — registered under the
+same dispatch keys as the FFM and JNI ternary packs (`WasmTernaryKernelPack`).
+
+The dense `x · Wᵀ` row is the product a `Linear` layer computes. The JVM and the wasm tier multiply
+the `[out, in]` weight in place through `skainet_fp32_matmul_wt` (`KernelProvider.matmulFp32Wt`;
+the JVM makes a heap weight resident off-heap once, the sidecar keeps it in its linear memory) and
+never materialise the transposed copy the Panama path caches per weight.
+
+The fused row kernels — softmax / log-softmax over the last dimension, layer norm and RMS norm
+(what `LayerNormalization` calls through the optional `FusedNormOps` hook), and scaled dot-product
+attention with its structural masks and grouped-query heads — are one C pass each (`f32_fused.c`),
+served by the JVM FFM provider (`softmaxFp32`, `normFp32`, `attentionFp32`) and the wasm sidecar
+alike. Where neither is present the ops layer runs its common implementation: attention scalar per
+head and parallel over heads through the schedule, the normalisation as its decomposed elementwise
+chain. The decomposition is also what a recording or differentiating op set always sees, so the
+hook changes no recorded graph and no gradient.
 
 Packed weights reach a kernel through the registry by declaring their
 xref:explanation/packed-weight-layout.adoc[block order]; a canonical weight is relayouted once,
@@ -93,19 +112,17 @@ not per call.
 | *Android* | Scalar + Native-JNI | Panama (`jdk.incubator.vector`) and FFM (`java.lang.foreign`) are JDK-only — ART has neither, so Android's native tier is JNI, not a degraded JVM. Includes the NEON `bitnet_gemv` pack.
 | *Kotlin/Native — linux x64/arm64* | Scalar + Native-cinterop | Static archive embedded in the klib; manual `installNativeKernels()`, no ServiceLoader on K/N.
 | *Kotlin/Native — macOS/iOS* | Scalar + Apple Accelerate + Native-cinterop | Accelerate covers dense FP32 and reductions; cinterop covers packed quant with runtime `FEAT_DotProd` dispatch, so one archive serves A12 through M-series.
-| *JS / WASM (Js, Wasi)* | Scalar; wasmJs + `skainet-backend-wasm-simd` | No SIMD in Kotlin code: packed weights are relaid once per weight into kernel feed order and served by the scalar packed kernels; dense `[out, in]` weights are multiplied in place without a transpose copy (SKEEP-007 Tier 0). On wasmJs, `WasmSimd.install()` adds the SIMD128 sidecar pack — the engine's C kernels compiled by Emscripten, with pthreads on cross-origin-isolated pages (SKEEP-007 Tier 1).
+| *JS / WASM (Js, Wasi)* | Scalar; wasmJs + `skainet-backend-wasm-simd` | No SIMD in Kotlin code: packed weights are relaid once per weight into kernel feed order and served by the scalar packed kernels; dense `[out, in]` weights are multiplied in place without a transpose copy (SKEEP-007 Tier 0). On wasmJs, `WasmSimd.install()` adds the SIMD128 sidecar pack — the engine's C kernels compiled by Emscripten, with pthreads on cross-origin-isolated pages (SKEEP-007 Tier 1) — and binds `HOST_OFFHEAP` to the sidecar's linear memory, so matmul outputs, elementwise arithmetic, activations, reductions, softmax, the normalisations and attention stay resident there between kernels and cross back to the Kotlin heap only when read.
 |===
 
 Packed-quant matmul works on every target: those formats have a `commonMain` scalar kernel and
 `DefaultCpuOpsBase` dispatches packed weights through the registry.
 
 == Gaps
 
-* *Wasm ternary / BitNet* — the ternary gemv and planes lm-head kernels are not in the sidecar
-build yet, so BitNet models in a browser run the decoding reference kernel.
-* *Wasm beyond matmul* — elementwise, activations, reductions, normalisation and attention are the
-common scalar Kotlin paths on wasmJs; Kotlin/Wasm has no SIMD types and no shared-memory threads, so
-these follow the matmuls into the sidecar (SKEEP-007 rollout step 3).
+* *Wasm beyond the fused set* — convolutions (`conv1d/2d/3d`) are still the common scalar Kotlin
+path on wasmJs (im2col + the Vector API on the JVM); everything else a transformer forward pass
+runs between its matmuls is in the sidecar with the activations resident in its linear memory.
 * *Native JNI dense FP32/BF16* — the Android JNI provider has no GEMM shim, so dense ops fall
 through to scalar there whichever `.so` tier loaded (SKaiNET#920).
 * *Dense FP32/BF16 SIMD on Kotlin/Native linux* — still the scalar floor; Apple has Accelerate

docs/modules/ROOT/pages/explanation/kernel-selection.adoc

@@ -91,7 +91,7 @@ platform support is absent (a missing native library, no vector unit).
 
 | Kotlin/Native, wasm, JS
 | manual — e.g. `installNativeKernels()`; on wasmJs `WasmSimd.install()` (SKEEP-007)
-| manual — call the pack's `install()` yourself; `WasmSimd.install()` registers the browser pack and its provider
+| manual — call the pack's `install()` yourself; `WasmSimd.install()` registers the browser pack, its ternary pack and its provider
 |===
 
 Without a row-major pack, a canonical packed weight is relaid **once per weight** into kernel feed