Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,26 @@

## [Unreleased]

Headline: **model-specific detector code leaves core; the reusable detection pieces stay.**

### Added

- **`Letterbox`** (`skainet-data-media`, `sk.ainet.data.media`): aspect-preserving fit into a square canvas without a platform image. `Letterbox.fit(width, height, targetSize)` computes the scale and offsets, `fillNchw(rgb888, padValue = 114)` letterboxes packed RGB888 bytes into a planar `[3, size, size]` float array in `[0, 1]` (bilinear), `toSource(box)` maps a canvas box back to source pixels, clamped.
- **`BoundingBox`, `ScoredBox`, `nonMaxSuppression`** (`skainet-data-media`): pixel boxes with `iou`, `fromCenter`, and greedy, class-aware-by-default NMS with `maxDetections`.

### Fixed

- **StableHLO `softmax` over a non-last axis** (`skainet-compile-hlo`): the converter read the axis from `parameters["axis"]`, the tape records it as `dim`, so every exported `softmax(t, dim)` reduced over the last axis. It now reads `dim` (`axis` still accepted).
- **StableHLO `logSoftmax`**: recorded as `softmax` with `log = true`, it was exported as a plain softmax; it now lowers to `x - max - log(sum(exp(x - max)))`.

### Removed

- **`skainet-model-yolo`**: the YOLOv8 graph, config and decoder move to their cartridge blueprint (`detection-yolov8-iree` in SKaiNET-cartridge-blueprints), which owns the model end to end. The 0.58.0–0.60.0 artifacts stay on Maven Central. Its letterbox metadata and NMS are replaced by `Letterbox` and `nonMaxSuppression` above.
- **`yolo.class_names`, `yolo.names`, `yolo.num_classes`, `yolo.input_size`** are no longer read by `GgufModelMetadata`; the `general.*` and `model.*` keys are.

### Deprecated

- **`ModelArchitecture.YOLO`**: nothing in core maps to it any more; removed in the next minor release.
### Fixed

- **Multi-row activations through the native Q8_0 / Q4_0 / Q5_0 / Q5_1 kernels returned zeros for every row but the last.** The FFM bindings of these four kernels staged a zero-filled output segment of `outputOffset + outputDim` floats and copied the whole reach back from index 0, so when `DefaultCpuOpsJvm` serves a `[rows, in] × packed [out, in]ᵀ` product row by row into one array, each call wiped the rows computed before it. A single-row product (a decode step) was right, which is why the GGUF decode paths never showed it; the token rows of an encoder on the quantize-at-load lane (Laya on Q8_0 with `skainet-backend-native-cpu` on the classpath) came back uniform. The bindings now copy back exactly the slice they produced, like the K-quant bindings always did. `NativeQ8CanonicalWeightTransposedReproTest` covers every packed format for 1, 7 and 50 rows against the dequant ground truth, with the native provider alone and with the default registry, and the SPI contract that a call at an output offset writes only its own row.
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -243,7 +243,7 @@ Runnable examples:
### Neural Network DSL

- **Sequential**: `nn { input(); dense(); relu(); dense() }`
- **DAG / Graph**: arbitrary wiring with `dag { }` for ResNet, YOLO-style architectures
- **DAG / Graph**: arbitrary wiring with `dag { }` for ResNet-style skip connections and multi-scale detector heads
- Layers: Dense, Conv1d/2d/3d, MaxPool, AvgPool, BatchNorm, Dropout, LeakyReLU, ELU
- KAN (Kolmogorov–Arnold Networks) layer (experimental)
- Autograd engine with reverse-mode gradients, SGD and Adam/AdamW optimizers
Expand Down
1 change: 0 additions & 1 deletion build.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -226,7 +226,6 @@ dependencies {

// Other
dokka(project(":skainet-pipeline"))
dokka(project(":skainet-models:skainet-model-yolo"))
}

// Copy the Dokka-generated HTML aggregate into the Antora site
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ The plugin is applied to 21 library modules:
|skainet-backends |`skainet-backend-cpu`
|skainet-data |`skainet-data-api`, `skainet-data-transform`, `skainet-data-simple`, `skainet-data-media`
|skainet-io |`skainet-io-core`, `skainet-io-gguf`, `skainet-io-image`, `skainet-io-onnx`, `skainet-io-safetensors`
|Other |`skainet-pipeline`, `skainet-model-yolo`
|Other |`skainet-pipeline`
|===

*Excluded:* `skainet-bom` (no source), `skainet-apps/++*++`, `skainet-test/++*++`, benchmarks, and `skainet-lang-ksp-processor` (internal).
Expand Down
4 changes: 2 additions & 2 deletions docs/modules/ROOT/pages/tutorials/graph-dsl.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -77,9 +77,9 @@ if (validation is ValidationResult.Valid) {
}
----

=== YOLO-style Example
=== Multi-scale Head Example

The Graph DSL is particularly useful for complex architectures like YOLO heads:
The Graph DSL is particularly useful for complex architectures such as multi-scale detector heads:

[source,kotlin]
----
Expand Down
22 changes: 12 additions & 10 deletions docs/modules/ROOT/pages/tutorials/image-data-getting-started.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ This guide shows how the three image-oriented modules fit together:
By the end you will:

. Load an image from disk on the JVM.
. Letterbox it into a YOLO-style `(1, 3, H, W)` tensor.
. Letterbox it into a detector-style `(1, 3, H, W)` tensor.
. Wrap that tensor in the `Image` metadata API.

=== Add the modules
Expand Down Expand Up @@ -69,16 +69,16 @@ println("Loaded image: ${width}x${height}")

`platformImageSize(...)` is the portable way to inspect dimensions.

=== Step 2: Letterbox an image for YOLO
=== Step 2: Letterbox an image for a detector

Object detectors such as YOLO commonly keep aspect ratio, resize the
Single-shot object detectors commonly keep aspect ratio, resize the
image to fit inside a square canvas, and pad the remaining area with a
constant color. This is usually called *letterboxing*.

The image transform DSL makes that flow explicit. `toTensor(ctx)`
converts the letterboxed platform image to an RGB tensor with shape
`(1, 3, H, W)`, and `rescale(ctx, 255f)` moves pixel values into the
`[0, 1]` range expected by most YOLOv8-style exports.
`[0, 1]` range most detector exports expect.

[source,kotlin]
----
Expand Down Expand Up @@ -107,7 +107,7 @@ val right = padX - left
val top = padY / 2
val bottom = padY - top

val yoloInput = pipeline<PlatformBitmapImage>()
val detectorInput = pipeline<PlatformBitmapImage>()
.resize(resizedWidth, resizedHeight)
.pad(
top = top,
Expand All @@ -122,7 +122,7 @@ val yoloInput = pipeline<PlatformBitmapImage>()
.rescale(ctx, 255f)
.apply(input)

println("Tensor shape: ${yoloInput.shape}")
println("Tensor shape: ${detectorInput.shape}")
println("Letterbox scale: $scale")
println("Top/left padding: $top / $left")
----
Expand All @@ -147,7 +147,7 @@ import sk.ainet.data.media.Image
import sk.ainet.data.media.ImageLayout

val image = Image.fromTensor(
tensor = yoloInput,
tensor = detectorInput,
layout = ImageLayout.NCHW,
colorSpace = ColorSpace.RGB
)
Expand All @@ -164,9 +164,11 @@ tracking which axis is width, height, or channels.

[NOTE]
====
If you use `skainet-model-yolo`, the same `scale`, `left`, and `top`
values from the letterbox step are the metadata needed to remap decoded
detections back to the original image coordinates.
The same geometry is available without a platform image as
`sk.ainet.data.media.Letterbox` (`skainet-data-media`): `Letterbox.fit(...)`
computes `scale` and the offsets, `fillNchw(rgb888)` letterboxes raw RGB888
bytes, and `toSource(box)` remaps a detection back to the original image.
`nonMaxSuppression(...)` in the same module does class-aware NMS.
====

=== Step 4: Start from a tensor you already have
Expand Down
1 change: 0 additions & 1 deletion settings.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,6 @@ include("skainet-io:skainet-io-onnx")

// ====== models
include("skainet-models:skainet-model-identity")
include("skainet-models:skainet-model-yolo")

// ====== FRONT ENDS (SKEEP-006)
include("skainet-frontend:skainet-frontend-core")
Expand Down
1 change: 0 additions & 1 deletion skainet-apps/skainet-tensor-tools/build.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,6 @@ dependencies {
implementation(project(":skainet-io:skainet-io-safetensors"))
implementation(libs.kotlinx.io.core)
implementation(project(":skainet-backends:skainet-backend-cpu"))
implementation(project(":skainet-models:skainet-model-yolo"))
implementation(libs.pbandk.runtime)

testImplementation(kotlin("test"))
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,13 @@ import kotlin.test.assertEquals
import kotlin.test.assertTrue

/**
* Smoke-test a YOLOv8-like head expressed with the DAG DSL:
* Smoke-test a multi-scale detector-style head expressed with the DAG DSL:
* stem conv -> downsample conv -> upsample -> 1x1 head conv, with multiple parameter/constant nodes.
*/
class YoloStyleGraphDslTest {
class MultiScaleHeadGraphDslTest {

@Test
fun yoloish_head_graph_builds_and_validates() {
fun multi_scale_head_graph_builds_and_validates() {
val program = dag {
val input = input<FP32>("input", TensorSpec("input", listOf(1, 3, 640, 640), "FP32"))

Expand All @@ -35,7 +35,7 @@ class YoloStyleGraphDslTest {
val bHead = constant<FP32, Float>("b_head") { shape(3) { zeros() } }
val head = conv2d(up, wHead, bHead, stride = 1 to 1, padding = 0 to 0)

output(c2, head) // multi-scale outputs like YOLO heads
output(c2, head) // multi-scale outputs, one per feature level
}

val graph = program.toComputeGraph() as DefaultComputeGraph
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,7 @@ public class ActivationOperationsConverter : StableHloOperationConverter {
/**
* Convert softmax activation using real reductions and broadcast_in_dim.
* softmax(x) = exp(x - max(x)) / sum(exp(x - max(x)))
* log_softmax(x) = (x - max(x)) - log(sum(exp(x - max(x)))) (parameter `log = true`)
*
* The max and sum terms are lowered to stablehlo.custom_call @reduce_max
* and @reduce_sum — matching ReductionOperationsConverter's style — then
Expand Down Expand Up @@ -124,8 +125,12 @@ public class ActivationOperationsConverter : StableHloOperationConverter {
val inputShape = node.inputs.firstOrNull()?.shape ?: outputSpec?.shape ?: emptyList()
val rank = inputShape.size

// Normalize axis against rank. Default to the last dimension.
val rawAxis = node.operation.parameters["axis"] as? Int ?: -1
// Normalize axis against rank. Default to the last dimension. The tape records the axis as `dim`
// (TensorOps.softmax / logSoftmax); `axis` is accepted for hand-built graphs.
val params = node.operation.parameters
val rawAxis = (params["dim"] ?: params["axis"]) as? Int ?: -1
// logSoftmax is recorded as softmax with `log = true`: x - max - log(sum(exp(x - max))).
val log = params["log"] == true
val axis = when {
rank == 0 -> 0
rawAxis < 0 -> rank + rawAxis
Expand Down Expand Up @@ -215,7 +220,13 @@ public class ActivationOperationsConverter : StableHloOperationConverter {
} else {
add("$sumBroadcast = stablehlo.broadcast_in_dim $sumValue, dims = [$broadcastDims] : ($reducedType) -> $outputType")
}
add("$resultValue = stablehlo.divide $expValue, $sumBroadcast : $outputType")
if (log) {
val logSum = context.nextTempValue()
add("$logSum = stablehlo.log $sumBroadcast : $outputType")
add("$resultValue = stablehlo.subtract $shiftedValue, $logSum : $outputType")
} else {
add("$resultValue = stablehlo.divide $expValue, $sumBroadcast : $outputType")
}
}

operations.forEach { context.emitOperation(it) }
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -216,6 +216,28 @@ class ActivationOperationsConverterTest {
assertTrue(module.content.contains("stablehlo.divide"))
}

@Test
fun testSoftmaxReducesOverTheRecordedDim() {
// The tape records TensorOps.softmax(t, dim) as parameters["dim"]; a non-last axis must be honoured.
val graph = createActivationGraphWithParameters("softmax", mapOf("dim" to 1), listOf(1, 16, 4, 8))
val module = StableHloConverterFactory.createExtended().convert(graph, "test_softmax_dim")

assertTrue(module.content.contains("applies stablehlo.maximum across dimensions = [1]"), module.content)
assertTrue(module.content.contains("applies stablehlo.add across dimensions = [1]"), module.content)
assertTrue(module.content.contains("-> tensor<1x4x8xf32>"), "reduced type drops the softmax axis")
assertTrue(module.content.contains("dims = [0, 2, 3]"), "broadcast back over the kept axes")
}

@Test
fun testLogSoftmaxSubtractsTheLogSum() {
// TensorOps.logSoftmax is recorded as softmax with log = true.
val graph = createActivationGraphWithParameters("softmax", mapOf("dim" to -1, "log" to true), listOf(2, 3))
val module = StableHloConverterFactory.createExtended().convert(graph, "test_log_softmax")

assertTrue(module.content.contains("stablehlo.log "), module.content)
assertFalse(module.content.contains("stablehlo.divide"), "log-softmax must not normalize by division")
}

@Test
fun testActivationChaining() {
// Test chaining multiple activation functions
Expand Down
1 change: 1 addition & 0 deletions skainet-data/skainet-data-media/build.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import org.jetbrains.kotlin.gradle.dsl.JvmTarget
plugins {
alias(libs.plugins.kotlinMultiplatform)
alias(libs.plugins.androidMultiplatformLibrary)
alias(libs.plugins.vanniktech.mavenPublish)
id("sk.ainet.dokka")
}

Expand Down
2 changes: 2 additions & 0 deletions skainet-data/skainet-data-media/gradle.properties
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
POM_ARTIFACT_ID=skainet-data-media
POM_NAME=skainet data media
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
package sk.ainet.data.media

import kotlin.math.max
import kotlin.math.min

/**
* Axis-aligned box in pixel coordinates, origin top-left, `x1 <= x2` and `y1 <= y2` for a
* non-empty box.
*/
public data class BoundingBox(
val x1: Float,
val y1: Float,
val x2: Float,
val y2: Float
) {
public val width: Float get() = max(0f, x2 - x1)
public val height: Float get() = max(0f, y2 - y1)
public val area: Float get() = width * height

/** Intersection over union with [other]; 0 when the union is empty. */
public fun iou(other: BoundingBox): Float {
val iw = max(0f, min(x2, other.x2) - max(x1, other.x1))
val ih = max(0f, min(y2, other.y2) - max(y1, other.y1))
val inter = iw * ih
val union = area + other.area - inter
return if (union <= 0f) 0f else inter / union
}

public companion object {
/** Box from a center point and size. */
public fun fromCenter(cx: Float, cy: Float, w: Float, h: Float): BoundingBox =
BoundingBox(cx - w / 2f, cy - h / 2f, cx + w / 2f, cy + h / 2f)
}
}

/** A box with a class id and a confidence score, as produced by a detector before or after NMS. */
public data class ScoredBox(
val box: BoundingBox,
val classId: Int,
val score: Float
)

/**
* Greedy non-maximum suppression.
*
* Candidates are visited by descending score; a candidate is dropped when it overlaps an already
* kept box by more than [iouThreshold]. With [classAware] (the default) only boxes of the same
* class suppress each other. At most [maxDetections] boxes are kept. Ties keep input order.
*/
public fun nonMaxSuppression(
candidates: List<ScoredBox>,
iouThreshold: Float,
maxDetections: Int = Int.MAX_VALUE,
classAware: Boolean = true
): List<ScoredBox> {
require(maxDetections >= 0) { "maxDetections must be >= 0, got $maxDetections" }
val sorted = candidates.sortedByDescending { it.score }
val kept = ArrayList<ScoredBox>()
for (c in sorted) {
if (kept.size >= maxDetections) break
val suppressed = kept.any { k ->
(!classAware || k.classId == c.classId) && k.box.iou(c.box) > iouThreshold
}
if (!suppressed) kept += c
}
return kept
}
Loading
Loading