diff --git a/CHANGELOG.md b/CHANGELOG.md index 8372137d9..bf7e0fe79 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,26 @@ ## [Unreleased] +Headline: **model-specific detector code leaves core; the reusable detection pieces stay.** + +### Added + +- **`Letterbox`** (`skainet-data-media`, `sk.ainet.data.media`): aspect-preserving fit into a square canvas without a platform image. `Letterbox.fit(width, height, targetSize)` computes the scale and offsets, `fillNchw(rgb888, padValue = 114)` letterboxes packed RGB888 bytes into a planar `[3, size, size]` float array in `[0, 1]` (bilinear), `toSource(box)` maps a canvas box back to source pixels, clamped. +- **`BoundingBox`, `ScoredBox`, `nonMaxSuppression`** (`skainet-data-media`): pixel boxes with `iou`, `fromCenter`, and greedy, class-aware-by-default NMS with `maxDetections`. + +### Fixed + +- **StableHLO `softmax` over a non-last axis** (`skainet-compile-hlo`): the converter read the axis from `parameters["axis"]`, the tape records it as `dim`, so every exported `softmax(t, dim)` reduced over the last axis. It now reads `dim` (`axis` still accepted). +- **StableHLO `logSoftmax`**: recorded as `softmax` with `log = true`, it was exported as a plain softmax; it now lowers to `x - max - log(sum(exp(x - max)))`. + +### Removed + +- **`skainet-model-yolo`**: the YOLOv8 graph, config and decoder move to their cartridge blueprint (`detection-yolov8-iree` in SKaiNET-cartridge-blueprints), which owns the model end to end. The 0.58.0–0.60.0 artifacts stay on Maven Central. Its letterbox metadata and NMS are replaced by `Letterbox` and `nonMaxSuppression` above. +- **`yolo.class_names`, `yolo.names`, `yolo.num_classes`, `yolo.input_size`** are no longer read by `GgufModelMetadata`; the `general.*` and `model.*` keys are. + +### Deprecated + +- **`ModelArchitecture.YOLO`**: nothing in core maps to it any more; removed in the next minor release. ### Fixed - **Multi-row activations through the native Q8_0 / Q4_0 / Q5_0 / Q5_1 kernels returned zeros for every row but the last.** The FFM bindings of these four kernels staged a zero-filled output segment of `outputOffset + outputDim` floats and copied the whole reach back from index 0, so when `DefaultCpuOpsJvm` serves a `[rows, in] × packed [out, in]ᵀ` product row by row into one array, each call wiped the rows computed before it. A single-row product (a decode step) was right, which is why the GGUF decode paths never showed it; the token rows of an encoder on the quantize-at-load lane (Laya on Q8_0 with `skainet-backend-native-cpu` on the classpath) came back uniform. The bindings now copy back exactly the slice they produced, like the K-quant bindings always did. `NativeQ8CanonicalWeightTransposedReproTest` covers every packed format for 1, 7 and 50 rows against the dequant ground truth, with the native provider alone and with the default registry, and the SPI contract that a call at an output offset writes only its own row. diff --git a/README.md b/README.md index 1c338cad0..d56d0b0dd 100644 --- a/README.md +++ b/README.md @@ -243,7 +243,7 @@ Runnable examples: ### Neural Network DSL - **Sequential**: `nn { input(); dense(); relu(); dense() }` -- **DAG / Graph**: arbitrary wiring with `dag { }` for ResNet, YOLO-style architectures +- **DAG / Graph**: arbitrary wiring with `dag { }` for ResNet-style skip connections and multi-scale detector heads - Layers: Dense, Conv1d/2d/3d, MaxPool, AvgPool, BatchNorm, Dropout, LeakyReLU, ELU - KAN (Kolmogorov–Arnold Networks) layer (experimental) - Autograd engine with reverse-mode gradients, SGD and Adam/AdamW optimizers diff --git a/build.gradle.kts b/build.gradle.kts index 3f9db1536..d23146fcc 100644 --- a/build.gradle.kts +++ b/build.gradle.kts @@ -226,7 +226,6 @@ dependencies { // Other dokka(project(":skainet-pipeline")) - dokka(project(":skainet-models:skainet-model-yolo")) } // Copy the Dokka-generated HTML aggregate into the Antora site diff --git a/docs/modules/ROOT/pages/contributing/build-from-source.adoc b/docs/modules/ROOT/pages/contributing/build-from-source.adoc index e9b2780fc..c677ffc38 100644 --- a/docs/modules/ROOT/pages/contributing/build-from-source.adoc +++ b/docs/modules/ROOT/pages/contributing/build-from-source.adoc @@ -57,7 +57,7 @@ The plugin is applied to 21 library modules: |skainet-backends |`skainet-backend-cpu` |skainet-data |`skainet-data-api`, `skainet-data-transform`, `skainet-data-simple`, `skainet-data-media` |skainet-io |`skainet-io-core`, `skainet-io-gguf`, `skainet-io-image`, `skainet-io-onnx`, `skainet-io-safetensors` -|Other |`skainet-pipeline`, `skainet-model-yolo` +|Other |`skainet-pipeline` |=== *Excluded:* `skainet-bom` (no source), `skainet-apps/++*++`, `skainet-test/++*++`, benchmarks, and `skainet-lang-ksp-processor` (internal). diff --git a/docs/modules/ROOT/pages/tutorials/graph-dsl.adoc b/docs/modules/ROOT/pages/tutorials/graph-dsl.adoc index 3112dfc5d..c0c66e533 100644 --- a/docs/modules/ROOT/pages/tutorials/graph-dsl.adoc +++ b/docs/modules/ROOT/pages/tutorials/graph-dsl.adoc @@ -77,9 +77,9 @@ if (validation is ValidationResult.Valid) { } ---- -=== YOLO-style Example +=== Multi-scale Head Example -The Graph DSL is particularly useful for complex architectures like YOLO heads: +The Graph DSL is particularly useful for complex architectures such as multi-scale detector heads: [source,kotlin] ---- diff --git a/docs/modules/ROOT/pages/tutorials/image-data-getting-started.adoc b/docs/modules/ROOT/pages/tutorials/image-data-getting-started.adoc index c7f44fb6d..e8e13257d 100644 --- a/docs/modules/ROOT/pages/tutorials/image-data-getting-started.adoc +++ b/docs/modules/ROOT/pages/tutorials/image-data-getting-started.adoc @@ -22,7 +22,7 @@ This guide shows how the three image-oriented modules fit together: By the end you will: . Load an image from disk on the JVM. -. Letterbox it into a YOLO-style `(1, 3, H, W)` tensor. +. Letterbox it into a detector-style `(1, 3, H, W)` tensor. . Wrap that tensor in the `Image` metadata API. === Add the modules @@ -69,16 +69,16 @@ println("Loaded image: ${width}x${height}") `platformImageSize(...)` is the portable way to inspect dimensions. -=== Step 2: Letterbox an image for YOLO +=== Step 2: Letterbox an image for a detector -Object detectors such as YOLO commonly keep aspect ratio, resize the +Single-shot object detectors commonly keep aspect ratio, resize the image to fit inside a square canvas, and pad the remaining area with a constant color. This is usually called *letterboxing*. The image transform DSL makes that flow explicit. `toTensor(ctx)` converts the letterboxed platform image to an RGB tensor with shape `(1, 3, H, W)`, and `rescale(ctx, 255f)` moves pixel values into the -`[0, 1]` range expected by most YOLOv8-style exports. +`[0, 1]` range most detector exports expect. [source,kotlin] ---- @@ -107,7 +107,7 @@ val right = padX - left val top = padY / 2 val bottom = padY - top -val yoloInput = pipeline() +val detectorInput = pipeline() .resize(resizedWidth, resizedHeight) .pad( top = top, @@ -122,7 +122,7 @@ val yoloInput = pipeline() .rescale(ctx, 255f) .apply(input) -println("Tensor shape: ${yoloInput.shape}") +println("Tensor shape: ${detectorInput.shape}") println("Letterbox scale: $scale") println("Top/left padding: $top / $left") ---- @@ -147,7 +147,7 @@ import sk.ainet.data.media.Image import sk.ainet.data.media.ImageLayout val image = Image.fromTensor( - tensor = yoloInput, + tensor = detectorInput, layout = ImageLayout.NCHW, colorSpace = ColorSpace.RGB ) @@ -164,9 +164,11 @@ tracking which axis is width, height, or channels. [NOTE] ==== -If you use `skainet-model-yolo`, the same `scale`, `left`, and `top` -values from the letterbox step are the metadata needed to remap decoded -detections back to the original image coordinates. +The same geometry is available without a platform image as +`sk.ainet.data.media.Letterbox` (`skainet-data-media`): `Letterbox.fit(...)` +computes `scale` and the offsets, `fillNchw(rgb888)` letterboxes raw RGB888 +bytes, and `toSource(box)` remaps a detection back to the original image. +`nonMaxSuppression(...)` in the same module does class-aware NMS. ==== === Step 4: Start from a tensor you already have diff --git a/settings.gradle.kts b/settings.gradle.kts index d54bbc646..d4d9f523a 100644 --- a/settings.gradle.kts +++ b/settings.gradle.kts @@ -67,7 +67,6 @@ include("skainet-io:skainet-io-onnx") // ====== models include("skainet-models:skainet-model-identity") -include("skainet-models:skainet-model-yolo") // ====== FRONT ENDS (SKEEP-006) include("skainet-frontend:skainet-frontend-core") diff --git a/skainet-apps/skainet-tensor-tools/build.gradle.kts b/skainet-apps/skainet-tensor-tools/build.gradle.kts index efe3d1bd1..229f3754d 100644 --- a/skainet-apps/skainet-tensor-tools/build.gradle.kts +++ b/skainet-apps/skainet-tensor-tools/build.gradle.kts @@ -20,7 +20,6 @@ dependencies { implementation(project(":skainet-io:skainet-io-safetensors")) implementation(libs.kotlinx.io.core) implementation(project(":skainet-backends:skainet-backend-cpu")) - implementation(project(":skainet-models:skainet-model-yolo")) implementation(libs.pbandk.runtime) testImplementation(kotlin("test")) diff --git a/skainet-compile/skainet-compile-dag/src/commonTest/kotlin/sk/ainet/compile/graph/YoloStyleGraphDslTest.kt b/skainet-compile/skainet-compile-dag/src/commonTest/kotlin/sk/ainet/compile/graph/MultiScaleHeadGraphDslTest.kt similarity index 90% rename from skainet-compile/skainet-compile-dag/src/commonTest/kotlin/sk/ainet/compile/graph/YoloStyleGraphDslTest.kt rename to skainet-compile/skainet-compile-dag/src/commonTest/kotlin/sk/ainet/compile/graph/MultiScaleHeadGraphDslTest.kt index d47535ad3..edc25f793 100644 --- a/skainet-compile/skainet-compile-dag/src/commonTest/kotlin/sk/ainet/compile/graph/YoloStyleGraphDslTest.kt +++ b/skainet-compile/skainet-compile-dag/src/commonTest/kotlin/sk/ainet/compile/graph/MultiScaleHeadGraphDslTest.kt @@ -11,13 +11,13 @@ import kotlin.test.assertEquals import kotlin.test.assertTrue /** - * Smoke-test a YOLOv8-like head expressed with the DAG DSL: + * Smoke-test a multi-scale detector-style head expressed with the DAG DSL: * stem conv -> downsample conv -> upsample -> 1x1 head conv, with multiple parameter/constant nodes. */ -class YoloStyleGraphDslTest { +class MultiScaleHeadGraphDslTest { @Test - fun yoloish_head_graph_builds_and_validates() { + fun multi_scale_head_graph_builds_and_validates() { val program = dag { val input = input("input", TensorSpec("input", listOf(1, 3, 640, 640), "FP32")) @@ -35,7 +35,7 @@ class YoloStyleGraphDslTest { val bHead = constant("b_head") { shape(3) { zeros() } } val head = conv2d(up, wHead, bHead, stride = 1 to 1, padding = 0 to 0) - output(c2, head) // multi-scale outputs like YOLO heads + output(c2, head) // multi-scale outputs, one per feature level } val graph = program.toComputeGraph() as DefaultComputeGraph diff --git a/skainet-compile/skainet-compile-hlo/src/commonMain/kotlin/sk/ainet/compile/hlo/converters/ActivationOperationsConverter.kt b/skainet-compile/skainet-compile-hlo/src/commonMain/kotlin/sk/ainet/compile/hlo/converters/ActivationOperationsConverter.kt index f2018d623..ce87495d7 100644 --- a/skainet-compile/skainet-compile-hlo/src/commonMain/kotlin/sk/ainet/compile/hlo/converters/ActivationOperationsConverter.kt +++ b/skainet-compile/skainet-compile-hlo/src/commonMain/kotlin/sk/ainet/compile/hlo/converters/ActivationOperationsConverter.kt @@ -96,6 +96,7 @@ public class ActivationOperationsConverter : StableHloOperationConverter { /** * Convert softmax activation using real reductions and broadcast_in_dim. * softmax(x) = exp(x - max(x)) / sum(exp(x - max(x))) + * log_softmax(x) = (x - max(x)) - log(sum(exp(x - max(x)))) (parameter `log = true`) * * The max and sum terms are lowered to stablehlo.custom_call @reduce_max * and @reduce_sum — matching ReductionOperationsConverter's style — then @@ -124,8 +125,12 @@ public class ActivationOperationsConverter : StableHloOperationConverter { val inputShape = node.inputs.firstOrNull()?.shape ?: outputSpec?.shape ?: emptyList() val rank = inputShape.size - // Normalize axis against rank. Default to the last dimension. - val rawAxis = node.operation.parameters["axis"] as? Int ?: -1 + // Normalize axis against rank. Default to the last dimension. The tape records the axis as `dim` + // (TensorOps.softmax / logSoftmax); `axis` is accepted for hand-built graphs. + val params = node.operation.parameters + val rawAxis = (params["dim"] ?: params["axis"]) as? Int ?: -1 + // logSoftmax is recorded as softmax with `log = true`: x - max - log(sum(exp(x - max))). + val log = params["log"] == true val axis = when { rank == 0 -> 0 rawAxis < 0 -> rank + rawAxis @@ -215,7 +220,13 @@ public class ActivationOperationsConverter : StableHloOperationConverter { } else { add("$sumBroadcast = stablehlo.broadcast_in_dim $sumValue, dims = [$broadcastDims] : ($reducedType) -> $outputType") } - add("$resultValue = stablehlo.divide $expValue, $sumBroadcast : $outputType") + if (log) { + val logSum = context.nextTempValue() + add("$logSum = stablehlo.log $sumBroadcast : $outputType") + add("$resultValue = stablehlo.subtract $shiftedValue, $logSum : $outputType") + } else { + add("$resultValue = stablehlo.divide $expValue, $sumBroadcast : $outputType") + } } operations.forEach { context.emitOperation(it) } diff --git a/skainet-compile/skainet-compile-hlo/src/commonTest/kotlin/sk/ainet/compile/hlo/ActivationOperationsConverterTest.kt b/skainet-compile/skainet-compile-hlo/src/commonTest/kotlin/sk/ainet/compile/hlo/ActivationOperationsConverterTest.kt index 3d89e8af3..a4ec67430 100644 --- a/skainet-compile/skainet-compile-hlo/src/commonTest/kotlin/sk/ainet/compile/hlo/ActivationOperationsConverterTest.kt +++ b/skainet-compile/skainet-compile-hlo/src/commonTest/kotlin/sk/ainet/compile/hlo/ActivationOperationsConverterTest.kt @@ -216,6 +216,28 @@ class ActivationOperationsConverterTest { assertTrue(module.content.contains("stablehlo.divide")) } + @Test + fun testSoftmaxReducesOverTheRecordedDim() { + // The tape records TensorOps.softmax(t, dim) as parameters["dim"]; a non-last axis must be honoured. + val graph = createActivationGraphWithParameters("softmax", mapOf("dim" to 1), listOf(1, 16, 4, 8)) + val module = StableHloConverterFactory.createExtended().convert(graph, "test_softmax_dim") + + assertTrue(module.content.contains("applies stablehlo.maximum across dimensions = [1]"), module.content) + assertTrue(module.content.contains("applies stablehlo.add across dimensions = [1]"), module.content) + assertTrue(module.content.contains("-> tensor<1x4x8xf32>"), "reduced type drops the softmax axis") + assertTrue(module.content.contains("dims = [0, 2, 3]"), "broadcast back over the kept axes") + } + + @Test + fun testLogSoftmaxSubtractsTheLogSum() { + // TensorOps.logSoftmax is recorded as softmax with log = true. + val graph = createActivationGraphWithParameters("softmax", mapOf("dim" to -1, "log" to true), listOf(2, 3)) + val module = StableHloConverterFactory.createExtended().convert(graph, "test_log_softmax") + + assertTrue(module.content.contains("stablehlo.log "), module.content) + assertFalse(module.content.contains("stablehlo.divide"), "log-softmax must not normalize by division") + } + @Test fun testActivationChaining() { // Test chaining multiple activation functions diff --git a/skainet-data/skainet-data-media/build.gradle.kts b/skainet-data/skainet-data-media/build.gradle.kts index fef837f27..6e2fcd474 100644 --- a/skainet-data/skainet-data-media/build.gradle.kts +++ b/skainet-data/skainet-data-media/build.gradle.kts @@ -4,6 +4,7 @@ import org.jetbrains.kotlin.gradle.dsl.JvmTarget plugins { alias(libs.plugins.kotlinMultiplatform) alias(libs.plugins.androidMultiplatformLibrary) + alias(libs.plugins.vanniktech.mavenPublish) id("sk.ainet.dokka") } diff --git a/skainet-data/skainet-data-media/gradle.properties b/skainet-data/skainet-data-media/gradle.properties new file mode 100644 index 000000000..07be245b5 --- /dev/null +++ b/skainet-data/skainet-data-media/gradle.properties @@ -0,0 +1,2 @@ +POM_ARTIFACT_ID=skainet-data-media +POM_NAME=skainet data media diff --git a/skainet-data/skainet-data-media/src/commonMain/kotlin/sk/ainet/data/media/BoundingBox.kt b/skainet-data/skainet-data-media/src/commonMain/kotlin/sk/ainet/data/media/BoundingBox.kt new file mode 100644 index 000000000..75a79b1be --- /dev/null +++ b/skainet-data/skainet-data-media/src/commonMain/kotlin/sk/ainet/data/media/BoundingBox.kt @@ -0,0 +1,67 @@ +package sk.ainet.data.media + +import kotlin.math.max +import kotlin.math.min + +/** + * Axis-aligned box in pixel coordinates, origin top-left, `x1 <= x2` and `y1 <= y2` for a + * non-empty box. + */ +public data class BoundingBox( + val x1: Float, + val y1: Float, + val x2: Float, + val y2: Float +) { + public val width: Float get() = max(0f, x2 - x1) + public val height: Float get() = max(0f, y2 - y1) + public val area: Float get() = width * height + + /** Intersection over union with [other]; 0 when the union is empty. */ + public fun iou(other: BoundingBox): Float { + val iw = max(0f, min(x2, other.x2) - max(x1, other.x1)) + val ih = max(0f, min(y2, other.y2) - max(y1, other.y1)) + val inter = iw * ih + val union = area + other.area - inter + return if (union <= 0f) 0f else inter / union + } + + public companion object { + /** Box from a center point and size. */ + public fun fromCenter(cx: Float, cy: Float, w: Float, h: Float): BoundingBox = + BoundingBox(cx - w / 2f, cy - h / 2f, cx + w / 2f, cy + h / 2f) + } +} + +/** A box with a class id and a confidence score, as produced by a detector before or after NMS. */ +public data class ScoredBox( + val box: BoundingBox, + val classId: Int, + val score: Float +) + +/** + * Greedy non-maximum suppression. + * + * Candidates are visited by descending score; a candidate is dropped when it overlaps an already + * kept box by more than [iouThreshold]. With [classAware] (the default) only boxes of the same + * class suppress each other. At most [maxDetections] boxes are kept. Ties keep input order. + */ +public fun nonMaxSuppression( + candidates: List, + iouThreshold: Float, + maxDetections: Int = Int.MAX_VALUE, + classAware: Boolean = true +): List { + require(maxDetections >= 0) { "maxDetections must be >= 0, got $maxDetections" } + val sorted = candidates.sortedByDescending { it.score } + val kept = ArrayList() + for (c in sorted) { + if (kept.size >= maxDetections) break + val suppressed = kept.any { k -> + (!classAware || k.classId == c.classId) && k.box.iou(c.box) > iouThreshold + } + if (!suppressed) kept += c + } + return kept +} diff --git a/skainet-data/skainet-data-media/src/commonMain/kotlin/sk/ainet/data/media/Letterbox.kt b/skainet-data/skainet-data-media/src/commonMain/kotlin/sk/ainet/data/media/Letterbox.kt new file mode 100644 index 000000000..5a54220ac --- /dev/null +++ b/skainet-data/skainet-data-media/src/commonMain/kotlin/sk/ainet/data/media/Letterbox.kt @@ -0,0 +1,109 @@ +package sk.ainet.data.media + +import kotlin.math.min +import kotlin.math.roundToInt + +/** + * Aspect-preserving fit of a source image into a square [targetSize] canvas, padded with a constant + * color: the usual input preparation for single-shot detectors. + * + * The geometry is kept so predictions made on the canvas can be mapped back to the source image + * with [toSource]. + * + * Example: + * ```kotlin + * val lb = Letterbox.fit(width = 1280, height = 720, targetSize = 640) + * val nchw = lb.fillNchw(rgb888) // FloatArray, 1 x 3 x 640 x 640, values in [0, 1] + * val box = lb.toSource(BoundingBox(100f, 200f, 300f, 400f)) + * ``` + * + * @property sourceWidth width of the source image in pixels + * @property sourceHeight height of the source image in pixels + * @property targetSize side of the square canvas in pixels + * @property scale source → canvas scale factor + * @property resizedWidth width of the scaled image on the canvas + * @property resizedHeight height of the scaled image on the canvas + * @property padLeft horizontal offset of the scaled image on the canvas + * @property padTop vertical offset of the scaled image on the canvas + */ +public data class Letterbox( + val sourceWidth: Int, + val sourceHeight: Int, + val targetSize: Int, + val scale: Float, + val resizedWidth: Int, + val resizedHeight: Int, + val padLeft: Int, + val padTop: Int +) { + /** Maps a box in canvas pixels back to source pixels, clamped to the source image. */ + public fun toSource(box: BoundingBox): BoundingBox = BoundingBox( + x1 = ((box.x1 - padLeft) / scale).coerceIn(0f, sourceWidth.toFloat()), + y1 = ((box.y1 - padTop) / scale).coerceIn(0f, sourceHeight.toFloat()), + x2 = ((box.x2 - padLeft) / scale).coerceIn(0f, sourceWidth.toFloat()), + y2 = ((box.y2 - padTop) / scale).coerceIn(0f, sourceHeight.toFloat()) + ) + + /** + * Letterboxes packed RGB888 pixels (row-major, 3 bytes per pixel, no padding) into a planar + * NCHW float array of size `3 * targetSize * targetSize`, values divided by 255. Resizing is + * bilinear; the padding is [padValue] (0..255) on every channel. + */ + public fun fillNchw(rgb888: ByteArray, padValue: Int = 114): FloatArray { + require(rgb888.size == sourceWidth * sourceHeight * 3) { + "Expected ${sourceWidth * sourceHeight * 3} bytes for ${sourceWidth}x$sourceHeight RGB888, got ${rgb888.size}" + } + val plane = targetSize * targetSize + val out = FloatArray(3 * plane) + val pad = padValue / 255f + out.fill(pad) + val sx = sourceWidth.toFloat() / resizedWidth + val sy = sourceHeight.toFloat() / resizedHeight + for (y in 0 until resizedHeight) { + // pixel-center sampling, clamped to the source edge + val fy = ((y + 0.5f) * sy - 0.5f).coerceIn(0f, (sourceHeight - 1).toFloat()) + val y0 = fy.toInt() + val y1 = min(y0 + 1, sourceHeight - 1) + val wy = fy - y0 + val row = (y + padTop) * targetSize + for (x in 0 until resizedWidth) { + val fx = ((x + 0.5f) * sx - 0.5f).coerceIn(0f, (sourceWidth - 1).toFloat()) + val x0 = fx.toInt() + val x1 = min(x0 + 1, sourceWidth - 1) + val wx = fx - x0 + val o = row + x + padLeft + for (c in 0 until 3) { + val p00 = rgb888[(y0 * sourceWidth + x0) * 3 + c].toInt() and 0xFF + val p01 = rgb888[(y0 * sourceWidth + x1) * 3 + c].toInt() and 0xFF + val p10 = rgb888[(y1 * sourceWidth + x0) * 3 + c].toInt() and 0xFF + val p11 = rgb888[(y1 * sourceWidth + x1) * 3 + c].toInt() and 0xFF + val top = p00 + (p01 - p00) * wx + val bottom = p10 + (p11 - p10) * wx + out[c * plane + o] = (top + (bottom - top) * wy) / 255f + } + } + } + return out + } + + public companion object { + /** Computes the letterbox geometry for a [width] x [height] source on a [targetSize] canvas. */ + public fun fit(width: Int, height: Int, targetSize: Int): Letterbox { + require(width > 0 && height > 0) { "Source size must be positive, got ${width}x$height" } + require(targetSize > 0) { "Target size must be positive, got $targetSize" } + val scale = min(targetSize.toFloat() / width, targetSize.toFloat() / height) + val rw = (width * scale).roundToInt().coerceIn(1, targetSize) + val rh = (height * scale).roundToInt().coerceIn(1, targetSize) + return Letterbox( + sourceWidth = width, + sourceHeight = height, + targetSize = targetSize, + scale = scale, + resizedWidth = rw, + resizedHeight = rh, + padLeft = (targetSize - rw) / 2, + padTop = (targetSize - rh) / 2 + ) + } + } +} diff --git a/skainet-data/skainet-data-media/src/commonTest/kotlin/sk/ainet/data/media/BoundingBoxTest.kt b/skainet-data/skainet-data-media/src/commonTest/kotlin/sk/ainet/data/media/BoundingBoxTest.kt new file mode 100644 index 000000000..e6ee57b22 --- /dev/null +++ b/skainet-data/skainet-data-media/src/commonTest/kotlin/sk/ainet/data/media/BoundingBoxTest.kt @@ -0,0 +1,59 @@ +package sk.ainet.data.media + +import kotlin.test.Test +import kotlin.test.assertEquals +import kotlin.test.assertTrue + +class BoundingBoxTest { + + @Test + fun iouOfIdenticalBoxesIsOne() { + val a = BoundingBox(0f, 0f, 10f, 10f) + assertEquals(1f, a.iou(a)) + } + + @Test + fun iouOfDisjointBoxesIsZero() { + assertEquals(0f, BoundingBox(0f, 0f, 1f, 1f).iou(BoundingBox(2f, 2f, 3f, 3f))) + } + + @Test + fun iouOfHalfOverlap() { + // overlap 5x10 = 50, union 100 + 100 - 50 = 150 + val iou = BoundingBox(0f, 0f, 10f, 10f).iou(BoundingBox(5f, 0f, 15f, 10f)) + assertEquals(1f / 3f, iou, 1e-6f) + } + + @Test + fun fromCenter() { + assertEquals(BoundingBox(5f, 10f, 15f, 30f), BoundingBox.fromCenter(10f, 20f, 10f, 20f)) + } + + @Test + fun nmsKeepsTheBestOfAnOverlappingCluster() { + val kept = nonMaxSuppression( + listOf( + ScoredBox(BoundingBox(0f, 0f, 10f, 10f), classId = 0, score = 0.6f), + ScoredBox(BoundingBox(1f, 1f, 11f, 11f), classId = 0, score = 0.9f), + ScoredBox(BoundingBox(50f, 50f, 60f, 60f), classId = 0, score = 0.5f) + ), + iouThreshold = 0.45f + ) + assertEquals(listOf(0.9f, 0.5f), kept.map { it.score }) + } + + @Test + fun nmsIsClassAwareByDefault() { + val same = BoundingBox(0f, 0f, 10f, 10f) + val boxes = listOf(ScoredBox(same, 0, 0.9f), ScoredBox(same, 1, 0.8f)) + assertEquals(2, nonMaxSuppression(boxes, iouThreshold = 0.45f).size) + assertEquals(1, nonMaxSuppression(boxes, iouThreshold = 0.45f, classAware = false).size) + } + + @Test + fun nmsHonoursMaxDetectionsAndEmptyInput() { + val boxes = (0 until 5).map { ScoredBox(BoundingBox(it * 20f, 0f, it * 20f + 10f, 10f), 0, 0.1f * it) } + assertEquals(2, nonMaxSuppression(boxes, 0.45f, maxDetections = 2).size) + assertTrue(nonMaxSuppression(emptyList(), 0.45f).isEmpty()) + } +} diff --git a/skainet-data/skainet-data-media/src/commonTest/kotlin/sk/ainet/data/media/LetterboxTest.kt b/skainet-data/skainet-data-media/src/commonTest/kotlin/sk/ainet/data/media/LetterboxTest.kt new file mode 100644 index 000000000..bf78e29a2 --- /dev/null +++ b/skainet-data/skainet-data-media/src/commonTest/kotlin/sk/ainet/data/media/LetterboxTest.kt @@ -0,0 +1,68 @@ +package sk.ainet.data.media + +import kotlin.test.Test +import kotlin.test.assertEquals +import kotlin.test.assertFailsWith + +class LetterboxTest { + + @Test + fun landscapeIsPaddedTopAndBottom() { + val lb = Letterbox.fit(width = 1280, height = 720, targetSize = 640) + assertEquals(0.5f, lb.scale) + assertEquals(640, lb.resizedWidth) + assertEquals(360, lb.resizedHeight) + assertEquals(0, lb.padLeft) + assertEquals(140, lb.padTop) + } + + @Test + fun portraitIsPaddedLeftAndRight() { + val lb = Letterbox.fit(width = 300, height = 600, targetSize = 640) + assertEquals(320, lb.resizedWidth) + assertEquals(640, lb.resizedHeight) + assertEquals(160, lb.padLeft) + assertEquals(0, lb.padTop) + } + + @Test + fun toSourceInvertsTheFit() { + val lb = Letterbox.fit(width = 1280, height = 720, targetSize = 640) + val onCanvas = BoundingBox(x1 = 50f, y1 = 140f + 25f, x2 = 150f, y2 = 140f + 75f) + assertEquals(BoundingBox(100f, 50f, 300f, 150f), lb.toSource(onCanvas)) + } + + @Test + fun toSourceClampsToTheImage() { + val lb = Letterbox.fit(width = 1280, height = 720, targetSize = 640) + val b = lb.toSource(BoundingBox(-10f, 0f, 700f, 640f)) + assertEquals(BoundingBox(0f, 0f, 1280f, 720f), b) + } + + @Test + fun fillNchwPadsAndScalesValues() { + // 2x1 source: one red pixel, one blue pixel, on a 4x4 canvas → resized 4x2, padded 1 row top and bottom + val rgb = byteArrayOf(255.toByte(), 0, 0, 0, 0, 255.toByte()) + val lb = Letterbox.fit(width = 2, height = 1, targetSize = 4) + assertEquals(1, lb.padTop) + val out = lb.fillNchw(rgb, padValue = 0) + val plane = 16 + assertEquals(3 * plane, out.size) + // padding rows stay at the pad value + for (x in 0 until 4) { + assertEquals(0f, out[x]) // R, row 0 + assertEquals(0f, out[3 * 4 + x]) // R, row 3 + } + // left edge is pure red, right edge pure blue + assertEquals(1f, out[1 * 4 + 0]) // R at (0,1) + assertEquals(0f, out[2 * plane + 1 * 4 + 0]) // B at (0,1) + assertEquals(0f, out[1 * 4 + 3]) // R at (3,1) + assertEquals(1f, out[2 * plane + 1 * 4 + 3]) // B at (3,1) + } + + @Test + fun fillNchwRejectsAWrongBufferSize() { + val lb = Letterbox.fit(width = 2, height = 2, targetSize = 4) + assertFailsWith { lb.fillNchw(ByteArray(5)) } + } +} diff --git a/skainet-frontend/skainet-frontend-text/src/commonTest/kotlin/sk/ainet/frontend/tokenizer/TextFrontEndsDispatchTest.kt b/skainet-frontend/skainet-frontend-text/src/commonTest/kotlin/sk/ainet/frontend/tokenizer/TextFrontEndsDispatchTest.kt index 64b4ae161..fade77481 100644 --- a/skainet-frontend/skainet-frontend-text/src/commonTest/kotlin/sk/ainet/frontend/tokenizer/TextFrontEndsDispatchTest.kt +++ b/skainet-frontend/skainet-frontend-text/src/commonTest/kotlin/sk/ainet/frontend/tokenizer/TextFrontEndsDispatchTest.kt @@ -245,9 +245,9 @@ class TextFrontEndsDispatchTest { @Test fun `a view without text evidence resolves to null and resolveText names the registry`() { - val view = InMemoryCheckpointView(ModelFormat.ONNX, mapOf("architecture" to "yolo")) + val view = InMemoryCheckpointView(ModelFormat.ONNX, mapOf("architecture" to "resnet")) assertNull(TextFrontEnds.resolver().resolve(view).text) val ex = assertFailsWith { TextFrontEnds.resolver().resolveText(view) } - assertTrue("llama" in ex.message!! && "yolo" in ex.message!!, ex.message) + assertTrue("llama" in ex.message!! && "resnet" in ex.message!!, ex.message) } } diff --git a/skainet-io/skainet-io-core/src/commonMain/kotlin/sk/ainet/io/model/ClassNames.kt b/skainet-io/skainet-io-core/src/commonMain/kotlin/sk/ainet/io/model/ClassNames.kt index b2e9e7668..660162f20 100644 --- a/skainet-io/skainet-io-core/src/commonMain/kotlin/sk/ainet/io/model/ClassNames.kt +++ b/skainet-io/skainet-io-core/src/commonMain/kotlin/sk/ainet/io/model/ClassNames.kt @@ -23,7 +23,7 @@ public object ClassNames { /** * COCO 2017 dataset class names (80 classes). * - * Used by: YOLOv3, YOLOv4, YOLOv5, YOLOv8, Faster R-CNN, Mask R-CNN, etc. + * Used by most detectors and segmenters trained on COCO. */ public fun coco(): List = COCO_80 diff --git a/skainet-io/skainet-io-core/src/commonMain/kotlin/sk/ainet/io/model/ModelArchitecture.kt b/skainet-io/skainet-io-core/src/commonMain/kotlin/sk/ainet/io/model/ModelArchitecture.kt index 9126390d8..c0a2ee74b 100644 --- a/skainet-io/skainet-io-core/src/commonMain/kotlin/sk/ainet/io/model/ModelArchitecture.kt +++ b/skainet-io/skainet-io-core/src/commonMain/kotlin/sk/ainet/io/model/ModelArchitecture.kt @@ -8,6 +8,7 @@ public enum class ModelArchitecture { GEMMA, BERT, QWEN, + @Deprecated("Model-specific detectors live outside SKaiNET core (e.g. in their cartridge blueprint); nothing in core maps to this value. Removed in the next minor release.") YOLO, UNKNOWN; diff --git a/skainet-io/skainet-io-core/src/commonMain/kotlin/sk/ainet/io/weights/WeightMapper.kt b/skainet-io/skainet-io-core/src/commonMain/kotlin/sk/ainet/io/weights/WeightMapper.kt index 75eaaa8a1..75c42aced 100644 --- a/skainet-io/skainet-io-core/src/commonMain/kotlin/sk/ainet/io/weights/WeightMapper.kt +++ b/skainet-io/skainet-io-core/src/commonMain/kotlin/sk/ainet/io/weights/WeightMapper.kt @@ -191,7 +191,7 @@ public object WeightMapper { /** * Extract the most specific "model.X.Y.Z" pattern from a DSL module path. - * E.g., "Yolo8/model.22.cv2.0/model.22.cv2.0.0/Conv2d-0" -> "model.22.cv2.0.0" + * E.g., "Net/model.22.cv2.0/model.22.cv2.0.0/Conv2d-0" -> "model.22.cv2.0.0" * * @param path The full DSL module path * @return The extracted layer name in normalized format, or null if not found diff --git a/skainet-io/skainet-io-gguf/src/commonMain/kotlin/sk/ainet/io/gguf/GgufModelMetadata.kt b/skainet-io/skainet-io-gguf/src/commonMain/kotlin/sk/ainet/io/gguf/GgufModelMetadata.kt index 44c17f9ed..deefa8ca7 100644 --- a/skainet-io/skainet-io-gguf/src/commonMain/kotlin/sk/ainet/io/gguf/GgufModelMetadata.kt +++ b/skainet-io/skainet-io-gguf/src/commonMain/kotlin/sk/ainet/io/gguf/GgufModelMetadata.kt @@ -17,7 +17,7 @@ package sk.ainet.io.gguf * ``` */ public data class GgufModelMetadata( - /** Model architecture identifier (e.g., "llama", "yolov3-tiny", "bert") */ + /** Model architecture identifier (e.g., "llama", "gemma3", "bert") */ val architecture: String?, /** Model name/description */ @@ -119,19 +119,15 @@ public data class GgufModelMetadata( version = fields.getString("general.version"), url = fields.getString("general.url"), classNames = fields.getStringList( - "yolo.class_names", - "yolo.names", "general.class_names", "general.names", "model.class_names" ), numClasses = fields.getInt( - "yolo.num_classes", "general.num_classes", "model.num_classes" ), inputSize = fields.getInt( - "yolo.input_size", "general.input_size", "model.input_size" ), diff --git a/skainet-io/skainet-io-onnx/src/commonMain/kotlin/sk/ainet/io/onnx/OnnxModelMetadata.kt b/skainet-io/skainet-io-onnx/src/commonMain/kotlin/sk/ainet/io/onnx/OnnxModelMetadata.kt index 515211375..330597e48 100644 --- a/skainet-io/skainet-io-onnx/src/commonMain/kotlin/sk/ainet/io/onnx/OnnxModelMetadata.kt +++ b/skainet-io/skainet-io-onnx/src/commonMain/kotlin/sk/ainet/io/onnx/OnnxModelMetadata.kt @@ -8,7 +8,7 @@ import onnx.StringStringEntryProto * * This class extracts common metadata fields from ONNX ModelProto's * metadataProps in a structured, type-safe manner. Supports various - * model types including object detectors (YOLO), classifiers, etc. + * model types including object detectors, classifiers, etc. * * Usage: * ```kotlin @@ -122,7 +122,7 @@ public data class OnnxModelMetadata( /** * Parse class names from ONNX metadata. * - * YOLO/ultralytics format stores names as a Python dict string: + * Some exporters store names as a Python dict string: * `{0: 'person', 1: 'bicycle', 2: 'car', ...}` * * Also checks for plain comma-separated format: @@ -132,7 +132,7 @@ public data class OnnxModelMetadata( * @return Map of classId to className, or null if not found */ public fun parseClassNames(metadata: Map): Map? { - // Try "names" key (ultralytics/YOLO format) + // Try "names" key (Python-dict style) val namesValue = metadata["names"] ?: metadata["class_names"] ?: metadata["classes"] diff --git a/skainet-io/skainet-io-onnx/src/commonMain/kotlin/sk/ainet/io/onnx/OnnxWeightLoader.kt b/skainet-io/skainet-io-onnx/src/commonMain/kotlin/sk/ainet/io/onnx/OnnxWeightLoader.kt index dcfe95a14..901184d04 100644 --- a/skainet-io/skainet-io-onnx/src/commonMain/kotlin/sk/ainet/io/onnx/OnnxWeightLoader.kt +++ b/skainet-io/skainet-io-onnx/src/commonMain/kotlin/sk/ainet/io/onnx/OnnxWeightLoader.kt @@ -194,7 +194,7 @@ public object OnnxWeightLoader { /** * Extract the most specific "model.X.Y.Z" pattern from a DSL module path. - * E.g., "Yolo8/model.22.cv2.0/model.22.cv2.0.0/Conv2d-0" -> "model.22.cv2.0.0" + * E.g., "Net/model.22.cv2.0/model.22.cv2.0.0/Conv2d-0" -> "model.22.cv2.0.0" * * Delegates to [WeightMapper.extractLayerNameFromPath]. */ diff --git a/skainet-lang/skainet-lang-core/src/commonMain/kotlin/sk/ainet/lang/nn/Upsample2d.kt b/skainet-lang/skainet-lang-core/src/commonMain/kotlin/sk/ainet/lang/nn/Upsample2d.kt index b6c939ad0..1e174dd98 100644 --- a/skainet-lang/skainet-lang-core/src/commonMain/kotlin/sk/ainet/lang/nn/Upsample2d.kt +++ b/skainet-lang/skainet-lang-core/src/commonMain/kotlin/sk/ainet/lang/nn/Upsample2d.kt @@ -8,7 +8,7 @@ import sk.ainet.lang.types.DType /** * 2D upsampling layer for increasing spatial resolution. * - * Supports nearest-neighbor mode (typical for YOLO necks). Bilinear can be wired + * Supports nearest-neighbor mode (typical for feature-pyramid necks). Bilinear can be wired * once backends add support. */ public class Upsample2d( diff --git a/skainet-lang/skainet-lang-core/src/commonMain/kotlin/sk/ainet/lang/tensor/ops/UpsampleMode.kt b/skainet-lang/skainet-lang-core/src/commonMain/kotlin/sk/ainet/lang/tensor/ops/UpsampleMode.kt index 5819bbf9a..5d86a5532 100644 --- a/skainet-lang/skainet-lang-core/src/commonMain/kotlin/sk/ainet/lang/tensor/ops/UpsampleMode.kt +++ b/skainet-lang/skainet-lang-core/src/commonMain/kotlin/sk/ainet/lang/tensor/ops/UpsampleMode.kt @@ -3,7 +3,7 @@ package sk.ainet.lang.tensor.ops /** * Interpolation modes for 2D upsampling. * - * YOLO-style models typically use nearest-neighbor. Bilinear is provided for parity + * Detector necks typically use nearest-neighbor. Bilinear is provided for parity * but may ignore `alignCorners` when not applicable. */ public enum class UpsampleMode { diff --git a/skainet-lang/skainet-lang-core/src/commonTest/kotlin/sk/ainet/lang/nn/activations/ActivationDSLTest.kt b/skainet-lang/skainet-lang-core/src/commonTest/kotlin/sk/ainet/lang/nn/activations/ActivationDSLTest.kt index b33d8d1c4..6ce7b3829 100644 --- a/skainet-lang/skainet-lang-core/src/commonTest/kotlin/sk/ainet/lang/nn/activations/ActivationDSLTest.kt +++ b/skainet-lang/skainet-lang-core/src/commonTest/kotlin/sk/ainet/lang/nn/activations/ActivationDSLTest.kt @@ -63,7 +63,7 @@ class ActivationDSLTest { network { input(20) - // First layer with SiLU (YOLO8 style) + // First layer with SiLU dense(16) { activation = { tensor -> with(tensor) { silu() } } } @@ -85,9 +85,9 @@ class ActivationDSLTest { } @Test - fun testYOLO8StyleActivationChain() { - // Simulate YOLO8-style network using SiLU as primary activation - val yoloModel = definition { + fun testSiLUActivationChain() { + // A network using SiLU as its primary activation + val siluModel = definition { network { input(512) @@ -110,7 +110,7 @@ class ActivationDSLTest { } } - assertNotNull(yoloModel) - assertTrue(yoloModel.modules.size >= 4) // Input + 4 dense layers + assertNotNull(siluModel) + assertTrue(siluModel.modules.size >= 4) // Input + 4 dense layers } } \ No newline at end of file diff --git a/skainet-lang/skainet-lang-core/src/commonTest/kotlin/sk/ainet/lang/tensor/ops/VoidTensorOpsTest.kt b/skainet-lang/skainet-lang-core/src/commonTest/kotlin/sk/ainet/lang/tensor/ops/VoidTensorOpsTest.kt index 4199e633c..bba39bc67 100644 --- a/skainet-lang/skainet-lang-core/src/commonTest/kotlin/sk/ainet/lang/tensor/ops/VoidTensorOpsTest.kt +++ b/skainet-lang/skainet-lang-core/src/commonTest/kotlin/sk/ainet/lang/tensor/ops/VoidTensorOpsTest.kt @@ -574,7 +574,7 @@ class VoidTensorOpsTest { assertEquals(Shape(20, 1000), reshapeResult.shape) } - // Tests for new YOLO shape operations + // Tests for concat/split/upsample shape operations @Test fun testConcat_SameDimensions_Success() { diff --git a/skainet-models/skainet-model-yolo/api/jvm/skainet-model-yolo.api b/skainet-models/skainet-model-yolo/api/jvm/skainet-model-yolo.api deleted file mode 100644 index 58d9637e7..000000000 --- a/skainet-models/skainet-model-yolo/api/jvm/skainet-model-yolo.api +++ /dev/null @@ -1,139 +0,0 @@ -public final class sk/ainet/lang/model/dnn/yolo/Box { - public fun (FFFF)V - public final fun component1 ()F - public final fun component2 ()F - public final fun component3 ()F - public final fun component4 ()F - public final fun copy (FFFF)Lsk/ainet/lang/model/dnn/yolo/Box; - public static synthetic fun copy$default (Lsk/ainet/lang/model/dnn/yolo/Box;FFFFILjava/lang/Object;)Lsk/ainet/lang/model/dnn/yolo/Box; - public fun equals (Ljava/lang/Object;)Z - public final fun getHeight ()F - public final fun getWidth ()F - public final fun getX1 ()F - public final fun getX2 ()F - public final fun getY1 ()F - public final fun getY2 ()F - public fun hashCode ()I - public fun toString ()Ljava/lang/String; -} - -public final class sk/ainet/lang/model/dnn/yolo/Detection { - public fun (Lsk/ainet/lang/model/dnn/yolo/Box;FILjava/lang/String;)V - public synthetic fun (Lsk/ainet/lang/model/dnn/yolo/Box;FILjava/lang/String;ILkotlin/jvm/internal/DefaultConstructorMarker;)V - public final fun component1 ()Lsk/ainet/lang/model/dnn/yolo/Box; - public final fun component2 ()F - public final fun component3 ()I - public final fun component4 ()Ljava/lang/String; - public final fun copy (Lsk/ainet/lang/model/dnn/yolo/Box;FILjava/lang/String;)Lsk/ainet/lang/model/dnn/yolo/Detection; - public static synthetic fun copy$default (Lsk/ainet/lang/model/dnn/yolo/Detection;Lsk/ainet/lang/model/dnn/yolo/Box;FILjava/lang/String;ILjava/lang/Object;)Lsk/ainet/lang/model/dnn/yolo/Detection; - public fun equals (Ljava/lang/Object;)Z - public final fun getBox ()Lsk/ainet/lang/model/dnn/yolo/Box; - public final fun getClassId ()I - public final fun getLabel ()Ljava/lang/String; - public final fun getScore ()F - public fun hashCode ()I - public fun toString ()Ljava/lang/String; -} - -public final class sk/ainet/lang/model/dnn/yolo/HeadOutputs { - public fun (Lsk/ainet/lang/model/dnn/yolo/HeadTensor;Lsk/ainet/lang/model/dnn/yolo/HeadTensor;Lsk/ainet/lang/model/dnn/yolo/HeadTensor;)V - public final fun component1 ()Lsk/ainet/lang/model/dnn/yolo/HeadTensor; - public final fun component2 ()Lsk/ainet/lang/model/dnn/yolo/HeadTensor; - public final fun component3 ()Lsk/ainet/lang/model/dnn/yolo/HeadTensor; - public final fun copy (Lsk/ainet/lang/model/dnn/yolo/HeadTensor;Lsk/ainet/lang/model/dnn/yolo/HeadTensor;Lsk/ainet/lang/model/dnn/yolo/HeadTensor;)Lsk/ainet/lang/model/dnn/yolo/HeadOutputs; - public static synthetic fun copy$default (Lsk/ainet/lang/model/dnn/yolo/HeadOutputs;Lsk/ainet/lang/model/dnn/yolo/HeadTensor;Lsk/ainet/lang/model/dnn/yolo/HeadTensor;Lsk/ainet/lang/model/dnn/yolo/HeadTensor;ILjava/lang/Object;)Lsk/ainet/lang/model/dnn/yolo/HeadOutputs; - public fun equals (Ljava/lang/Object;)Z - public final fun getLarge ()Lsk/ainet/lang/model/dnn/yolo/HeadTensor; - public final fun getMedium ()Lsk/ainet/lang/model/dnn/yolo/HeadTensor; - public final fun getSmall ()Lsk/ainet/lang/model/dnn/yolo/HeadTensor; - public fun hashCode ()I - public fun toString ()Ljava/lang/String; -} - -public final class sk/ainet/lang/model/dnn/yolo/HeadTensor { - public fun (Lsk/ainet/lang/tensor/Tensor;Lsk/ainet/lang/tensor/Tensor;)V - public final fun component1 ()Lsk/ainet/lang/tensor/Tensor; - public final fun component2 ()Lsk/ainet/lang/tensor/Tensor; - public final fun copy (Lsk/ainet/lang/tensor/Tensor;Lsk/ainet/lang/tensor/Tensor;)Lsk/ainet/lang/model/dnn/yolo/HeadTensor; - public static synthetic fun copy$default (Lsk/ainet/lang/model/dnn/yolo/HeadTensor;Lsk/ainet/lang/tensor/Tensor;Lsk/ainet/lang/tensor/Tensor;ILjava/lang/Object;)Lsk/ainet/lang/model/dnn/yolo/HeadTensor; - public fun equals (Ljava/lang/Object;)Z - public final fun getCls ()Lsk/ainet/lang/tensor/Tensor; - public final fun getReg ()Lsk/ainet/lang/tensor/Tensor; - public fun hashCode ()I - public fun toString ()Ljava/lang/String; -} - -public final class sk/ainet/lang/model/dnn/yolo/Yolo8 : sk/ainet/lang/model/Model { - public fun ()V - public fun (Lsk/ainet/lang/model/dnn/yolo/YoloConfig;)V - public synthetic fun (Lsk/ainet/lang/model/dnn/yolo/YoloConfig;ILkotlin/jvm/internal/DefaultConstructorMarker;)V - public synthetic fun calculate (Lsk/ainet/lang/nn/Module;Ljava/lang/Object;Lsk/ainet/context/ExecutionContext;Lkotlin/jvm/functions/Function4;Lkotlin/coroutines/Continuation;)Ljava/lang/Object; - public fun calculate (Lsk/ainet/lang/nn/Module;Lsk/ainet/lang/tensor/Tensor;Lsk/ainet/context/ExecutionContext;Lkotlin/jvm/functions/Function4;Lkotlin/coroutines/Continuation;)Ljava/lang/Object; - public final fun calculateHeads (Lsk/ainet/lang/nn/Module;Lsk/ainet/lang/tensor/Tensor;Lsk/ainet/context/ExecutionContext;)Lsk/ainet/lang/model/dnn/yolo/HeadOutputs; - public fun create (Lsk/ainet/context/ExecutionContext;)Lsk/ainet/lang/nn/Module; - public synthetic fun fit (Lsk/ainet/lang/nn/Module;Ljava/lang/Object;Lsk/ainet/context/ExecutionContext;)Lkotlinx/coroutines/flow/Flow; - public fun fit (Lsk/ainet/lang/nn/Module;Lsk/ainet/lang/tensor/Tensor;Lsk/ainet/context/ExecutionContext;)Lkotlinx/coroutines/flow/Flow; - public fun getCoroutineContext ()Lkotlinx/coroutines/CoroutineDispatcher; - public final fun infer (Lsk/ainet/lang/nn/Module;Lsk/ainet/lang/model/dnn/yolo/YoloInput;Lsk/ainet/context/ExecutionContext;Lkotlin/jvm/functions/Function4;Lkotlin/coroutines/Continuation;)Ljava/lang/Object; - public static synthetic fun infer$default (Lsk/ainet/lang/model/dnn/yolo/Yolo8;Lsk/ainet/lang/nn/Module;Lsk/ainet/lang/model/dnn/yolo/YoloInput;Lsk/ainet/context/ExecutionContext;Lkotlin/jvm/functions/Function4;Lkotlin/coroutines/Continuation;ILjava/lang/Object;)Ljava/lang/Object; - public final fun model (Lsk/ainet/context/ExecutionContext;)Lsk/ainet/lang/nn/Module; - public fun modelCard ()Lsk/ainet/lang/model/ModelCard; -} - -public final class sk/ainet/lang/model/dnn/yolo/YoloConfig { - public fun ()V - public fun (IIFFILjava/util/List;IIF)V - public synthetic fun (IIFFILjava/util/List;IIFILkotlin/jvm/internal/DefaultConstructorMarker;)V - public final fun component1 ()I - public final fun component2 ()I - public final fun component3 ()F - public final fun component4 ()F - public final fun component5 ()I - public final fun component6 ()Ljava/util/List; - public final fun component7 ()I - public final fun component8 ()I - public final fun component9 ()F - public final fun copy (IIFFILjava/util/List;IIF)Lsk/ainet/lang/model/dnn/yolo/YoloConfig; - public static synthetic fun copy$default (Lsk/ainet/lang/model/dnn/yolo/YoloConfig;IIFFILjava/util/List;IIFILjava/lang/Object;)Lsk/ainet/lang/model/dnn/yolo/YoloConfig; - public fun equals (Ljava/lang/Object;)Z - public final fun getBaseChannels ()I - public final fun getClassNames ()Ljava/util/List; - public final fun getConfThreshold ()F - public final fun getDepthMultiple ()F - public final fun getInputSize ()I - public final fun getIouThreshold ()F - public final fun getMaxDetections ()I - public final fun getNumClasses ()I - public final fun getRegMax ()I - public fun hashCode ()I - public fun toString ()Ljava/lang/String; -} - -public final class sk/ainet/lang/model/dnn/yolo/YoloInput { - public fun (Lsk/ainet/lang/tensor/Tensor;IIFII)V - public synthetic fun (Lsk/ainet/lang/tensor/Tensor;IIFIIILkotlin/jvm/internal/DefaultConstructorMarker;)V - public final fun component1 ()Lsk/ainet/lang/tensor/Tensor; - public final fun component2 ()I - public final fun component3 ()I - public final fun component4 ()F - public final fun component5 ()I - public final fun component6 ()I - public final fun copy (Lsk/ainet/lang/tensor/Tensor;IIFII)Lsk/ainet/lang/model/dnn/yolo/YoloInput; - public static synthetic fun copy$default (Lsk/ainet/lang/model/dnn/yolo/YoloInput;Lsk/ainet/lang/tensor/Tensor;IIFIIILjava/lang/Object;)Lsk/ainet/lang/model/dnn/yolo/YoloInput; - public fun equals (Ljava/lang/Object;)Z - public final fun getLetterboxScale ()F - public final fun getOriginalHeight ()I - public final fun getOriginalWidth ()I - public final fun getPadH ()I - public final fun getPadW ()I - public final fun getTensor ()Lsk/ainet/lang/tensor/Tensor; - public fun hashCode ()I - public fun toString ()Ljava/lang/String; -} - -public final class sk/ainet/lang/model/dnn/yolo/YoloPreprocess { - public static final field INSTANCE Lsk/ainet/lang/model/dnn/yolo/YoloPreprocess; - public final fun fromReadyTensor (Lsk/ainet/lang/tensor/Tensor;IIIIIF)Lsk/ainet/lang/model/dnn/yolo/YoloInput; - public static synthetic fun fromReadyTensor$default (Lsk/ainet/lang/model/dnn/yolo/YoloPreprocess;Lsk/ainet/lang/tensor/Tensor;IIIIIFILjava/lang/Object;)Lsk/ainet/lang/model/dnn/yolo/YoloInput; -} - diff --git a/skainet-models/skainet-model-yolo/build.gradle.kts b/skainet-models/skainet-model-yolo/build.gradle.kts deleted file mode 100644 index 35211d9fc..000000000 --- a/skainet-models/skainet-model-yolo/build.gradle.kts +++ /dev/null @@ -1,59 +0,0 @@ -import org.jetbrains.kotlin.gradle.ExperimentalWasmDsl -import org.jetbrains.kotlin.gradle.dsl.JvmTarget - -plugins { - alias(libs.plugins.kotlinMultiplatform) - alias(libs.plugins.androidMultiplatformLibrary) - alias(libs.plugins.vanniktech.mavenPublish) - alias(libs.plugins.binary.compatibility.validator) - id("sk.ainet.dokka") -} - -kotlin { - explicitApi() - - android { - namespace = "sk.ainet.lang.model.yolo" - compileSdk = libs.versions.android.compileSdk.get().toInt() - minSdk = libs.versions.android.minSdk.get().toInt() - compilerOptions { - jvmTarget.set(JvmTarget.JVM_11) - } - } - - iosArm64() - iosSimulatorArm64() - macosArm64 () - linuxX64 () - linuxArm64 () - - jvm() - - js { - browser() - } - - @OptIn(ExperimentalWasmDsl::class) - wasmJs { - browser() - } - - @OptIn(ExperimentalWasmDsl::class) - wasmWasi { - nodejs() - } - - sourceSets { - commonMain.dependencies { - implementation(project(":skainet-lang:skainet-lang-core")) - implementation(project(":skainet-lang:skainet-lang-models")) - implementation(libs.kotlinx.coroutines) - } - - commonTest.dependencies { - implementation(libs.kotlin.test) - implementation(project(":skainet-backends:skainet-backend-cpu")) - - } - } -} diff --git a/skainet-models/skainet-model-yolo/gradle.properties b/skainet-models/skainet-model-yolo/gradle.properties deleted file mode 100644 index cd80139a5..000000000 --- a/skainet-models/skainet-model-yolo/gradle.properties +++ /dev/null @@ -1,2 +0,0 @@ -POM_ARTIFACT_ID=skainet-model-yolo -POM_NAME=skainet neural network scripting API \ No newline at end of file diff --git a/skainet-models/skainet-model-yolo/karma.config.d/skainet-browser-resilience.js b/skainet-models/skainet-model-yolo/karma.config.d/skainet-browser-resilience.js deleted file mode 100644 index 53d775403..000000000 --- a/skainet-models/skainet-model-yolo/karma.config.d/skainet-browser-resilience.js +++ /dev/null @@ -1,20 +0,0 @@ -// SKaiNET: harden Karma browser tests against launch/capture flakiness. -// -// During `./gradlew allTests` many wasmJs/js browser test tasks start a -// ChromeHeadless instance at the same time. On a loaded machine the browser -// can take longer than Karma's defaults to capture or to emit progress, so -// Karma disconnects it ("Disconnected, no message in 30000 ms" / -// "ChromeHeadless was not killed"), ends up discovering zero tests and the -// build fails on Gradle's `failOnNoDiscoveredTests` check — even though the -// tests pass fine when the task runs in isolation. -// -// Generous capture/disconnect/no-activity timeouts plus a few disconnect -// retries make the run wait patiently for a starved browser instead of -// giving up. Kotlin's Gradle plugin merges every *.js here into karma.conf.js. -config.set({ - captureTimeout: 120000, - browserDisconnectTimeout: 30000, - browserDisconnectTolerance: 3, - browserNoActivityTimeout: 120000, - pingTimeout: 120000, -}); diff --git a/skainet-models/skainet-model-yolo/src/commonMain/kotlin/sk/ainet/lang/model/dnn/yolo/Yolo8.kt b/skainet-models/skainet-model-yolo/src/commonMain/kotlin/sk/ainet/lang/model/dnn/yolo/Yolo8.kt deleted file mode 100644 index 83bc4d086..000000000 --- a/skainet-models/skainet-model-yolo/src/commonMain/kotlin/sk/ainet/lang/model/dnn/yolo/Yolo8.kt +++ /dev/null @@ -1,98 +0,0 @@ -package sk.ainet.lang.model.dnn.yolo - -import sk.ainet.context.ExecutionContext -import sk.ainet.lang.model.Model -import sk.ainet.lang.model.ModelCard -import sk.ainet.lang.nn.Module -import sk.ainet.lang.tensor.Tensor -import sk.ainet.lang.types.FP32 - -public class Yolo8( - private val config: YoloConfig = YoloConfig() -) : Model, Tensor> { - - private fun buildModel(executionContext: ExecutionContext): Module = - Yolo8Module(executionContext, config) - - // Backward-compatible helper for old call sites - public fun model(executionContext: ExecutionContext): Module = create(executionContext) - - override fun create(executionContext: ExecutionContext): Module = buildModel(executionContext) - - override suspend fun calculate( - module: Module, - inputValue: Tensor, - executionContext: ExecutionContext, - reportProgress: suspend (current: Int, total: Int, message: String?) -> Unit - ): Tensor { - reportProgress(0, 1, "starting yolo8 forward") - val out = module.forward(inputValue, executionContext) - reportProgress(1, 1, "done") - return out - } - - /** - * Runs the YOLO graph and returns all head outputs (small/medium/large) for downstream post-processing. - */ - public fun calculateHeads( - module: Module, - inputValue: Tensor, - executionContext: ExecutionContext - ): HeadOutputs { - require(module is Yolo8Module) { "calculateHeads expects a Yolo8Module instance" } - return module.forwardHeads(inputValue, executionContext) - } - - /** - * Full inference: forward pass + decode + NMS. Expects the input tensor to be - * preprocessed to the configured model size (see [YoloPreprocess]). - */ - public suspend fun infer( - module: Module, - input: YoloInput, - executionContext: ExecutionContext, - reportProgress: suspend (current: Int, total: Int, message: String?) -> Unit = { _, _, _ -> } - ): List { - require(module is Yolo8Module) { "infer expects a Yolo8Module instance" } - reportProgress(0, 2, "yolo forward") - val heads = module.forwardHeads(input.tensor, executionContext) - reportProgress(1, 2, "decode") - val detections = YoloDecoder.decode(heads, config, input) - reportProgress(2, 2, "done") - return detections - } - - override fun modelCard(): ModelCard { - return ModelCard( - license = "apache-2.0", - libraryName = "skainet", - pipelineTag = "object-detection", - language = listOf("en"), - modalities = listOf("vision"), - baseModel = "yolov8", - contextLength = 0, - datasets = emptyList(), - metrics = emptyList(), - modelIndex = emptyList(), - intendedUse = "YOLOv8-style object detection skeleton using SKaiNET DSL blocks.", - limitations = "Weights and image preprocessing are not included; outputs depend on provided weights and input normalization." - ) - } -} - -private class Yolo8Module( - private val executionContext: ExecutionContext, - private val config: YoloConfig, override val name: String = "Yolo8" -) : Module() { - - private val graph = Yolo8Graph(executionContext, config) - - override val modules: List> - get() = graph.modules() - - override fun forward(input: Tensor, ctx: ExecutionContext): Tensor = - graph.forward(input, ctx).large.cls - - fun forwardHeads(input: Tensor, ctx: ExecutionContext): HeadOutputs = - graph.forward(input, ctx) -} diff --git a/skainet-models/skainet-model-yolo/src/commonMain/kotlin/sk/ainet/lang/model/dnn/yolo/Yolo8Graph.kt b/skainet-models/skainet-model-yolo/src/commonMain/kotlin/sk/ainet/lang/model/dnn/yolo/Yolo8Graph.kt deleted file mode 100644 index d285aaad3..000000000 --- a/skainet-models/skainet-model-yolo/src/commonMain/kotlin/sk/ainet/lang/model/dnn/yolo/Yolo8Graph.kt +++ /dev/null @@ -1,612 +0,0 @@ -package sk.ainet.lang.model.dnn.yolo - -import sk.ainet.context.ExecutionContext -import sk.ainet.lang.nn.Module -import sk.ainet.lang.nn.definition -import sk.ainet.lang.nn.network -import sk.ainet.lang.tensor.Tensor -import sk.ainet.lang.tensor.ops.UpsampleMode -import sk.ainet.lang.tensor.silu -import sk.ainet.lang.types.FP32 -import kotlin.math.round - -/** - * Minimal YOLOv8-style graph assembled from reusable blocks. - * - * This is intentionally compact: it wires the main backbone, a lightweight neck - * with upsampling/concats, and three detection heads. The output currently returns - * the largest-scale head; smaller heads are kept for future concatenation when - * post-processing is added. - */ -internal class Yolo8Graph(private val executionContext: ExecutionContext, private val config: YoloConfig) { - - private val base = config.baseChannels - private val c1 = base - private val c2 = base * 2 - private val c3 = base * 4 - private val c4 = base * 8 - private val c5 = base * 16 - - private fun scaleDepth(baseCount: Int): Int = - kotlin.math.max(1, round(baseCount * config.depthMultiple).toInt()) - - private val stem = ConvSiLU( - inChannels = 3, - outChannels = c1, - kernel = 3, - stride = 2, - padding = 1, - name = "stem", - executionContext = executionContext - ) - - private val stage1 = ConvSiLU( - inChannels = c1, - outChannels = c2, - kernel = 3, - stride = 2, - padding = 1, - name = "stage1_down", - executionContext = executionContext - ) - private val c2f1 = C2f( - inChannels = c2, - outChannels = c2, - bottlenecks = scaleDepth(3), - name = "c2f1", - executionContext = executionContext - ) - - private val stage2 = ConvSiLU( - inChannels = c2, - outChannels = c3, - kernel = 3, - stride = 2, - padding = 1, - name = "stage2_down", - executionContext = executionContext - ) - private val c2f2 = C2f( - inChannels = c3, - outChannels = c3, - bottlenecks = scaleDepth(6), - name = "c2f2", - executionContext = executionContext - ) - - private val stage3 = ConvSiLU( - inChannels = c3, - outChannels = c4, - kernel = 3, - stride = 2, - padding = 1, - name = "stage3_down", - executionContext = executionContext - ) - private val c2f3 = C2f( - inChannels = c4, - outChannels = c4, - bottlenecks = scaleDepth(6), - name = "c2f3", - executionContext = executionContext - ) - - private val stage4 = ConvSiLU( - inChannels = c4, - outChannels = c5, - kernel = 3, - stride = 2, - padding = 1, - name = "stage4_down", - executionContext = executionContext - ) - private val c2f4 = C2f( - inChannels = c5, - outChannels = c5, - bottlenecks = scaleDepth(3), - name = "c2f4", - executionContext = executionContext - ) - - private val sppf = Sppf(channels = c5, name = "sppf", executionContext = executionContext) - - // Neck - private val up1 = UpsampleWrapper(scale = 2 to 2, name = "up1", executionContext = executionContext) - private val c2fNeck1 = C2f( - inChannels = c5 + c4, - outChannels = c4, - bottlenecks = scaleDepth(3), - name = "model.12", - executionContext = executionContext - ) - - private val up2 = UpsampleWrapper(scale = 2 to 2, name = "up2", executionContext = executionContext) - private val c2fNeck2 = C2f( - inChannels = c4 + c3, - outChannels = c3, - bottlenecks = scaleDepth(3), - name = "model.15", - executionContext = executionContext - ) - - // Bottom-up path - private val down1 = ConvSiLU( - inChannels = c3, - outChannels = c3, - kernel = 3, - stride = 2, - padding = 1, - name = "model.16", - executionContext = executionContext - ) - private val c2fDown1 = C2f( - inChannels = c3 + c4, - outChannels = c4, - bottlenecks = scaleDepth(3), - name = "model.18", - executionContext = executionContext - ) - private val down2 = ConvSiLU( - inChannels = c4, - outChannels = c4, - kernel = 3, - stride = 2, - padding = 1, - name = "model.19", - executionContext = executionContext - ) - private val c2fDown2 = C2f( - inChannels = c4 + c5, - outChannels = c5, - bottlenecks = scaleDepth(3), - name = "model.21", - executionContext = executionContext - ) - - private val detect = DecoupledDetectHead( - ch = listOf(c3, c4, c5), - numClasses = config.numClasses, - regMax = config.regMax, - name = "model.22", - executionContext = executionContext - ) - - fun modules(): List> = buildList { - add(stem.module) - add(stage1.module); add(c2f1) - add(stage2.module); add(c2f2) - add(stage3.module); add(c2f3) - add(stage4.module); add(c2f4) - add(sppf) - add(up1) - add(c2fNeck1) - add(up2) - add(c2fNeck2) - add(down1.module) - add(c2fDown1) - add(down2.module) - add(c2fDown2) - add(detect) - } - - fun forward(input: Tensor, ctx: ExecutionContext): HeadOutputs { - val x0 = stem.forward(input, ctx) - val x1 = c2f1.forward(stage1.forward(x0, ctx), ctx) // 1/4 - val x2 = c2f2.forward(stage2.forward(x1, ctx), ctx) // 1/8 - val x3 = c2f3.forward(stage3.forward(x2, ctx), ctx) // 1/16 - val x4 = c2f4.forward(stage4.forward(x3, ctx), ctx) // 1/32 - val p5 = sppf.forward(x4, ctx) - - // Neck path - val p4 = c2fNeck1.forward( - concatAlongChannels( - up1.forward(p5, ctx), - x3, - ctx - ), - ctx - ) - val p3 = c2fNeck2.forward( - concatAlongChannels( - up2.forward(p4, ctx), - x2, - ctx - ), - ctx - ) - - // Bottom-up refinements - val p4d = c2fDown1.forward( - concatAlongChannels( - down1.forward(p3, ctx), - p4, - ctx - ), - ctx - ) - val p5d = c2fDown2.forward( - concatAlongChannels( - down2.forward(p4d, ctx), - p5, - ctx - ), - ctx - ) - - return detect.forward(listOf(p3, p4d, p5d), ctx) - } - - private fun concatAlongChannels( - a: Tensor, - b: Tensor, - ctx: ExecutionContext - ): Tensor = ctx.ops.concat(listOf(a, b), dim = 1) -} - -internal class ConvBnSiLU( - private val inChannels: Int, - private val outChannels: Int, - private val kernel: Int, - private val stride: Int, - private val padding: Int, - private val name: String, - executionContext: ExecutionContext -) { - val module: Module = definition { - network(executionContext) { - sequential { - stage(name) { - conv2d( - outChannels = outChannels, - kernelSize = kernel to kernel, - stride = stride to stride, - padding = padding to padding - ) { - this.inChannels = this@ConvBnSiLU.inChannels - } - batchNorm(numFeatures = outChannels) - activation { tensor -> tensor.silu() } - } - } - } - } - - fun forward(input: Tensor, ctx: ExecutionContext): Tensor = - module.forward(input, ctx) -} - -internal class ConvSiLU( - private val inChannels: Int, - private val outChannels: Int, - private val kernel: Int, - private val stride: Int, - private val padding: Int, - private val name: String, - executionContext: ExecutionContext -) { - val module: Module = definition { - network(executionContext) { - sequential { - stage(name) { - conv2d( - outChannels = outChannels, - kernelSize = kernel to kernel, - stride = stride to stride, - padding = padding to padding - ) { - this.inChannels = this@ConvSiLU.inChannels - this.bias = true - } - activation { tensor -> tensor.silu() } - } - } - } - } - - fun forward(input: Tensor, ctx: ExecutionContext): Tensor = - module.forward(input, ctx) -} - -internal class C2f( - private val inChannels: Int, - private val outChannels: Int, - bottlenecks: Int, - override val name: String, - private val executionContext: ExecutionContext -) : Module() { - - private val hidden = outChannels / 2 - private val cv1 = ConvSiLU( - inChannels = inChannels, - outChannels = hidden * 2, - kernel = 1, - stride = 1, - padding = 0, - name = "${name}_cv1", - executionContext = executionContext - ) - private val bottleneckBlocks = List(bottlenecks) { idx -> - Bottleneck( - channels = hidden, - name = "${name}_m$idx", - executionContext = executionContext - ) - } - private val cv2 = ConvSiLU( - inChannels = hidden * (2 + bottlenecks), - outChannels = outChannels, - kernel = 1, - stride = 1, - padding = 0, - name = "${name}_cv2", - executionContext = executionContext - ) - - override val modules: List> = buildList { - add(cv1.module) - bottleneckBlocks.forEach { add(it) } - add(cv2.module) - } - - override fun forward(input: Tensor, ctx: ExecutionContext): Tensor { - val y = ctx.ops.split(cv1.forward(input, ctx), splitSize = hidden, dim = 1).toMutableList() - var last = y.last() - bottleneckBlocks.forEach { block -> - last = block.forward(last, ctx) - y += last - } - val concat = ctx.ops.concat(y, dim = 1) - return cv2.forward(concat, ctx) - } -} - -private class Bottleneck( - channels: Int, - override val name: String, - private val executionContext: ExecutionContext -) : Module() { - - private val cv1 = ConvSiLU( - inChannels = channels, - outChannels = channels, - kernel = 3, - stride = 1, - padding = 1, - name = "${name}_cv1", - executionContext = executionContext - ) - private val cv2 = ConvSiLU( - inChannels = channels, - outChannels = channels, - kernel = 3, - stride = 1, - padding = 1, - name = "${name}_cv2", - executionContext = executionContext - ) - - override val modules: List> = listOf(cv1.module, cv2.module) - - override fun forward(input: Tensor, ctx: ExecutionContext): Tensor { - val h1 = cv1.forward(input, ctx) - val h2 = cv2.forward(h1, ctx) - return ctx.ops.add(input, h2) - } -} - -public data class HeadTensor( - val reg: Tensor, - val cls: Tensor -) - -public data class HeadOutputs( - val small: HeadTensor, - val medium: HeadTensor, - val large: HeadTensor -) - -internal class Sppf( - channels: Int, - override val name: String, - private val executionContext: ExecutionContext -) : Module() { - - private val reduce = ConvSiLU( - inChannels = channels, - outChannels = channels / 2, - kernel = 1, - stride = 1, - padding = 0, - name = "${name}_reduce", - executionContext = executionContext - ) - private val expand = ConvSiLU( - inChannels = channels * 2, - outChannels = channels, - kernel = 1, - stride = 1, - padding = 0, - name = "${name}_expand", - executionContext = executionContext - ) - - override val modules: List> = listOf(reduce.module, expand.module) - - override fun forward(input: Tensor, ctx: ExecutionContext): Tensor { - val x = reduce.forward(input, ctx) - val p1 = ctx.ops.maxPool2d(x, kernelSize = 5 to 5, stride = 1 to 1, padding = 2 to 2) - val p2 = ctx.ops.maxPool2d(p1, kernelSize = 5 to 5, stride = 1 to 1, padding = 2 to 2) - val p3 = ctx.ops.maxPool2d(p2, kernelSize = 5 to 5, stride = 1 to 1, padding = 2 to 2) - val concat = ctx.ops.concat(listOf(x, p1, p2, p3), dim = 1) - return expand.forward(concat, ctx) - } -} - -internal class UpsampleWrapper( - private val scale: Pair, - override val name: String, - private val executionContext: ExecutionContext -) : Module() { - private val module: Module = definition { - network(executionContext) { - sequential { - stage(name) { - upsample2d(scale = scale, mode = UpsampleMode.Nearest) - } - } - } - } - - override val modules: List> = listOf(module) - - override fun forward(input: Tensor, ctx: ExecutionContext): Tensor = - module.forward(input, ctx) -} - -internal class DecoupledDetectHead( - ch: List, - private val numClasses: Int, - private val regMax: Int, - override val name: String, - private val executionContext: ExecutionContext -) : Module() { - - private val regBranches = ch.mapIndexed { idx, c -> - DecoupledBranch( - inChannels = c, - midChannels = 64, - outChannels = regMax * 4, - baseName = "$name.cv2.$idx", - executionContext = executionContext - ) - } - private val clsBranches = ch.mapIndexed { idx, c -> - DecoupledBranch( - inChannels = c, - midChannels = 64, - outChannels = numClasses, - baseName = "$name.cv3.$idx", - executionContext = executionContext - ) - } - - // Holder for DFL projection weights so the ONNX initializer maps cleanly. - private val dfl = DflProjection(regMax, name = "$name.dfl", executionContext = executionContext) - - override val modules: List> = buildList { - addAll(regBranches) - addAll(clsBranches) - add(dfl) - } - - fun forward(inputs: List>, ctx: ExecutionContext): HeadOutputs { - require(inputs.size == 3) { "Detect head expects 3 feature maps, got ${inputs.size}" } - val reg = regBranches.mapIndexed { idx, branch -> branch.forward(inputs[idx], ctx) } - val cls = clsBranches.mapIndexed { idx, branch -> branch.forward(inputs[idx], ctx) } - return HeadOutputs( - small = HeadTensor(reg = reg[0], cls = cls[0]), - medium = HeadTensor(reg = reg[1], cls = cls[1]), - large = HeadTensor(reg = reg[2], cls = cls[2]) - ) - } - - override fun forward(input: Tensor, ctx: ExecutionContext): Tensor { - // Return a placeholder tensor (small cls) to satisfy Module contract; inference uses forward(inputs). - return clsBranches.first().forward(input, ctx) - } -} - -private class DecoupledBranch( - private val inChannels: Int, - midChannels: Int, - private val outChannels: Int, - baseName: String, - private val executionContext: ExecutionContext -) : Module() { - override val name: String = baseName - private val c1 = ConvSiLU( - inChannels = inChannels, - outChannels = midChannels, - kernel = 3, - stride = 1, - padding = 1, - name = "$baseName.0", - executionContext = executionContext - ) - private val c2 = ConvSiLU( - inChannels = midChannels, - outChannels = midChannels, - kernel = 3, - stride = 1, - padding = 1, - name = "$baseName.1", - executionContext = executionContext - ) - private val out = ConvNoAct( - inChannels = midChannels, - outChannels = outChannels, - kernel = 1, - stride = 1, - padding = 0, - name = "$baseName.2", - executionContext = executionContext - ) - - override val modules: List> = listOf(c1.module, c2.module, out.module) - - override fun forward(input: Tensor, ctx: ExecutionContext): Tensor { - val h1 = c1.forward(input, ctx) - val h2 = c2.forward(h1, ctx) - return out.forward(h2, ctx) - } -} - -private class ConvNoAct( - private val inChannels: Int, - private val outChannels: Int, - private val kernel: Int, - private val stride: Int, - private val padding: Int, - private val name: String, - executionContext: ExecutionContext -) { - val module: Module = definition { - network(executionContext) { - sequential { - stage(name) { - conv2d( - outChannels = outChannels, - kernelSize = kernel to kernel, - stride = stride to stride, - padding = padding to padding - ) { - this.inChannels = this@ConvNoAct.inChannels - this.bias = true - } - } - } - } - } - - fun forward(input: Tensor, ctx: ExecutionContext): Tensor = - module.forward(input, ctx) -} - -private class DflProjection( - regMax: Int, - override val name: String, - executionContext: ExecutionContext -) : Module(), sk.ainet.lang.nn.topology.ModuleParameters { - private val weightShape = sk.ainet.lang.tensor.Shape(intArrayOf(1, regMax, 1, 1)) - override val params: List> = - listOf( - sk.ainet.lang.nn.topology.ModuleParameter.WeightParameter( - "$name.conv.weight", - executionContext.zeros(weightShape, FP32::class) - ) - ) - override val modules: List> = emptyList() - - override fun forward(input: Tensor, ctx: ExecutionContext): Tensor = - input -} diff --git a/skainet-models/skainet-model-yolo/src/commonMain/kotlin/sk/ainet/lang/model/dnn/yolo/YoloConfig.kt b/skainet-models/skainet-model-yolo/src/commonMain/kotlin/sk/ainet/lang/model/dnn/yolo/YoloConfig.kt deleted file mode 100644 index 75fa64431..000000000 --- a/skainet-models/skainet-model-yolo/src/commonMain/kotlin/sk/ainet/lang/model/dnn/yolo/YoloConfig.kt +++ /dev/null @@ -1,25 +0,0 @@ -package sk.ainet.lang.model.dnn.yolo - -/** - * YOLO configuration used for decoding and thresholds. - */ -public data class YoloConfig( - val numClasses: Int = 80, - val inputSize: Int = 640, - val confThreshold: Float = 0.25f, - val iouThreshold: Float = 0.45f, - val maxDetections: Int = 300, - val classNames: List = emptyList(), - val regMax: Int = 16, - /** - * Channel multiplier used to size the backbone/neck to match ONNX weights. - * Example: YOLOv8n exported from Ultralytics uses baseChannels=16 (width_mult=0.25), - * while larger configs use higher numbers. - */ - val baseChannels: Int = 32, - /** - * Depth multiplier controlling how many bottlenecks each C2f block uses. - * Example: Ultralytics YOLOv8n uses ~0.33 which turns base counts [3,6,6,3] into [1,2,2,1]. - */ - val depthMultiple: Float = 1.0f -) diff --git a/skainet-models/skainet-model-yolo/src/commonMain/kotlin/sk/ainet/lang/model/dnn/yolo/YoloPost.kt b/skainet-models/skainet-model-yolo/src/commonMain/kotlin/sk/ainet/lang/model/dnn/yolo/YoloPost.kt deleted file mode 100644 index 5c73a089e..000000000 --- a/skainet-models/skainet-model-yolo/src/commonMain/kotlin/sk/ainet/lang/model/dnn/yolo/YoloPost.kt +++ /dev/null @@ -1,203 +0,0 @@ -package sk.ainet.lang.model.dnn.yolo - -import kotlin.math.exp -import kotlin.math.max -import kotlin.math.min -import sk.ainet.lang.tensor.Tensor -import sk.ainet.lang.types.FP32 - -public data class Detection( - val box: Box, - val score: Float, - val classId: Int, - val label: String? = null -) - -public data class Box( - val x1: Float, - val y1: Float, - val x2: Float, - val y2: Float -) { - val width: Float get() = max(0f, x2 - x1) - val height: Float get() = max(0f, y2 - y1) -} - -public data class YoloInput( - val tensor: Tensor, - val originalWidth: Int, - val originalHeight: Int, - val letterboxScale: Float = 1f, - val padW: Int = 0, - val padH: Int = 0 -) - -internal object YoloDecoder { - fun decode( - heads: HeadOutputs, - config: YoloConfig, - inputMeta: YoloInput - ): List { - val small = decodeHead(heads.small, stride = 8, config, inputMeta) - val medium = decodeHead(heads.medium, stride = 16, config, inputMeta) - val large = decodeHead(heads.large, stride = 32, config, inputMeta) - return nms(small + medium + large, config) - } - - private fun decodeHead( - head: HeadTensor, - stride: Int, - config: YoloConfig, - inputMeta: YoloInput - ): List { - val (batch, regChannels, h, w) = head.reg.shape.dimensions - val classes = config.numClasses - require(batch == 1) { "Only batch size 1 is supported for decode, got $batch" } - require(regChannels == config.regMax * 4) { - "Reg branch has $regChannels channels, expected ${config.regMax * 4}" - } - val clsShape = head.cls.shape.dimensions - require(clsShape[1] == classes) { "Cls branch has ${clsShape[1]} channels, expected $classes" } - - val out = mutableListOf() - val bins = FloatArray(config.regMax) { it.toFloat() } - var yIdx = 0 - while (yIdx < h) { - var xIdx = 0 - while (xIdx < w) { - val dist = FloatArray(4) - var c = 0 - while (c < 4) { - val start = c * config.regMax - val logits = FloatArray(config.regMax) { k -> - head.reg.data.get(0, start + k, yIdx, xIdx) - } - val probs = softmax(logits) - var exp = 0f - var k = 0 - while (k < config.regMax) { - exp += probs[k] * bins[k] - k++ - } - dist[c] = exp * stride - c++ - } - - val clsScore = argMaxClass(head.cls, yIdx, xIdx, classes) - if (clsScore.score >= config.confThreshold) { - val box = decodeBoxFromDistances(dist, xIdx, yIdx, stride, inputMeta) - val label = config.classNames.getOrNull(clsScore.classId) - out += Detection(box, clsScore.score, clsScore.classId, label) - } - xIdx++ - } - yIdx++ - } - return out - } - - private data class ClassScore(val classId: Int, val score: Float) - - private fun argMaxClass( - cls: Tensor, - y: Int, - x: Int, - classes: Int - ): ClassScore { - var bestScore = Float.NEGATIVE_INFINITY - var bestId = -1 - var c = 0 - while (c < classes) { - val v = cls.data.get(0, c, y, x) - if (v > bestScore) { - bestScore = v - bestId = c - } - c++ - } - return ClassScore(bestId, sigmoid(bestScore)) - } - - private fun decodeBoxFromDistances( - dist: FloatArray, - gridX: Int, - gridY: Int, - stride: Int, - meta: YoloInput - ): Box { - val centerX = (gridX + 0.5f) * stride - val centerY = (gridY + 0.5f) * stride - val x1 = centerX - dist[0] - val y1 = centerY - dist[1] - val x2 = centerX + dist[2] - val y2 = centerY + dist[3] - - val scale = meta.letterboxScale - val padW = meta.padW - val padH = meta.padH - val mappedX1 = (x1 - padW) / scale - val mappedY1 = (y1 - padH) / scale - val mappedX2 = (x2 - padW) / scale - val mappedY2 = (y2 - padH) / scale - - val x1c = mappedX1.coerceIn(0f, meta.originalWidth.toFloat()) - val y1c = mappedY1.coerceIn(0f, meta.originalHeight.toFloat()) - val x2c = mappedX2.coerceIn(0f, meta.originalWidth.toFloat()) - val y2c = mappedY2.coerceIn(0f, meta.originalHeight.toFloat()) - return Box(x1c, y1c, x2c, y2c) - } - - private fun softmax(logits: FloatArray): FloatArray { - var max = logits.maxOrNull() ?: 0f - var sum = 0f - val exp = FloatArray(logits.size) - var i = 0 - while (i < logits.size) { - val e = kotlin.math.exp(logits[i] - max) - exp[i] = e - sum += e - i++ - } - var j = 0 - while (j < exp.size) { - exp[j] /= sum - j++ - } - return exp - } - - private fun sigmoid(x: Float): Float = (1f / (1f + exp(-x))) - - private fun nms(detections: List, config: YoloConfig): List { - if (detections.isEmpty()) return emptyList() - val sorted = detections.sortedByDescending { it.score }.toMutableList() - val keep = mutableListOf() - while (sorted.isNotEmpty() && keep.size < config.maxDetections) { - val best = sorted.removeAt(0) - keep += best - val iterator = sorted.iterator() - while (iterator.hasNext()) { - val other = iterator.next() - if (best.classId == other.classId) { - val iou = iou(best.box, other.box) - if (iou > config.iouThreshold) { - iterator.remove() - } - } - } - } - return keep - } - - private fun iou(a: Box, b: Box): Float { - val interX1 = max(a.x1, b.x1) - val interY1 = max(a.y1, b.y1) - val interX2 = min(a.x2, b.x2) - val interY2 = min(a.y2, b.y2) - val interW = max(0f, interX2 - interX1) - val interH = max(0f, interY2 - interY1) - val interArea = interW * interH - val union = a.width * a.height + b.width * b.height - interArea - return if (union <= 0f) 0f else interArea / union - } -} diff --git a/skainet-models/skainet-model-yolo/src/commonMain/kotlin/sk/ainet/lang/model/dnn/yolo/YoloPreprocess.kt b/skainet-models/skainet-model-yolo/src/commonMain/kotlin/sk/ainet/lang/model/dnn/yolo/YoloPreprocess.kt deleted file mode 100644 index d0a68e1fe..000000000 --- a/skainet-models/skainet-model-yolo/src/commonMain/kotlin/sk/ainet/lang/model/dnn/yolo/YoloPreprocess.kt +++ /dev/null @@ -1,52 +0,0 @@ -package sk.ainet.lang.model.dnn.yolo - -import sk.ainet.lang.tensor.Tensor -import sk.ainet.lang.types.FP32 - -/** - * Preprocessing helpers for YOLOv8 inputs. - * - * For now we assume the caller supplies an already resized and normalized NCHW tensor - * matching [YoloConfig.inputSize]. This keeps dependencies minimal and avoids - * platform-specific image handling. Letterbox metadata is carried to map boxes - * back to the original image size. - */ -public object YoloPreprocess { - - /** - * Wraps a tensor that is already in the expected model shape [1, 3, inputSize, inputSize]. - * - * @param tensor normalized tensor (values typically in [0,1]) - * @param originalWidth width of the source image - * @param originalHeight height of the source image - * @param inputSize target model resolution (defaults to config.inputSize) - * @param padW horizontal padding applied during letterbox (pixels) - * @param padH vertical padding applied during letterbox (pixels) - * @param scale scaling factor used during letterbox (original -> model space) - */ - public fun fromReadyTensor( - tensor: Tensor, - originalWidth: Int, - originalHeight: Int, - inputSize: Int, - padW: Int = 0, - padH: Int = 0, - scale: Float = 1f - ): YoloInput { - require(tensor.shape.rank == 4) { "Expected NCHW tensor rank 4, got ${tensor.shape.rank}" } - require(tensor.shape[0] == 1 && tensor.shape[1] == 3) { - "Expected batch=1 and channels=3, got shape ${tensor.shape.dimensions.contentToString()}" - } - require(tensor.shape[2] == inputSize && tensor.shape[3] == inputSize) { - "Tensor must already be resized to $inputSize, got ${tensor.shape[2]}x${tensor.shape[3]}" - } - return YoloInput( - tensor = tensor, - originalWidth = originalWidth, - originalHeight = originalHeight, - letterboxScale = scale, - padW = padW, - padH = padH - ) - } -}