Fix ten bugs found reviewing src/, three of them memory-safety; release 1.2.0

CHANGELOG.md

@@ -4,10 +4,12 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
-## [Unreleased]
+## [1.2.0] - 2026-11-01
 
-Documentation and tooling only — the extension itself is unchanged from 1.1.0, so a build
-from this revision produces the same binaries.
+A read-through of the sources, followed by a security review of all of `src/`, turned up
+ten bugs. Three of them write outside their buffers and are reachable from ordinary SQL,
+so upgrading is recommended for anyone whose application lets untrusted input reach a
+connection with the extension loaded. No API changed and no index needs rebuilding.
 
 ### Changed
 
@@ -32,6 +34,63 @@ from this revision produces the same binaries.
   faster** than the exact scan — it does beat brute force. What holds is the comparison
   that mattered: `INT8` is 4x faster again at twice the index size, so TurboQuant's
   argument is memory rather than throughput.
+- **A `NULL` key in `_sqliteai_vector` crashed the next `vector_init()`.** That table is
+  an ordinary writable one, and `PRIMARY KEY` on a rowid table does not imply `NOT NULL`,
+  so the key went to `strcmp()` unchecked. Such rows are now skipped.
+- **A failed `vector_quantize()` left the extension describing an index that no longer
+  existed.** The quantization type, bit width, scale and offset are written into the table
+  context before the pass that can still fail, and the savepoint rolls the shadow table
+  back without them. Scans then read the chunks at the wrong stride and reported made-up
+  rowids and distances with no error at all. Those fields, and the TurboQuant cache, are
+  now restored along with the rollback.
+- **A quant chunk claiming zero vectors was read past the end of its blob.** Zero is the
+  streaming scans' "load the next chunk" sentinel, so such a chunk slipped past the length
+  check beside it, which degenerates to `bytes < 0`, and one fabricated row came back.
+  Every count test now rejects it, and `vector_quantize_preload()` catches it at load time
+  rather than leaving it to query time.
+- **`vector_as_*()` ignored its optional dimension argument for BLOB input**, though it is
+  documented as a stricter check and the `BIT` and JSON paths already enforced it. A
+  wrong-sized vector was stored and then skipped in silence by every scan, so it never
+  appeared in a result.
+- **`max_memory=0` could not be set, and an unparseable value was dropped silently.** Zero
+  is the one value the quantizer reads as "use whatever memory the index needs", but the
+  parser used a `0` return to signal failure, so the two were indistinguishable. A bad
+  `max_memory` is now reported like every other invalid option.
+- **The `INT8` and `UINT8` cosine kernels accumulated in 32 bits** in the scalar and SSE2
+  backends, wrapping around past roughly 133k and 66k dimensions; NEON, AVX2 and AVX-512
+  already used 64-bit accumulators. SSE2 now also computes `sqrt(a)*sqrt(b)` like every
+  other backend instead of `sqrtf(a*b)`, which could overflow in `float`, so all five
+  compute the same cosine.
+
+### Security
+
+- **Out-of-bounds write from a large `k`.** The top-k arrays were allocated with
+  `sqlite3_malloc()`, whose parameter is an `int`, from a `size_t` byte count. `k` comes
+  straight from the query, so `k=536870913` asked for `0x100000008` bytes, got 8, and the
+  `memset` that follows overran them. Both allocations now go through
+  `sqlite3_malloc64()`, so an absurd `k` is a clean `SQLITE_NOMEM`. Manifests as a crash:
+  the length is not controllable and the value written is zero.
+- **Heap buffer overflow in JSON vector parsing.** `vector_from_json()` sized its buffer
+  and bounded its writes from two different numbers — the buffer from
+  `sqlite3_malloc((int)alloc)`, which truncated a `size_t`, and the bound from the
+  untruncated `alloc`. A JSON array with 2^30 commas got an 8-byte buffer and a bound of
+  1073741825 elements, and the parse loop wrote `float`s of the caller's choosing straight
+  past it; the comma count sizes the buffer to the byte, which makes this a precise write
+  primitive rather than a crash. Reachable through any `vector_as_*()` and through the TEXT
+  query-vector argument of both scans. It took a TEXT value of about a gigabyte, so a host
+  built with `SQLITE_MAX_LENGTH >= 1073741824` — the limit that applies is the host's,
+  since this ships as a loadable extension. There is now a single capacity in 64 bits: the
+  allocation uses `sqlite3_malloc64()` and the loop checks against that same number.
+- **Null write from an unbounded `dimension`.** The declared dimension was only checked
+  for being positive, so it reached `INT_MAX` and overflowed the size arithmetic
+  downstream. `(dim/2)*12` in the TurboQuant rotation plan wraps negative past 357913941,
+  which skipped the allocation of the plan's coordinate arrays and still reported the plan
+  as valid; `vector_init` with `dimension=400000000` followed by a TurboQuant
+  `vector_quantize` then crashed the process on an empty table, since the plan is built
+  before any row is read. Dimensions are now bounded at 2^24 where they are parsed, which
+  is upstream of every one of those computations — and of the signed-overflow `dim + 7`
+  expressions in the packing helpers. A dimension above `INT_MAX` is also rejected rather
+  than wrapped: `4294967300` used to come through as the dimension 4.
 
 ## [1.1.0] - 2026-08-24
 

src/distance-cpu.c

@@ -515,9 +515,11 @@ float uint8_distance_cosine_cpu (const void *v1, const void *v2, int n) {
     const uint8_t *a = (const uint8_t *)v1;
     const uint8_t *b = (const uint8_t *)v2;
 
-    uint32_t dot = 0;
-    uint32_t norm_a2 = 0;
-    uint32_t norm_b2 = 0;
+    // 64-bit for the same reason as the int8 kernel below: a product is at most 255*255,
+    // so uint32_t wrapped around silently past roughly 66k dimensions
+    uint64_t dot = 0;
+    uint64_t norm_a2 = 0;
+    uint64_t norm_b2 = 0;
 
     int i = 0;
     for (; i <= n - 4; i += 4) {
@@ -544,7 +546,7 @@ float uint8_distance_cosine_cpu (const void *v1, const void *v2, int n) {
         return 1.0f;
     }
 
-    float cosine_similarity = dot / (sqrtf((float)norm_a2) * sqrtf((float)norm_b2));
+    float cosine_similarity = (float)((double)dot / (sqrt((double)norm_a2) * sqrt((double)norm_b2)));
     if (cosine_similarity > 1.0f) cosine_similarity = 1.0f;
     if (cosine_similarity < -1.0f) cosine_similarity = -1.0f;
     return 1.0f - cosine_similarity;
@@ -629,9 +631,12 @@ float int8_distance_cosine_cpu (const void *v1, const void *v2, int n) {
     const int8_t *a = (const int8_t *)v1;
     const int8_t *b = (const int8_t *)v2;
 
-    int32_t dot = 0;
-    int32_t norm_a2 = 0;
-    int32_t norm_b2 = 0;
+    // 64-bit accumulators, as the NEON/AVX2/AVX-512 kernels use: a product is at most
+    // 127*127, so int32_t saturated at roughly 133k dimensions and the loop wrapped
+    // around silently. Each group of four products still fits in 32 bits.
+    int64_t dot = 0;
+    int64_t norm_a2 = 0;
+    int64_t norm_b2 = 0;
 
     int i = 0;
     for (; i <= n - 4; i += 4) {
@@ -658,7 +663,7 @@ float int8_distance_cosine_cpu (const void *v1, const void *v2, int n) {
         return 1.0f;
     }
 
-    float cosine_similarity = dot / (sqrtf((float)norm_a2) * sqrtf((float)norm_b2));
+    float cosine_similarity = (float)((double)dot / (sqrt((double)norm_a2) * sqrt((double)norm_b2)));
     if (cosine_similarity > 1.0f) cosine_similarity = 1.0f;
     if (cosine_similarity < -1.0f) cosine_similarity = -1.0f;
     return 1.0f - cosine_similarity;

src/distance-sse2.c

@@ -786,9 +786,11 @@ float uint8_distance_cosine_sse2 (const void *v1, const void *v2, int n) {
     _mm_storeu_si128((__m128i *)a2, acc_a2);
     _mm_storeu_si128((__m128i *)b2, acc_b2);
 
-    int32_t total_dot = dot[0] + dot[1] + dot[2] + dot[3];
-    int32_t total_a2  = a2[0] + a2[1] + a2[2] + a2[3];
-    int32_t total_b2  = b2[0] + b2[1] + b2[2] + b2[3];
+    // 64-bit totals, as the NEON/AVX2/AVX-512 kernels use: the four lanes together can
+    // hold more than a 32-bit sum, and the scalar tail adds to the same running total
+    int64_t total_dot = (int64_t)dot[0] + dot[1] + dot[2] + dot[3];
+    int64_t total_a2  = (int64_t)a2[0] + a2[1] + a2[2] + a2[3];
+    int64_t total_b2  = (int64_t)b2[0] + b2[1] + b2[2] + b2[3];
 
     for (; i < n; ++i) {
         int va = (int)a[i];
@@ -798,9 +800,9 @@ float uint8_distance_cosine_sse2 (const void *v1, const void *v2, int n) {
         total_b2  += vb * vb;
     }
 
-    float denom = sqrtf((float)total_a2 * (float)total_b2);
-    if (denom == 0.0f) return 1.0f; // orthogonal or zero
-    float cosine_sim = total_dot / denom;
+    double denom = sqrt((double)total_a2) * sqrt((double)total_b2);
+    if (denom == 0.0) return 1.0f; // orthogonal or zero
+    float cosine_sim = (float)((double)total_dot / denom);
     if (cosine_sim > 1.0f) cosine_sim = 1.0f;
     if (cosine_sim < -1.0f) cosine_sim = -1.0f;
     return 1.0f - cosine_sim; // cosine distance
@@ -1003,9 +1005,11 @@ float int8_distance_cosine_sse2 (const void *v1, const void *v2, int n) {
     _mm_storeu_si128((__m128i *)_a, acc_a2);
     _mm_storeu_si128((__m128i *)_b, acc_b2);
 
-    int32_t total_dot = _d[0] + _d[1] + _d[2] + _d[3];
-    int32_t total_a2  = _a[0] + _a[1] + _a[2] + _a[3];
-    int32_t total_b2  = _b[0] + _b[1] + _b[2] + _b[3];
+    // 64-bit totals, as the NEON/AVX2/AVX-512 kernels use: the four lanes together can
+    // hold more than a 32-bit sum, and the scalar tail adds to the same running total
+    int64_t total_dot = (int64_t)_d[0] + _d[1] + _d[2] + _d[3];
+    int64_t total_a2  = (int64_t)_a[0] + _a[1] + _a[2] + _a[3];
+    int64_t total_b2  = (int64_t)_b[0] + _b[1] + _b[2] + _b[3];
 
     // Handle tail
     for (; i < n; ++i) {
@@ -1016,9 +1020,9 @@ float int8_distance_cosine_sse2 (const void *v1, const void *v2, int n) {
         total_b2  += vb * vb;
     }
 
-    float denom = sqrtf((float)total_a2 * (float)total_b2);
-    if (denom == 0.0f) return 1.0f;
-    float cosine_sim = total_dot / denom;
+    double denom = sqrt((double)total_a2) * sqrt((double)total_b2);
+    if (denom == 0.0) return 1.0f;
+    float cosine_sim = (float)((double)total_dot / denom);
     if (cosine_sim > 1.0f) cosine_sim = 1.0f;
     if (cosine_sim < -1.0f) cosine_sim = -1.0f;
     return 1.0f - cosine_sim;