Skip to content

Fix ten bugs found reviewing src/, three of them memory-safety; release 1.2.0 - #62

Merged
marcobambini merged 3 commits into
mainfrom
fix/source-review-hardening
Nov 1, 2026
Merged

marcobambini merged 3 commits into
mainfrom
fix/source-review-hardening

Conversation

@marcobambini

Copy link
Copy Markdown
Member

Ten bugs found by reading through src/, then by a security review of the whole of it. Three write outside their buffers and are reachable from ordinary SQL. Each one was reproduced under AddressSanitizer before being fixed, and each has a regression test.

Release as 1.2.0. No API changed and no index needs rebuilding.

Memory safety

What Where Manifests as
Top-k arrays allocated from a size_t byte count through sqlite3_malloc()'s int parameter vCursorFilterCommon crash
Buffer sized and write-bounded from two different numbers vector_from_json controlled heap write
dimension unbounded, overflowing the size arithmetic downstream option parsing → turbo_rotation_plan_init write through NULL

k comes straight from the query, so vector_full_scan('t','v',vec,536870913) asked for 0x100000008 bytes, got 8, and the memset that follows overran them. Not controllable: the value written is zero and the length is fixed, so it is a crash. Both allocations now use sqlite3_malloc64().

vector_from_json took its buffer from sqlite3_malloc((int)alloc), truncating a size_t, while the write bound came from the untruncated alloc. A JSON array with 2^30 commas got an 8-byte buffer and a bound of 1073741825 elements, and the parse loop wrote floats the caller supplies straight past it — the comma count sizes the buffer to the byte, so this is a precise write primitive. Reachable through any vector_as_*() and through the TEXT query-vector argument of both scans. It needs a TEXT value of about a gigabyte, so a host SQLite built with SQLITE_MAX_LENGTH >= 1073741824; the limit that applies is the host's, not the bundled amalgamation's, since this ships as a loadable extension. There is now one capacity in 64 bits, with the allocation and the bound both taken from it.

dimension was only checked for being positive, so it reached INT_MAX. (dim/2)*12 in the TurboQuant rotation plan wraps negative past 357913941, which skips allocating the plan's coordinate arrays — they sit behind if (pair_count > 0), and so does the check that would have caught their absence — and the loop then writes through the resulting null pointer. vector_init with dimension=400000000 plus a TurboQuant vector_quantize crashed the process on an empty table, since the plan is built before any row is read. Dimensions are now bounded at 2^24 where they are parsed, upstream of every one of those computations and of the signed-overflow dim + 7 expressions in the packing helpers. A value above INT_MAX is also rejected rather than wrapped: 4294967300 used to come through as the dimension 4.

Also fixed

  • A NULL key in _sqliteai_vector reached strcmp() and crashed the next vector_init(). That table is ordinary and writable, and PRIMARY KEY on a rowid table does not imply NOT NULL.
  • A failed vector_quantize() left the context describing an index the rollback had undone, so scans read the chunks at the wrong stride and returned made-up rowids and distances with no error.
  • A quant chunk claiming zero vectors was read past the end of its blob: zero is the streaming scans' "load the next chunk" sentinel, so it slipped past the length check beside it, which degenerates to bytes < 0.
  • vector_as_*() ignored its optional dimension argument for BLOB input, so a wrong-sized vector was stored and then skipped in silence by every scan.
  • max_memory=0 could not be set and a bad value was dropped silently: the parser used a 0 return for failure, and zero is the one value the quantizer reads as "use whatever the index needs".
  • The INT8/UINT8 cosine kernels accumulated in 32 bits in the scalar and SSE2 backends. SSE2 now also computes sqrt(a)*sqrt(b) instead of sqrtf(a*b), so all five backends compute the same cosine.

For the reviewer

  • Package.swift is deliberately untouched although it still says 1.1.2. Its URL and checksum name a release artifact that does not exist yet, and CI rewrites both after the tag publishes one (see 0c2223a … [auto-update]). Editing the URL by hand would leave Swift consumers on 1.2.0 with 1.1.2's checksum.
  • The [Unreleased] changelog lead was removed, not kept. It said the extension was unchanged from 1.1.0 and that a build produced the same binaries — true while that section held only benchmark and README work, false for a release carrying three out-of-bounds writes. changelog.yml publishes this file to the website on the tag, so it is what someone reads to decide whether to upgrade.
  • The gigabyte JSON case is covered by a standalone probe, not by the suite. Running it needs that much memory and a different SQLITE_MAX_LENGTH than the project builds with, so putting it in make unittest would cost a gigabyte per CI run. Everything else is in test_malformed_input().
  • Coverage limits worth knowing. NEON was exercised natively under ASan. The three x86 backends were only compiled, with their ISAs enabled — there is no x86 host here and Rosetta was unavailable. RVV was read, not run: no RISC-V toolchain. For those four, loop bounds were checked statically only.

Verification

make unittest: 1482 passed, 0 failed (1447 before this branch), identical under ASan + UBSan. No warnings under -Wall -Wextra -Wshadow, on arm64 and on all three x86 backends cross-compiled with their ISAs. The universal dylib builds and loads, reporting 1.2.0. Both exploit probes now return a clean error instead of writing out of bounds.

🤖 Generated with Claude Code

marcobambini and others added 3 commits November 1, 2026 07:54
Two of them are memory-safety faults reachable from ordinary SQL, the rest are
correctness or validation gaps. Each one was reproduced before being fixed and
has a regression test in test_malformed_input().

vCursorFilterCommon() passed a size_t byte count to sqlite3_malloc(), whose
parameter is an int. k comes straight from the query, so k=536870913 asked for
0x100000008 bytes, got 8, and the memset that follows overran the top-k arrays.
Both arrays now go through sqlite3_malloc64(), which refuses anything near 2GB,
so an absurd k is a clean SQLITE_NOMEM. row_count also drops to zero before
either allocation: it is what tells the next filter whether the arrays can be
reused, and a cursor that failed between the two used to keep the old count
beside a freed array.

sqlite_unserialize() passed sqlite3_column_text() straight to strcmp().
_sqliteai_vector is an ordinary writable table, and PRIMARY KEY on a rowid table
does not imply NOT NULL, so one row with a NULL key crashed the next
vector_init().

vector_rebuild_quantization() writes q_type/q_bits/scale/offset into the table
context before the pass that can still fail, and vector_quantize() rolls the
shadow table back when it does. The context was left describing a layout the
table no longer had, and the next scan read the chunks at the wrong stride -
wrong rowids, wrong distances, no error at all. Those fields are now snapshotted
and restored on the cleanup path, together with the TurboQuant cache that
rebuild() drops whenever it quantizes to anything else.

Zero is the streaming scans' "load the next chunk" sentinel, so a chunk claiming
no vectors slipped past the length check beside it, which degenerates to
"bytes < 0", and a row was read from past the end of its blob. Every count test
is now <= 0, and preload rejects such a chunk when it loads rather than leaving
it to query time.

vector_as_*() ignored its optional dimension argument for non-BIT blobs, though
API.md documents it as a stricter check and both the BIT and the JSON paths
already enforced it. A wrong-sized vector was stored happily and then skipped in
silence by every scan, so it never appeared in a result. The new check reports the
size it wanted with %lld, because for a float type that size is dimension * 4 and
can exceed INT_MAX - with a %d it said "expected 0 bytes". The BIT branch beside
it moves to the same format for consistency, though there the value is
(dimension + 7) / 8 and cannot overflow. context_result_error() carries no printf
format attribute, so only the new runtime test catches this.

human_to_number() signalled failure by returning 0, which also made a literal
"0" unparseable - and zero is the one value the quantizer reads as "use whatever
memory the index needs", so that setting could not be expressed through the
option string. It now reports success separately, and an unparseable max_memory
is an error like every other bad option instead of being dropped.

The int8 and uint8 cosine kernels of the scalar and SSE2 backends accumulated in
32 bits, which wrapped around past roughly 133k and 66k dimensions; NEON, AVX2
and AVX-512 already used 64-bit accumulators. SSE2 now also computes
sqrt(a)*sqrt(b) like every other backend instead of sqrtf(a*b), which could
overflow in float, so the five backends compute the same cosine. The vector lane
accumulators in SSE2 and RVV are still 32-bit: widening those means
restructuring the kernels, and the limit is above half a million dimensions.

Also renamed a local in vector_rebuild_quantization() that shadowed the caller's
out-parameter, which was the build's only warning under -Wshadow.

make unittest: 1473 passed, 0 failed, and the same under ASan and UBSan.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A security review of all of src/, not only this branch's own changes, turned up a
heap buffer overflow and a null write. Both are reachable from ordinary SQL and both
were reproduced under AddressSanitizer before being fixed.

vector_from_json() sized its buffer and bounded its writes from two different
numbers. The buffer came from sqlite3_malloc((int)alloc), which truncated a size_t to
that function's int parameter, while max_count was still derived from the untruncated
alloc. A JSON array with 2^30 commas therefore got an 8-byte buffer and a bound of
1073741825 elements, and the parse loop wrote floats of the caller's choosing straight
past it; the comma count sizes the buffer to the byte, so this was a precise write
primitive rather than a crash. There is now a single int64_t capacity: the allocation
goes through sqlite3_malloc64() and the loop checks against that same capacity instead
of a figure recovered from the allocation, so the two cannot drift apart however the
arithmetic behaves. Reaching it took a TEXT value of about a gigabyte, so a host built
with SQLITE_MAX_LENGTH >= 1073741824 - the amalgamation bundled here uses the 1e9
default, but this ships as a loadable extension and runs against the host's limits,
not against these.

The declared dimension was only checked for being positive, so it reached INT_MAX and
overflowed the size arithmetic downstream. In turbo_rotation_plan_init() the
expression (dim/2)*12 wraps negative past 357913941, which skipped the allocation of
the rotation plan's coordinate arrays - they sit behind if (pair_count > 0), and so
does the test that would have caught their absence - and the loop then wrote through
the resulting null pointer. A vector_init with dimension=400000000 followed by a
TurboQuant vector_quantize crashed the process on an empty table, since the plan is
built before any row is read. The dimension is now range-checked where it is parsed,
upstream of every one of those computations, against a MAX_VECTOR_DIMENSION of 2^24:
three orders of magnitude above any real embedding, and small enough that (dim/2)*12
stays near a hundred million. It is also held in a long until it has been checked,
because casting strtol()'s result to int first wrapped and let "4294967300" through as
the dimension 4.

That same bound now applies to the optional dimension argument of vector_as_*(), whose
size check goes through vector_bytes_for_dim(): dim * item_size wraps where size_t is
32 bits, and a wrapped product would let a wrong-sized blob match. That check was
added two commits ago, so this closes a gap of its own making.

One consequence worth recording: with the dimension bounded, no accepted value yields
a byte count above INT_MAX, so the %lld conversion in those rejection messages is no
longer observable from SQL. Its test now pins the message on an in-range mismatch and
checks separately that a dimension past the bound is refused before any size is
computed at all.

The gigabyte case is covered by a standalone probe rather than by the suite, since
running it needs that much memory and a different SQLITE_MAX_LENGTH than the project
builds with. Everything else is in test_malformed_input().

make unittest: 1482 passed, 0 failed, and the same under ASan and UBSan.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Promotes the Unreleased section to 1.2.0 and records this branch's ten fixes under it.
That section's lead paragraph had to go: it said the extension was unchanged from 1.1.0
and that a build produced the same binaries, which was true while Unreleased held only
benchmark and README work, and is not true of a release carrying three out-of-bounds
writes. Those three are called out under Security with how each is reached, because this
file is what the release tag publishes to the website and what somebody reads to decide
whether to upgrade.

Package.swift is deliberately untouched. Its URL and checksum name a release artifact
that does not exist yet, and CI rewrites both once the tag publishes one, in the commit it marks as an
automated update. Bumping the URL by hand now would leave Swift consumers pointing at
1.2.0 with 1.1.2's checksum.

(Deliberately not quoting CI's sentinel string here: main.yml gates every job on the
head commit message not containing it, and the whole build skipped when it did.)

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@marcobambini
marcobambini force-pushed the fix/source-review-hardening branch from 2f29c0e to 62daad7 Compare November 1, 2026 07:24
@marcobambini
marcobambini merged commit 1ad6bb3 into main Nov 1, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant