Repository navigation
Fix ten bugs found reviewing src/, three of them memory-safety; release 1.2.0 - #62
Merged
Merged
Conversation
Two of them are memory-safety faults reachable from ordinary SQL, the rest are correctness or validation gaps. Each one was reproduced before being fixed and has a regression test in test_malformed_input(). vCursorFilterCommon() passed a size_t byte count to sqlite3_malloc(), whose parameter is an int. k comes straight from the query, so k=536870913 asked for 0x100000008 bytes, got 8, and the memset that follows overran the top-k arrays. Both arrays now go through sqlite3_malloc64(), which refuses anything near 2GB, so an absurd k is a clean SQLITE_NOMEM. row_count also drops to zero before either allocation: it is what tells the next filter whether the arrays can be reused, and a cursor that failed between the two used to keep the old count beside a freed array. sqlite_unserialize() passed sqlite3_column_text() straight to strcmp(). _sqliteai_vector is an ordinary writable table, and PRIMARY KEY on a rowid table does not imply NOT NULL, so one row with a NULL key crashed the next vector_init(). vector_rebuild_quantization() writes q_type/q_bits/scale/offset into the table context before the pass that can still fail, and vector_quantize() rolls the shadow table back when it does. The context was left describing a layout the table no longer had, and the next scan read the chunks at the wrong stride - wrong rowids, wrong distances, no error at all. Those fields are now snapshotted and restored on the cleanup path, together with the TurboQuant cache that rebuild() drops whenever it quantizes to anything else. Zero is the streaming scans' "load the next chunk" sentinel, so a chunk claiming no vectors slipped past the length check beside it, which degenerates to "bytes < 0", and a row was read from past the end of its blob. Every count test is now <= 0, and preload rejects such a chunk when it loads rather than leaving it to query time. vector_as_*() ignored its optional dimension argument for non-BIT blobs, though API.md documents it as a stricter check and both the BIT and the JSON paths already enforced it. A wrong-sized vector was stored happily and then skipped in silence by every scan, so it never appeared in a result. The new check reports the size it wanted with %lld, because for a float type that size is dimension * 4 and can exceed INT_MAX - with a %d it said "expected 0 bytes". The BIT branch beside it moves to the same format for consistency, though there the value is (dimension + 7) / 8 and cannot overflow. context_result_error() carries no printf format attribute, so only the new runtime test catches this. human_to_number() signalled failure by returning 0, which also made a literal "0" unparseable - and zero is the one value the quantizer reads as "use whatever memory the index needs", so that setting could not be expressed through the option string. It now reports success separately, and an unparseable max_memory is an error like every other bad option instead of being dropped. The int8 and uint8 cosine kernels of the scalar and SSE2 backends accumulated in 32 bits, which wrapped around past roughly 133k and 66k dimensions; NEON, AVX2 and AVX-512 already used 64-bit accumulators. SSE2 now also computes sqrt(a)*sqrt(b) like every other backend instead of sqrtf(a*b), which could overflow in float, so the five backends compute the same cosine. The vector lane accumulators in SSE2 and RVV are still 32-bit: widening those means restructuring the kernels, and the limit is above half a million dimensions. Also renamed a local in vector_rebuild_quantization() that shadowed the caller's out-parameter, which was the build's only warning under -Wshadow. make unittest: 1473 passed, 0 failed, and the same under ASan and UBSan. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A security review of all of src/, not only this branch's own changes, turned up a heap buffer overflow and a null write. Both are reachable from ordinary SQL and both were reproduced under AddressSanitizer before being fixed. vector_from_json() sized its buffer and bounded its writes from two different numbers. The buffer came from sqlite3_malloc((int)alloc), which truncated a size_t to that function's int parameter, while max_count was still derived from the untruncated alloc. A JSON array with 2^30 commas therefore got an 8-byte buffer and a bound of 1073741825 elements, and the parse loop wrote floats of the caller's choosing straight past it; the comma count sizes the buffer to the byte, so this was a precise write primitive rather than a crash. There is now a single int64_t capacity: the allocation goes through sqlite3_malloc64() and the loop checks against that same capacity instead of a figure recovered from the allocation, so the two cannot drift apart however the arithmetic behaves. Reaching it took a TEXT value of about a gigabyte, so a host built with SQLITE_MAX_LENGTH >= 1073741824 - the amalgamation bundled here uses the 1e9 default, but this ships as a loadable extension and runs against the host's limits, not against these. The declared dimension was only checked for being positive, so it reached INT_MAX and overflowed the size arithmetic downstream. In turbo_rotation_plan_init() the expression (dim/2)*12 wraps negative past 357913941, which skipped the allocation of the rotation plan's coordinate arrays - they sit behind if (pair_count > 0), and so does the test that would have caught their absence - and the loop then wrote through the resulting null pointer. A vector_init with dimension=400000000 followed by a TurboQuant vector_quantize crashed the process on an empty table, since the plan is built before any row is read. The dimension is now range-checked where it is parsed, upstream of every one of those computations, against a MAX_VECTOR_DIMENSION of 2^24: three orders of magnitude above any real embedding, and small enough that (dim/2)*12 stays near a hundred million. It is also held in a long until it has been checked, because casting strtol()'s result to int first wrapped and let "4294967300" through as the dimension 4. That same bound now applies to the optional dimension argument of vector_as_*(), whose size check goes through vector_bytes_for_dim(): dim * item_size wraps where size_t is 32 bits, and a wrapped product would let a wrong-sized blob match. That check was added two commits ago, so this closes a gap of its own making. One consequence worth recording: with the dimension bounded, no accepted value yields a byte count above INT_MAX, so the %lld conversion in those rejection messages is no longer observable from SQL. Its test now pins the message on an in-range mismatch and checks separately that a dimension past the bound is refused before any size is computed at all. The gigabyte case is covered by a standalone probe rather than by the suite, since running it needs that much memory and a different SQLITE_MAX_LENGTH than the project builds with. Everything else is in test_malformed_input(). make unittest: 1482 passed, 0 failed, and the same under ASan and UBSan. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Promotes the Unreleased section to 1.2.0 and records this branch's ten fixes under it. That section's lead paragraph had to go: it said the extension was unchanged from 1.1.0 and that a build produced the same binaries, which was true while Unreleased held only benchmark and README work, and is not true of a release carrying three out-of-bounds writes. Those three are called out under Security with how each is reached, because this file is what the release tag publishes to the website and what somebody reads to decide whether to upgrade. Package.swift is deliberately untouched. Its URL and checksum name a release artifact that does not exist yet, and CI rewrites both once the tag publishes one, in the commit it marks as an automated update. Bumping the URL by hand now would leave Swift consumers pointing at 1.2.0 with 1.1.2's checksum. (Deliberately not quoting CI's sentinel string here: main.yml gates every job on the head commit message not containing it, and the whole build skipped when it did.) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
marcobambini
force-pushed
the
fix/source-review-hardening
branch
from
November 1, 2026 07:24
2f29c0e to
62daad7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ten bugs found by reading through
src/, then by a security review of the whole of it. Three write outside their buffers and are reachable from ordinary SQL. Each one was reproduced under AddressSanitizer before being fixed, and each has a regression test.Release as 1.2.0. No API changed and no index needs rebuilding.
Memory safety
size_tbyte count throughsqlite3_malloc()'sintparametervCursorFilterCommonvector_from_jsondimensionunbounded, overflowing the size arithmetic downstreamturbo_rotation_plan_initNULLkcomes straight from the query, sovector_full_scan('t','v',vec,536870913)asked for0x100000008bytes, got 8, and thememsetthat follows overran them. Not controllable: the value written is zero and the length is fixed, so it is a crash. Both allocations now usesqlite3_malloc64().vector_from_jsontook its buffer fromsqlite3_malloc((int)alloc), truncating asize_t, while the write bound came from the untruncatedalloc. A JSON array with 2^30 commas got an 8-byte buffer and a bound of 1073741825 elements, and the parse loop wrotefloats the caller supplies straight past it — the comma count sizes the buffer to the byte, so this is a precise write primitive. Reachable through anyvector_as_*()and through the TEXT query-vector argument of both scans. It needs a TEXT value of about a gigabyte, so a host SQLite built withSQLITE_MAX_LENGTH >= 1073741824; the limit that applies is the host's, not the bundled amalgamation's, since this ships as a loadable extension. There is now one capacity in 64 bits, with the allocation and the bound both taken from it.dimensionwas only checked for being positive, so it reachedINT_MAX.(dim/2)*12in the TurboQuant rotation plan wraps negative past 357913941, which skips allocating the plan's coordinate arrays — they sit behindif (pair_count > 0), and so does the check that would have caught their absence — and the loop then writes through the resulting null pointer.vector_initwithdimension=400000000plus a TurboQuantvector_quantizecrashed the process on an empty table, since the plan is built before any row is read. Dimensions are now bounded at 2^24 where they are parsed, upstream of every one of those computations and of the signed-overflowdim + 7expressions in the packing helpers. A value aboveINT_MAXis also rejected rather than wrapped:4294967300used to come through as the dimension 4.Also fixed
NULLkey in_sqliteai_vectorreachedstrcmp()and crashed the nextvector_init(). That table is ordinary and writable, andPRIMARY KEYon a rowid table does not implyNOT NULL.vector_quantize()left the context describing an index the rollback had undone, so scans read the chunks at the wrong stride and returned made-up rowids and distances with no error.bytes < 0.vector_as_*()ignored its optional dimension argument for BLOB input, so a wrong-sized vector was stored and then skipped in silence by every scan.max_memory=0could not be set and a bad value was dropped silently: the parser used a0return for failure, and zero is the one value the quantizer reads as "use whatever the index needs".INT8/UINT8cosine kernels accumulated in 32 bits in the scalar and SSE2 backends. SSE2 now also computessqrt(a)*sqrt(b)instead ofsqrtf(a*b), so all five backends compute the same cosine.For the reviewer
Package.swiftis deliberately untouched although it still says1.1.2. Its URL and checksum name a release artifact that does not exist yet, and CI rewrites both after the tag publishes one (see0c2223a … [auto-update]). Editing the URL by hand would leave Swift consumers on 1.2.0 with 1.1.2's checksum.[Unreleased]changelog lead was removed, not kept. It said the extension was unchanged from 1.1.0 and that a build produced the same binaries — true while that section held only benchmark and README work, false for a release carrying three out-of-bounds writes.changelog.ymlpublishes this file to the website on the tag, so it is what someone reads to decide whether to upgrade.SQLITE_MAX_LENGTHthan the project builds with, so putting it inmake unittestwould cost a gigabyte per CI run. Everything else is intest_malformed_input().Verification
make unittest: 1482 passed, 0 failed (1447 before this branch), identical under ASan + UBSan. No warnings under-Wall -Wextra -Wshadow, on arm64 and on all three x86 backends cross-compiled with their ISAs. The universal dylib builds and loads, reporting1.2.0. Both exploit probes now return a clean error instead of writing out of bounds.🤖 Generated with Claude Code