Support CREATE VECTOR INDEX - #2437
Open
moshap-firebolt wants to merge 3 commits into
Open
Conversation
BigQuery can create a vector index for approximate nearest-neighbor search over
an embedding column:
CREATE [OR REPLACE] VECTOR INDEX [IF NOT EXISTS] <name>
ON <table>(<column>)
OPTIONS(index_type = 'IVF', distance_type = 'COSINE', ...)
`VECTOR` is not a keyword, so `parse_create` previously failed with
`Expected: an object type after CREATE, found: VECTOR`.
### Changes
- Add `vector`, `or_replace` and `options` fields to `CreateIndex`. `VECTOR` is
a modifier on `CREATE INDEX` (like `EXTERNAL` on `CREATE TABLE`), so it reuses
the existing node rather than a new statement variant. `Display` renders
`CREATE [OR REPLACE ]VECTOR INDEX ...` plus a trailing `OPTIONS(...)`.
- Parse the form in `parse_create` via a small `parse_create_vector_index`
helper; the `OPTIONS(...)` clause reuses `parse_options` / `SqlOption`, so it
parses and renders the same way as `CREATE TABLE` / `CREATE VIEW` OPTIONS.
- Plain `CREATE INDEX` is unchanged (all three fields default to false/empty).
- New test `parse_bigquery_create_vector_index` in `tests/sqlparser_bigquery.rs`
verifies the round-trip and covers `OR REPLACE`, `IF NOT EXISTS`, multi-part
names and the OPTIONS-less form.
Docs: https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#create_vector_index_statement
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
LucaCappelletti94
suggested changes
Aug 18, 2026
LucaCappelletti94
left a comment
Contributor
There was a problem hiding this comment.
At this time, this PR edits would suggest that CREATE VECTOR INDEX is a BigQuery-specific syntax, but it is not, as it is supported also by:
- Oracle SQL
- Microsoft SQL
- Therefore also Generic Dialect
Therefore, move the tests to sqlparser common and ensure that all dialects that support the syntax successfully parse it. It may be acceptable to maintainers that dialects that do not support the syntax also parse it given the current preference for permissive parsing. I suggest you also review the possible syntax variations that these other dialects may have on this syntax, so as to cover more of its variants completely.
moshap-firebolt
force-pushed
the
bigquery-create-vector-index
branch
from
August 18, 2026 18:51
1e9b099 to
8c5ccd7
Compare
CREATE VECTOR INDEXCREATE VECTOR INDEX
moshap-firebolt
force-pushed
the
bigquery-create-vector-index
branch
2 times, most recently
from
August 18, 2026 19:31
de8c8a1 to
1e9b099
Compare
CREATE VECTOR INDEXCREATE VECTOR INDEX
Address review feedback that `CREATE VECTOR INDEX` is not BigQuery-specific (Oracle, SQL Server, MariaDB and TiDB also have it, hence Generic too). - Route the statement through `parse_create_index` instead of a separate BigQuery helper, so it inherits the standard index trailers (`USING`, `INCLUDE`, `WITH`, expression targets, index options) that cover the Oracle / SQL Server / TiDB variants, plus the BigQuery `OPTIONS(...)` clause. - Align `Display` order (OPTIONS after WITH) with parse order so `INCLUDE` + `OPTIONS` combinations round-trip. - Drop the BigQuery-specific doc-comment framing. - Move the test to `tests/sqlparser_common.rs` as `parse_create_vector_index`, running across all dialects and covering `OR REPLACE`, `IF NOT EXISTS`, schema-qualified names, `OPTIONS(...)`, the `INCLUDE` trailer, and an expression target. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Extend the shared `CREATE VECTOR INDEX` parsing to the trailers used by the other dialects, and add per-dialect tests: - `STORING(...)` covering-column clause (BigQuery); new `storing` field on `CreateIndex`, rendered after `INCLUDE`. - Accept a `WITH (...)` options clause on a vector index in every dialect (SQL Server `WITH (METRIC = ..., TYPE = ..., MAXDOP = ...)`), not only the dialects that enable it for a plain `CREATE INDEX`. Tests: - `tests/sqlparser_common.rs` — the generic core plus the shared trailers (`INCLUDE`, `STORING`, `WITH`, `USING`, expression targets) across all dialects. - `tests/sqlparser_bigquery.rs` — `OPTIONS(...)` with index_type / distance_type / JSON tuning keys, and `STORING(...)`. - `tests/sqlparser_mssql.rs` — bracket-quoted names with `WITH (...)`. - `tests/sqlparser_mysql.rs` — TiDB's distance-function target with `USING`. - `tests/sqlparser_oracle.rs` — the core, an expression target and an `INCLUDE` list (Oracle's `ORGANIZATION` / `DISTANCE` / `WITH TARGET ACCURACY` / `PARAMETERS` clauses are not yet parsed). Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
moshap-firebolt
force-pushed
the
bigquery-create-vector-index
branch
from
August 19, 2026 00:27
b6aa124 to
dd1de78
Compare
CREATE VECTOR INDEXCREATE VECTOR INDEX
Contributor
Author
|
@LucaCappelletti94 - done, added tests demonstrating different syntaxes. Note - didn't implement full blown Oracle's one. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CREATE VECTOR INDEXcreates an index for approximate nearest-neighbor search over an embedding column. It is not specific to one dialect — BigQuery, Oracle, SQL Server, MariaDB and TiDB all have it (so the Generic dialect should too).VECTORis not a keyword, soparse_createpreviously failed withExpected: an object type after CREATE, found: VECTOR.The statement shares a common core across dialects and then diverges into dialect-specific trailers. It is parsed permissively for every dialect.
Common core
VECTORis treated as a modifier onCREATE INDEX(likeUNIQUE), reusing the existingCreateIndexnode rather than a new statement variant. It is routed throughparse_create_index, so it inherits the standard index trailers —USING <method>,INCLUDE (...),WITH (...), expression targets and index options.Per-dialect forms
OPTIONS(index_type=…, distance_type=…, ivf_options='{…}'),STORING(col, …)WITH (METRIC=…, TYPE=…, MAXDOP=…), bracket-quoted names((VEC_COSINE_DISTANCE(col))), trailingUSING HNSWINCLUDE (...)ORGANIZATION …,DISTANCE …,WITH TARGET ACCURACY …,PARAMETERS(...)DISTANCE = … M = …optionsThe two follow-up rows need dedicated grammar with real keyword-collision hazards —
WITH TARGET ACCURACYoverloads the sameWITHkeyword as SQL Server'sWITH (...), and MariaDB's bare single-letterMcollides with identifiers — so they are left out of this change.Changes
vector,or_replace,optionsandstoringfields toCreateIndex.DisplayrendersCREATE [OR REPLACE ]VECTOR INDEX …and theSTORING(...)/OPTIONS(...)trailers.WITH (...)clause on a vector index in every dialect (for SQL Server), not only the dialects that enable it for a plainCREATE INDEX.CREATE INDEXis unchanged (the new fields default to false/empty).Tests
All are round-trip (
verified_stmt) tests.tests/sqlparser_common.rs— the generic core plus the shared trailers (INCLUDE,STORING,WITH,USING, expression targets), across all dialects.tests/sqlparser_bigquery.rs—OPTIONS(...)with real index_type / distance_type / JSON keys, andSTORING(...).tests/sqlparser_mssql.rs— bracket-quoted names withWITH (...).tests/sqlparser_mysql.rs— TiDB's distance-function target withUSING.tests/sqlparser_oracle.rs— the core, an expression target and anINCLUDElist.Docs: BigQuery · Oracle · SQL Server · MariaDB · TiDB