Skip to content

feat(scenario): map scenario column types with SQLQuack's type parsers - #1364

Closed
kvlonge wants to merge 7 commits into
mainfrom
feat/sqlquack-type-and-local-sql
Closed

kvlonge wants to merge 7 commits into
mainfrom
feat/sqlquack-type-and-local-sql

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Why

Scenario capture still asked Polyglot (parse_data_type) to read warehouse column types. This replaces it with SQLQuack's type parsers, one of the last two runtime uses of the Polyglot package.

Changes

  • One native entry point, parse_warehouse_type(spelling, dialect). Postgres uses sqlquack-syntax-postgres; Snowflake and BigQuery use the cast target of CAST(NULL AS <type>). Other dialects return None.
  • columns.py: _local_type_with_polyglot and its four helpers are replaced by a walk of the SQLQuack tree. DuckDB and MotherDuck spellings pass through unchanged.
  • Databricks ARRAY/MAP/STRUCT map recursively to DuckDB T[], MAP(K, V) and STRUCT(a T). An unmappable element is refused with X605 instead of becoming JSON. SQL Server's date, time, uuid, bit and money types get their own pre-mapper. The fallback keeps integer and 4-byte float widths and contains no Polyglot.
  • The SQLQuack rev moves 4d0c6bc5 → 98eefae6, and sqlquack-syntax-bigquery is added. Fingerprints are unchanged because the tokenizer and scanner sources are identical. The next SQLQuack commit refuses FROM __ref(...) placeholders on Snowflake, so this rev stops before it.
  • import_polyglot.py is deleted. The local-SQL path and the dependency drop follow separately.

Verification

  • Before/after over 156 dialect and type pairs: the capture tests, SQL Server's information_schema names and float spellings. 31 rows change, each named in the latest table comment: spacing, TEXT→VARCHAR, BigQuery INTEGER→BIGINT (INT64), Snowflake VARIANT→JSON, DuckDB spellings passing through, and SQL Server types that were invalid in DuckDB.
  • Scenario unit and integration tests: 339 passed. Scenario E2E (16 files): 120 passed.
  • cargo clippy -D warnings, cargo fmt, ruff, ty, fensu, and the native stub, boundary and layering checks pass.

Local scenario capture no longer asks Polyglot to read warehouse column
types. Postgres types go through sqlquack-syntax-postgres's type parser,
and Snowflake and BigQuery types through their SQLQuack cast-target
parse, behind one native function (parse_warehouse_type). DuckDB and
MotherDuck spellings pass through unchanged. Databricks nested types map
recursively to DuckDB's LIST, MAP and STRUCT, and a nested spelling that
cannot be mapped is refused with a named message instead of falling back
to JSON. Other types keep their integer width or fall back to VARCHAR.

kvlonge commented Nov 8, 2026

Copy link
Copy Markdown
Contributor Author

Before/after type mapping for every dialect and type pair in the scenario capture tests (Polyglot on main vs SQLQuack here). 13 of 116 rows change.

Dialect Warehouse type Before (Polyglot) After (SQLQuack) Change
bigquery ARRAY<INT64> BIGINT[] BIGINT[]
bigquery ARRAY<STRUCT<a INT64, b STRING>> STRUCT(a BIGINT, b TEXT)[] STRUCT(a BIGINT, b TEXT)[]
bigquery BIGNUMERIC DECIMAL(38, 5) DECIMAL(38, 5)
bigquery BIGNUMERIC(38,9) DECIMAL(38, 9) DECIMAL(38, 9)
bigquery BOOL BOOLEAN BOOLEAN
bigquery BYTES VARCHAR VARCHAR
bigquery BYTES(10) VARCHAR VARCHAR
bigquery DATE DATE DATE
bigquery DATETIME TIMESTAMP TIMESTAMP
bigquery FLOAT64 DOUBLE DOUBLE
bigquery GEOGRAPHY VARCHAR VARCHAR
bigquery INT64 BIGINT BIGINT
bigquery INTEGER INT BIGINT BigQuery INTEGER is INT64
bigquery JSON JSON JSON
bigquery NUMERIC DECIMAL DECIMAL
bigquery NUMERIC(10,2) DECIMAL(10, 2) DECIMAL(10, 2)
bigquery RANGE<DATE> VARCHAR VARCHAR
bigquery RANGE<DATETIME> VARCHAR VARCHAR
bigquery RANGE<TIMESTAMP> VARCHAR VARCHAR
bigquery STRING TEXT TEXT
bigquery STRUCT<a INT64> STRUCT(a BIGINT) STRUCT(a BIGINT)
bigquery TIMESTAMP TIMESTAMPTZ TIMESTAMPTZ
databricks ARRAY<INT> INT[] INT[]
databricks DECIMAL(10,2) DECIMAL(10, 2) DECIMAL(10,2) spacing only
databricks GEOGRAPHY VARCHAR VARCHAR
databricks GEOMETRY VARCHAR VARCHAR
databricks INTERVAL YEAR INTERVAL YEAR INTERVAL YEAR
databricks MAP<STRING, ARRAY<INT>> MAP(TEXT, INT[]) MAP(VARCHAR, INT[]) TEXT and VARCHAR are the same DuckDB type
databricks MAP<STRING, INT> MAP(TEXT, INT) MAP(VARCHAR, INT) TEXT and VARCHAR are the same DuckDB type
databricks STRING TEXT VARCHAR TEXT and VARCHAR are the same DuckDB type
databricks STRUCT<a: INT> STRUCT(a INT) STRUCT(a INT)
databricks TIMESTAMP TIMESTAMPTZ TIMESTAMPTZ
databricks TIMESTAMP_NTZ TIMESTAMP TIMESTAMP
databricks TINYINT TINYINT TINYINT
databricks VARCHAR(20) TEXT(20) VARCHAR TEXT(n) and VARCHAR are the same DuckDB type; DuckDB ignores the length
databricks VARIANT VARIANT VARIANT
databricks VOID VARCHAR VARCHAR
duckdb BLOB VARCHAR VARCHAR
duckdb ENUM('a', 'b') ENUM('a', 'b') ENUM('a', 'b')
duckdb HUGEINT INT128 INT128
duckdb LIST<INTEGER> INT[] INT[]
duckdb STRUCT(a INTEGER) STRUCT(a INT) STRUCT(a INTEGER) DuckDB spelling passes through; INTEGER is INT
duckdb TIME WITH TIME ZONE TIMETZ TIME WITH TIME ZONE DuckDB spelling passes through; same type as TIMETZ
duckdb TIMESTAMP_NS TIMESTAMP_NS TIMESTAMP_NS
duckdb UHUGEINT UINT128 UINT128
duckdb UINTEGER UINTEGER UINTEGER
duckdb UNION(num INTEGER, str VARCHAR) UNION(num INT, str TEXT) UNION(num INTEGER, str VARCHAR) DuckDB spelling passes through; same types
generic DATE DATE DATE
generic DECIMAL(10,2) DECIMAL(10, 2) DECIMAL(10,2) spacing only
generic DECIMAL(38,0) DECIMAL(38, 0) DECIMAL(38,0) spacing only
generic INTEGER INT INT
generic NUMBER VARCHAR VARCHAR
generic NUMBER(10,2) VARCHAR VARCHAR
generic STANDARD VARCHAR VARCHAR
generic TIMESTAMP TIMESTAMP TIMESTAMP
generic VARCHAR TEXT VARCHAR TEXT and VARCHAR are the same DuckDB type
generic WIDGET VARCHAR VARCHAR
postgres bigint BIGINT BIGINT
postgres bigserial BIGINT BIGINT
postgres bytea VARCHAR VARCHAR
postgres cidr VARCHAR VARCHAR
postgres double precision DOUBLE DOUBLE
postgres geography VARCHAR VARCHAR
postgres geometry VARCHAR VARCHAR
postgres inet INET INET
postgres int4range VARCHAR VARCHAR
postgres integer INT INT
postgres integer[] INT[] INT[]
postgres interval INTERVAL INTERVAL
postgres jsonb JSON JSON
postgres macaddr VARCHAR VARCHAR
postgres macaddr8 VARCHAR VARCHAR
postgres money VARCHAR VARCHAR
postgres name VARCHAR VARCHAR
postgres numeric DECIMAL DECIMAL
postgres numeric(10,2) DECIMAL(10, 2) DECIMAL(10, 2)
postgres point VARCHAR VARCHAR
postgres real REAL REAL
postgres regclass VARCHAR VARCHAR
postgres serial INT INT
postgres smallint SMALLINT SMALLINT
postgres smallserial SMALLINT SMALLINT
postgres text TEXT TEXT
postgres timestamp with time zone TIMESTAMPTZ TIMESTAMPTZ
postgres tstzrange VARCHAR VARCHAR
postgres uuid UUID UUID
postgres varchar(255) TEXT(255) TEXT(255)
postgres xml VARCHAR VARCHAR
snowflake ARRAY JSON JSON
snowflake BIGINT BIGINT BIGINT
snowflake BINARY VARCHAR VARCHAR
snowflake BOOLEAN BOOLEAN BOOLEAN
snowflake DATE DATE DATE
snowflake DECIMAL(12,4) DECIMAL(12, 4) DECIMAL(12, 4)
snowflake FLOAT DOUBLE DOUBLE
snowflake FLOAT4 REAL REAL
snowflake FLOAT8 DOUBLE DOUBLE
snowflake GEOGRAPHY VARCHAR VARCHAR
snowflake GEOMETRY VARCHAR VARCHAR
snowflake INT INT INT
snowflake NUMBER DECIMAL(38, 0) DECIMAL(38, 0)
snowflake NUMBER(10,2) DECIMAL(10, 2) DECIMAL(10, 2)
snowflake NUMERIC(9,0) DECIMAL(9, 0) DECIMAL(9, 0)
snowflake OBJECT JSON JSON
snowflake REAL REAL REAL
snowflake STRING TEXT TEXT
snowflake TIME TIME TIME
snowflake TIMESTAMP_LTZ TIMESTAMPTZ TIMESTAMPTZ
snowflake TIMESTAMP_LTZ(9) TIMESTAMPTZ TIMESTAMPTZ
snowflake TIMESTAMP_NTZ TIMESTAMP TIMESTAMP
snowflake TIMESTAMP_NTZ(9) TIMESTAMP(9) TIMESTAMP(9)
snowflake TIMESTAMP_TZ TIMESTAMPTZ TIMESTAMPTZ
snowflake VARCHAR TEXT TEXT
snowflake VARCHAR(255) TEXT(255) TEXT(255)
snowflake VARIANT VARIANT JSON held as JSON, as SQLQuack's conversion reads it
snowflake VECTOR(FLOAT, 3) JSON JSON

SQL Server has no SQLQuack type parser, so its information_schema names
reached the generic fallback, which held datetime2, smalldatetime,
datetimeoffset and uniqueidentifier as VARCHAR. A SQL Server pre-mapper
now maps them to TIMESTAMP, TIMESTAMPTZ and UUID, bit to BOOLEAN, money
and smallmoney to exact decimals, and SQL Server's 8-byte float to
DOUBLE.

The fallback keeps 4-byte floats (FLOAT, FLOAT4, REAL) as FLOAT instead
of widening them to DOUBLE. A panic in the warehouse type parser now
surfaces as the ValueError capture already falls back on.

kvlonge commented Nov 8, 2026

Copy link
Copy Markdown
Contributor Author

Before/after type mapping at 7655ef3: every dialect and type pair in the capture tests, plus the SQL Server information_schema names and the float spellings from review. Polyglot on base 97ab62c is compared with this branch. 31 of 156 rows change; the T-SQL datetime2, smalldatetime, datetimeoffset and uniqueidentifier rows keep their timestamp and uuid types, so they don't appear as changes.

Dialect Warehouse type Before (Polyglot) After (SQLQuack) Change
bigquery ARRAY<INT64> BIGINT[] BIGINT[]
bigquery ARRAY<STRUCT<a INT64, b STRING>> STRUCT(a BIGINT, b TEXT)[] STRUCT(a BIGINT, b TEXT)[]
bigquery BIGNUMERIC DECIMAL(38, 5) DECIMAL(38, 5)
bigquery BIGNUMERIC(38,9) DECIMAL(38, 9) DECIMAL(38, 9)
bigquery BOOL BOOLEAN BOOLEAN
bigquery BYTES VARCHAR VARCHAR
bigquery BYTES(10) VARCHAR VARCHAR
bigquery DATE DATE DATE
bigquery DATETIME TIMESTAMP TIMESTAMP
bigquery FLOAT64 DOUBLE DOUBLE
bigquery GEOGRAPHY VARCHAR VARCHAR
bigquery INT64 BIGINT BIGINT
bigquery INTEGER INT BIGINT BigQuery INTEGER is INT64
bigquery JSON JSON JSON
bigquery NUMERIC DECIMAL DECIMAL
bigquery NUMERIC(10,2) DECIMAL(10, 2) DECIMAL(10, 2)
bigquery RANGE<DATE> VARCHAR VARCHAR
bigquery RANGE<DATETIME> VARCHAR VARCHAR
bigquery RANGE<TIMESTAMP> VARCHAR VARCHAR
bigquery STRING TEXT TEXT
bigquery STRUCT<a INT64> STRUCT(a BIGINT) STRUCT(a BIGINT)
bigquery TIMESTAMP TIMESTAMPTZ TIMESTAMPTZ
databricks ARRAY<FLOAT> FLOAT[] FLOAT[]
databricks ARRAY<INT> INT[] INT[]
databricks DECIMAL(10,2) DECIMAL(10, 2) DECIMAL(10,2) spacing only
databricks DOUBLE DOUBLE DOUBLE
databricks FLOAT FLOAT FLOAT
databricks GEOGRAPHY VARCHAR VARCHAR
databricks GEOMETRY VARCHAR VARCHAR
databricks INTERVAL YEAR INTERVAL YEAR INTERVAL YEAR
databricks MAP<STRING, ARRAY<INT>> MAP(TEXT, INT[]) MAP(VARCHAR, INT[]) TEXT and VARCHAR are the same DuckDB type
databricks MAP<STRING, INT> MAP(TEXT, INT) MAP(VARCHAR, INT) TEXT and VARCHAR are the same DuckDB type
databricks STRING TEXT VARCHAR TEXT and VARCHAR are the same DuckDB type
databricks STRUCT<a: FLOAT> STRUCT(a FLOAT) STRUCT(a FLOAT)
databricks STRUCT<a: INT> STRUCT(a INT) STRUCT(a INT)
databricks TIMESTAMP TIMESTAMPTZ TIMESTAMPTZ
databricks TIMESTAMP_NTZ TIMESTAMP TIMESTAMP
databricks TINYINT TINYINT TINYINT
databricks VARCHAR(20) TEXT(20) VARCHAR TEXT and VARCHAR are the same DuckDB type; DuckDB ignores the length
databricks VARIANT VARIANT VARIANT
databricks VOID VARCHAR VARCHAR
duckdb BLOB VARCHAR VARCHAR
duckdb ENUM('a', 'b') ENUM('a', 'b') ENUM('a', 'b')
duckdb HUGEINT INT128 INT128
duckdb LIST<INTEGER> INT[] INT[]
duckdb STRUCT(a INTEGER) STRUCT(a INT) STRUCT(a INTEGER) DuckDB spelling passes through unchanged; same type
duckdb TIME WITH TIME ZONE TIMETZ TIME WITH TIME ZONE DuckDB spelling passes through unchanged; same type
duckdb TIMESTAMP_NS TIMESTAMP_NS TIMESTAMP_NS
duckdb UHUGEINT UINT128 UINT128
duckdb UINTEGER UINTEGER UINTEGER
duckdb UNION(num INTEGER, str VARCHAR) UNION(num INT, str TEXT) UNION(num INTEGER, str VARCHAR) DuckDB spelling passes through unchanged; same type
generic DATE DATE DATE
generic DECIMAL(10,2) DECIMAL(10, 2) DECIMAL(10,2) spacing only
generic DECIMAL(38,0) DECIMAL(38, 0) DECIMAL(38,0) spacing only
generic DOUBLE DOUBLE DOUBLE
generic FLOAT FLOAT FLOAT
generic INTEGER INT INT
generic NUMBER VARCHAR VARCHAR
generic NUMBER(10,2) VARCHAR VARCHAR
generic REAL VARCHAR VARCHAR
generic STANDARD VARCHAR VARCHAR
generic TIMESTAMP TIMESTAMP TIMESTAMP
generic VARCHAR TEXT VARCHAR TEXT and VARCHAR are the same DuckDB type
generic WIDGET VARCHAR VARCHAR
postgres bigint BIGINT BIGINT
postgres bigserial BIGINT BIGINT
postgres bytea VARCHAR VARCHAR
postgres cidr VARCHAR VARCHAR
postgres double precision DOUBLE DOUBLE
postgres geography VARCHAR VARCHAR
postgres geometry VARCHAR VARCHAR
postgres inet INET INET
postgres int4range VARCHAR VARCHAR
postgres integer INT INT
postgres integer[] INT[] INT[]
postgres interval INTERVAL INTERVAL
postgres jsonb JSON JSON
postgres macaddr VARCHAR VARCHAR
postgres macaddr8 VARCHAR VARCHAR
postgres money VARCHAR VARCHAR
postgres name VARCHAR VARCHAR
postgres numeric DECIMAL DECIMAL
postgres numeric(10,2) DECIMAL(10, 2) DECIMAL(10, 2)
postgres point VARCHAR VARCHAR
postgres real REAL REAL
postgres regclass VARCHAR VARCHAR
postgres serial INT INT
postgres smallint SMALLINT SMALLINT
postgres smallserial SMALLINT SMALLINT
postgres text TEXT TEXT
postgres timestamp with time zone TIMESTAMPTZ TIMESTAMPTZ
postgres tstzrange VARCHAR VARCHAR
postgres uuid UUID UUID
postgres varchar(255) TEXT(255) TEXT(255)
postgres xml VARCHAR VARCHAR
snowflake ARRAY JSON JSON
snowflake BIGINT BIGINT BIGINT
snowflake BINARY VARCHAR VARCHAR
snowflake BOOLEAN BOOLEAN BOOLEAN
snowflake DATE DATE DATE
snowflake DECIMAL(12,4) DECIMAL(12, 4) DECIMAL(12, 4)
snowflake FLOAT DOUBLE DOUBLE
snowflake FLOAT4 REAL REAL
snowflake FLOAT8 DOUBLE DOUBLE
snowflake GEOGRAPHY VARCHAR VARCHAR
snowflake GEOMETRY VARCHAR VARCHAR
snowflake INT INT INT
snowflake NUMBER DECIMAL(38, 0) DECIMAL(38, 0)
snowflake NUMBER(10,2) DECIMAL(10, 2) DECIMAL(10, 2)
snowflake NUMERIC(9,0) DECIMAL(9, 0) DECIMAL(9, 0)
snowflake OBJECT JSON JSON
snowflake REAL REAL REAL
snowflake STRING TEXT TEXT
snowflake TIME TIME TIME
snowflake TIMESTAMP_LTZ TIMESTAMPTZ TIMESTAMPTZ
snowflake TIMESTAMP_LTZ(9) TIMESTAMPTZ TIMESTAMPTZ
snowflake TIMESTAMP_NTZ TIMESTAMP TIMESTAMP
snowflake TIMESTAMP_NTZ(9) TIMESTAMP(9) TIMESTAMP(9)
snowflake TIMESTAMP_TZ TIMESTAMPTZ TIMESTAMPTZ
snowflake VARCHAR TEXT TEXT
snowflake VARCHAR(255) TEXT(255) TEXT(255)
snowflake VARIANT VARIANT JSON held as JSON, as SQLQuack's conversion reads it
snowflake VECTOR(FLOAT, 3) JSON JSON
tsql bigint BIGINT BIGINT
tsql binary(16) VARCHAR VARCHAR
tsql bit BIT BOOLEAN SQL Server bit is a boolean
tsql char(10) TEXT VARCHAR TEXT and VARCHAR are the same DuckDB type
tsql date DATE DATE
tsql datetime TIMESTAMP TIMESTAMP
tsql datetime2 TIMESTAMP TIMESTAMP
tsql datetime2(7) TIMESTAMP TIMESTAMP
tsql datetimeoffset TIMESTAMPTZ TIMESTAMPTZ
tsql datetimeoffset(7) TIMESTAMPTZ TIMESTAMPTZ
tsql decimal(18,2) DECIMAL(18, 2) DECIMAL(18,2) spacing only
tsql float FLOAT DOUBLE SQL Server float is 8 bytes
tsql int INT INT
tsql money VARCHAR DECIMAL(19,4) exact decimal(19,4), as SQL Server stores it
tsql nchar(10) TEXT(10) VARCHAR TEXT and VARCHAR are the same DuckDB type; DuckDB ignores the length
tsql ntext TEXT VARCHAR TEXT and VARCHAR are the same DuckDB type
tsql numeric(10,0) DECIMAL(10, 0) DECIMAL(10,0) spacing only
tsql nvarchar(50) TEXT(50) VARCHAR TEXT and VARCHAR are the same DuckDB type; DuckDB ignores the length
tsql nvarchar(max) TEXT(MAX) VARCHAR TEXT and VARCHAR are the same DuckDB type; DuckDB ignores the length
tsql real REAL FLOAT REAL and FLOAT are the same 4-byte DuckDB type
tsql smalldatetime TIMESTAMP TIMESTAMP
tsql smallint SMALLINT SMALLINT
tsql smallmoney SMALLMONEY DECIMAL(10,4) exact decimal(10,4), as SQL Server stores it
tsql sql_variant SQL_VARIANT VARCHAR was an invalid DuckDB type
tsql text TEXT VARCHAR TEXT and VARCHAR are the same DuckDB type
tsql time TIME TIME
tsql time(7) TIME(7) TIME DuckDB TIME takes no precision; was invalid
tsql tinyint TINYINT TINYINT
tsql uniqueidentifier UUID UUID
tsql varbinary(max) VARBINARY(MAX) VARCHAR binary is held as VARCHAR, as for every dialect; was invalid
tsql varchar(50) TEXT(50) VARCHAR TEXT and VARCHAR are the same DuckDB type; DuckDB ignores the length
tsql varchar(max) VARCHAR(MAX) VARCHAR was an invalid DuckDB type
tsql xml VARCHAR VARCHAR

@kvlonge
kvlonge enabled auto-merge (squash) November 8, 2026 04:57
SQLQuack's next commit refuses a bare call in a Snowflake FROM clause,
which rejects the __ref and __source placeholders that compiled model
SQL keeps, so custom-rule syntax trees fail on Snowflake models. This
rev keeps the BigQuery parser the warehouse type parsers need.

kvlonge commented Nov 8, 2026

Copy link
Copy Markdown
Contributor Author

Closing: every commit on this branch is already contained in #1374 (the combined parser train), which will land this change. The branch is kept; reopen if #1374 is dropped.

@kvlonge kvlonge closed this Nov 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant