wip: drop expression ranges at rewritten edges and meet layout rules

Map a cleaned range edge only between source-adjacent characters, so a
range beside a removed reference call or inside generated text is
dropped instead of moved. Skip unnamed output columns. Split helpers
into single-function entry modules, make the tests table-driven, and
add a DuckShop edge model qualified by its reference call.

Author: kvlonge

benchmarks/duckshop/README.md

@@ -124,8 +124,8 @@ project and does not change it:
   `edge_running_peaks`, that a pre-hook of `int_edge_basket_running_peaks` creates
   (`models/intermediate/edge_cases/_sqlbuild/_hooks/sql/`) and SQLBuild does not declare.
 
-It has 165 models (13 staging, 138 intermediate, 14 marts), 5 Python functions, a SQL table
-function, a hook-created JavaScript function, 190 unit tests and 4 scenarios, in nine groups:
+It has 166 models (13 staging, 139 intermediate, 14 marts), 5 Python functions, a SQL table
+function, a hook-created JavaScript function, 191 unit tests and 4 scenarios, in nine groups:
 
 | Group | Covers |
 |-------|--------|
@@ -134,7 +134,7 @@ function, a hook-created JavaScript function, 190 unit tests and 4 scenarios, in
 | VARIANT comparisons | `ARRAY_MAX`, `ARRAY_MIN` and `ARRAY_REVERSE` over arrays mixing numbers, strings, booleans, JSON nulls and integers past 2^53; `MAX` across kinds and non-ASCII strings; `=` and `EQUAL_NULL` for `1`, `"1"` and `1.0`, reordered keys, JSON null and DATE or TIMESTAMP values held in a `VARIANT` against their text; `MIN_BY` and `MAX_BY` of `VARIANT` values; `PARTITION BY`, `SELECT DISTINCT` and `ARRAY_AGG(DISTINCT)` over `VARIANT`; JSON nulls in `ARRAY_REMOVE`, `ARRAY_DISTINCT` and `ARRAYS_OVERLAP`; decimals with trailing zeros in `ARRAY_CONTAINS` and `ARRAY_POSITION`; `OBJECT_CONSTRUCT(* EXCLUDE ("note"))` |
 | Dates and times | `TRY_TO_TIMESTAMP_NTZ` and its siblings over free text; epoch milliseconds as numbers and text; daylight-saving gaps and overlaps; fractional offsets; nanosecond truncation and epoch parts after 2242; pre-1970 text with sub-microsecond digits; date-part normalisation; month ends and leap days; week numbering across a year; sentinel timestamps; `TRY_TO_DATE` with numeric and month-name formats over unpadded fields, two-digit years, ISO `T` and `Z`, trailing letters, a long s, mixed separators and runs of spaces; text read as `TIMESTAMP_TZ` and `TIMESTAMP_LTZ` with `T`, `Z`, `+00:00`, `+0000`, `+09:00` and `-05:30`, back as the UTC instant, date, year and day start, and compared with text, `TIMESTAMP_NTZ` and `DATE`; `FIRST_VALUE` of a zoned value as a `DATE`; `DATEADD` and `TIMEADD` on `TIME(3)` and the type they return; `DATEDIFF` across units; `TIME(9)`, `FF3` and bare `FF` formats truncating; `DATE`, `TO_TIME`, `TRY_TO_TIMESTAMP` and `TO_TIMESTAMP_LTZ` of clock text, and `CURRENT_DATE` and `SYSDATE` read only for facts that hold on any day |
 | Ordering, numbers and text | NULL ordering under `DESC`; ties in ordered aggregates, `MAX_BY`, `MODE` and `ANY_VALUE`; default window frames; `UNION ALL BY NAME`; `ROUND` half-way values, decimal division, `MOD` and 38-digit amounts; integers past 2^53 against `FLOAT`; regular expressions and backslash escapes; `LIKE ... ESCAPE` and `ILIKE ANY`; `SPLIT_PART` and `SUBSTR` before the start; `NOT IN` with a NULL; names in many scripts; `JOIN ... USING` between `NUMBER(18,0)`, `NUMBER(38,0)` and text keys, with the types of the merged and qualified keys; NULL arithmetic and the types of NULL, decimal and exponent literals, `COUNT`, `ROW_NUMBER`, `%` and `NULLIF`; `MEDIAN` at scales 0 to 15; `DIV0` by literal zeros; `MODE`, `MEDIAN` and `COUNT_IF` over no rows; aggregates inside window functions and `QUALIFY`; `ROLLUP` totals; `TRUNC` of `FLOAT`; infinity words in other scripts; `CASE` without `ELSE` and `DECODE` over text; regular expressions with a `?` after a lazy quantifier, classes opening with `]`, empty matches and `.` against a carriage return; case-insensitive matches outside ASCII; `ILIKE` with a long s; `JAROWINKLER_SIMILARITY` of accented names in both cases; Base64 text with line feeds; `STARTSWITH`, `ENDSWITH`, `SUBSTRING`, `POSITION` with `IN` and with a comma, `CHARINDEX`, `CHR`, `REGEXP_SUBSTR_ALL`, `::STRING` and `CAST ... AS STRING`; `HEX_ENCODE`, `HEX_DECODE_STRING`, `TRY_HEX_DECODE_STRING` and `TRY_BASE64_DECODE_BINARY`; a comment attached to a text literal; `FLOOR` with and without a scale, `BITAND`, `ZEROIFNULL`, `::BIGINT` and `::DOUBLE`; `IS NOT DISTINCT FROM` and `EQUAL_NULL`; a comma join after an explicit join; `HASH_AGG` read in two row orders |
-| Lineage and analysis | clauses whose columns decide rows without being output (`WHERE`, `QUALIFY`, `GROUP BY` by position and `ALL`, `ORDER BY` under `LIMIT`, `EXISTS`, a `FLATTEN` input, `ROLLUP` by position, `EXCEPT` under `LIMIT`); `RIGHT JOIN ... USING`, `NATURAL JOIN`, a `USING` key under a later outer join, and unmatched `ASOF JOIN` rows; `CONNECT BY` beside a column named `level`; a recursive CTE without `RECURSIVE` whose columns shift per iteration; `LAST_DAY` with a date part; positional columns and the names Snowflake gives unaliased expressions; `TEXT(5)` and widened `COUNT` contracts; `LAG` and `LEAD` with one, two and three arguments; a comma-joined `LATERAL` whose nested derived table reads the left item; untyped and typed `FILTER` lambdas over `ARRAY_AGG` in a grouped query; column expression ranges in compiled SQL that SQL and Python macros, a variable and cursor bounds rewrote |
+| Lineage and analysis | clauses whose columns decide rows without being output (`WHERE`, `QUALIFY`, `GROUP BY` by position and `ALL`, `ORDER BY` under `LIMIT`, `EXISTS`, a `FLATTEN` input, `ROLLUP` by position, `EXCEPT` under `LIMIT`); `RIGHT JOIN ... USING`, `NATURAL JOIN`, a `USING` key under a later outer join, and unmatched `ASOF JOIN` rows; `CONNECT BY` beside a column named `level`; a recursive CTE without `RECURSIVE` whose columns shift per iteration; `LAST_DAY` with a date part; positional columns and the names Snowflake gives unaliased expressions; `TEXT(5)` and widened `COUNT` contracts; `LAG` and `LEAD` with one, two and three arguments; a comma-joined `LATERAL` whose nested derived table reads the left item; untyped and typed `FILTER` lambdas over `ARRAY_AGG` in a grouped query; column expression ranges in compiled SQL that SQL and Python macros, a variable and cursor bounds rewrote, and columns qualified by their reference call, whose ranges are dropped |
 | Python functions | volatile tokens and numbers from a column, a constant and no argument, and `DIV0` of a volatile call beside look-alike names, checked by assertions that hold for any values; a handler answering NULL input without `STRICT`; Python `None` as JSON null, and datetimes, Decimals and integers past 2^53 handed to `VARIANT` and `FLOAT` results |
 | Scenarios and lifecycle | typed values across model boundaries; `VARIANT` expected output; a star under a lower-case contract and a quoted camelCase export; a snapshot keyed and compared on quoted camelCase columns; a cursor-window incremental model over far-apart fixture loads; a change log merged across two loads, checked against a full rebuild |
 | Translation refusals | one model per construct that translation tools refuse or guard, most putting a construct refused only under strict checking before one refused by default, so a model shows both: `FLOAT` values through sums, averages, running totals, arrays, objects, `HASH`, `TO_CHAR` and `::NUMBER(38, 10)`; `VARIANT` dates, decimals and XML; `ARRAY_AGG` without `WITHIN GROUP`, sparse arrays and `ARRAY(NUMBER)` casts; `REDUCE`, `FILTER` and `TRANSFORM` lambdas; patterns and `REGEXP_REPLACE` replacements built from columns, and lazy patterns; `TRY_TO_DATE` with a format from a column; zoned literals, `CONVERT_TIMEZONE` from `CURRENT_TIMESTAMP`, `SEQ8`, `GENERATOR` with an expression `ROWCOUNT`; `PIVOT`, `ANY_VALUE`, `IGNORE NULLS` around `IFF`, unaliased columns read by Snowflake's name, `UNION` branches whose column is always NULL; `TO_NUMBER`, `TO_DECIMAL` and `TRY_BASE64_DECODE_BINARY`; an open (uncontracted) source `edge__supplier_notes`, the table function, read for one basket and after `LATERAL` for every basket, and the hook-created function. `rpt_edge_case_model_keys` and `rpt_edge_case_array_model_keys` read every model in the group, so a refusal there also refuses a downstream model |

benchmarks/duckshop/project/models/intermediate/edge_cases/int_edge_reference_qualified_spans.sql

@@ -0,0 +1,17 @@
+MODEL (
+  description """
+    One row per edge-case price window, each column qualified by the source reference call itself,
+    so every output column's expression starts at text the compiler rewrites; the case for column
+    expression ranges that must be dropped rather than moved onto the relation name.
+  """,
+  tags [edge_case],
+  columns (
+    window_id (type VARCHAR, nullable false, audits [not_null, unique]),
+    sku (type VARCHAR),
+  ),
+);
+
+SELECT
+  __source("edge__price_windows").window_id,
+  __source("edge__price_windows").sku
+FROM __source("edge__price_windows")

benchmarks/duckshop/project/sqlbuild_project.toml

@@ -339,11 +339,14 @@ that shape is the case under test.
 
 [[rules.rule_ignores]]
 rules = ["SQLCLEAN034"]
-paths = ["models/intermediate/edge_cases/int_edge_expression_spans.sql"]
+paths = [
+  "models/intermediate/edge_cases/int_edge_expression_spans.sql",
+  "models/intermediate/edge_cases/int_edge_reference_qualified_spans.sql",
+]
 reason = """
-The edge case keeps its macro calls, variable and literals in the terminal select on purpose:
-column expression ranges point at the terminal select's items, so that is where the compiled SQL
-must differ from the authored text.
+These edge cases keep their macro calls, variable, literals and reference calls in the terminal
+select on purpose: column expression ranges point at the terminal select's items, so that is where
+the compiled SQL must differ from the authored text.
 """
 
 [[rules.rule_ignores]]

benchmarks/duckshop/project/tests/unit/intermediate/edge_cases/test_int_edge_reference_qualified_spans.sql

@@ -0,0 +1,26 @@
+TEST (
+  name "int_edge_reference_qualified_spans__keeps_every_window",
+);
+
+WITH __source__edge__price_windows AS (
+  SELECT
+    v.window_id,
+    v.sku,
+    v.price_minor::NUMBER(38, 0) AS price_minor,
+    v.valid_from::TIMESTAMP_NTZ(6) AS valid_from,
+    v.valid_to::TIMESTAMP_NTZ(6) AS valid_to,
+    v._loaded_at::TIMESTAMP_NTZ AS _loaded_at
+  FROM VALUES
+    ('PW-01', 'SKU-1', 1234, '2026-01-01 00:00:00', '9999-12-31 00:00:00', '2026-03-02 09:00:00'),
+    ('PW-02', NULL, NULL, '2026-01-01 00:00:00', '9999-12-31 00:00:00', '2026-03-02 18:30:00') AS v(window_id, sku, price_minor, valid_from, valid_to, _loaded_at)
+),
+
+__expected__int_edge_reference_qualified_spans AS (
+  SELECT
+    'PW-01' AS window_id,
+    'SKU-1' AS sku
+  UNION ALL
+  SELECT
+    'PW-02' AS window_id,
+    CAST(NULL AS VARCHAR) AS sku
+)

clients/typescript/src/schema.d.ts

@@ -2009,10 +2009,7 @@ export interface components {
         /** @description SQL inspection never returns Python source, execution artifacts or unrelated node metadata. */
         NodeSql: {
             compiled_sql?: string | null;
-            /**
-             * @description Each output column's defining expression as a `[start, end)` UTF-8 byte range of
-             *     `compiled_sql`; absent for columns the analysis could not place.
-             */
+            /** @description Each placed column's defining expression as a `[start, end)` byte range of `compiled_sql`. */
             expression_ranges?: {
                 [key: string]: number[];
             };

crates/sqlbuild-analysis/src/analysis_route/_helpers/analysis/native_sharing.rs

@@ -213,13 +213,12 @@ fn mapped(
     Ok(analysis)
 }
 
-/// Move each expression range from the leader's cleaned SQL to the member's, which differ only in
-/// the relation names they spell; a range that cannot be moved exactly is dropped.
+/// Each expression range moved onto `member`'s relation names, or dropped where it cannot move.
 pub(crate) fn rebase_ranges(analysis: &mut NativeAnalysis, leader: &str, member: &str) {
     if leader == member {
         return;
     }
-    let renamed: Option<Vec<(usize, usize, usize, usize)>> = renamed_runs(leader, member);
+    let renamed: Option<Vec<(usize, usize, usize)>> = renamed_runs(leader, member);
     for row in &mut analysis.lineage {
         row.expression_range = match (&renamed, row.expression_range) {
             (Some(runs), Some((start, end))) => rebased(runs, start).zip(rebased(runs, end)),
@@ -228,13 +227,12 @@ pub(crate) fn rebase_ranges(analysis: &mut NativeAnalysis, leader: &str, member:
     }
 }
 
-/// `(leader start, leader end, member start, member end)` of each identifier run where two SQL
-/// texts differ, when they differ only in whole identifier runs.
-fn renamed_runs(leader: &str, member: &str) -> Option<Vec<(usize, usize, usize, usize)>> {
+/// `(leader start, leader end, member end)` of each differing identifier run, if only those differ.
+fn renamed_runs(leader: &str, member: &str) -> Option<Vec<(usize, usize, usize)>> {
     let identifier = |c: u8| c.is_ascii_alphanumeric() || c == b'_' || c == b'$';
     let (left, right): (&[u8], &[u8]) = (leader.as_bytes(), member.as_bytes());
     let (mut i, mut j): (usize, usize) = (0, 0);
-    let mut runs: Vec<(usize, usize, usize, usize)> = Vec::new();
+    let mut runs: Vec<(usize, usize, usize)> = Vec::new();
     while i < left.len() || j < right.len() {
         if i < left.len() && j < right.len() && left[i] == right[j] {
             i += 1;
@@ -255,25 +253,25 @@ fn renamed_runs(leader: &str, member: &str) -> Option<Vec<(usize, usize, usize,
         if left_end == i && right_end == j {
             return None;
         }
-        runs.push((i - back, left_end, j - back, right_end));
+        runs.push((i - back, left_end, right_end));
         (i, j) = (left_end, right_end);
     }
     Some(runs)
 }
 
 /// The member offset of leader `offset`, or `None` strictly inside a renamed run.
-fn rebased(runs: &[(usize, usize, usize, usize)], offset: usize) -> Option<usize> {
-    let mut shift: isize = 0;
-    for &(left_start, left_end, _, right_end) in runs {
+fn rebased(runs: &[(usize, usize, usize)], offset: usize) -> Option<usize> {
+    let mut moved: usize = offset;
+    for &(left_start, left_end, right_end) in runs {
         if offset <= left_start {
             break;
         }
         if offset < left_end {
             return None;
         }
-        shift = right_end as isize - left_end as isize;
+        moved = offset - left_end + right_end;
     }
-    usize::try_from(offset as isize + shift).ok()
+    Some(moved)
 }
 
 /// Whether `leader`'s `template` succeeded, found nothing and spells none of its relation names.

crates/sqlbuild-analysis/src/analysis_route/main/query_expression_ranges.rs

@@ -2,12 +2,10 @@
 
 use crate::analysis_route::main::analysis_route::analysis_route;
 use crate::analysis_route::models::AnalysisRoute;
-use crate::semantic_validation::_helpers::normalization::normalize_native_sql_mapped;
+use crate::semantic_validation::main::normalize_native_filled::normalize_native_filled;
 use crate::sql_macros::models::TextSplice;
 
-/// Each byte range of `analysis_sql`'s cleaned native analysis SQL as the byte range of the query
-/// SQL that `passes` rewrote, in order, into `analysis_sql`; `None` where there was none, where an
-/// end falls inside rewritten text, or where the dialect has no native analysis.
+/// Each cleaned-SQL byte range as a range of the SQL `passes` rewrote into `analysis_sql`.
 #[must_use]
 pub fn query_expression_ranges(
     dialect: &str,
@@ -19,7 +17,7 @@ pub fn query_expression_ranges(
         return vec![None; ranges.len()];
     };
     let Ok((cleaned, offsets)) =
-        normalize_native_sql_mapped(analysis_sql, dialect, module.stub_prefix(), placeholders)
+        normalize_native_filled(analysis_sql, dialect, module.stub_prefix(), placeholders)
     else {
         return vec![None; ranges.len()];
     };
@@ -35,6 +33,9 @@ pub fn query_expression_ranges(
     characters[cleaned.len()] = Some(cleaned.chars().count());
     let query_byte = |cleaned_byte: usize| -> Option<usize> {
         let character: usize = (*characters.get(cleaned_byte)?)?;
+        if !copied_boundary(&offsets, character) {
+            return None;
+        }
         let byte: usize = *analysis_bytes.get(*offsets.get(character)?)?;
         passes
             .iter()
@@ -51,8 +52,18 @@ pub fn query_expression_ranges(
         .collect()
 }
 
-/// The source byte of output byte `byte` of a pass whose `splices` are in text order; a byte at
-/// a splice's edge maps to the same edge of its source, and one inside rewritten text has none.
+/// Whether cleaned boundary `character` lies between source-adjacent characters, so maps exactly.
+fn copied_boundary(offsets: &[usize], character: usize) -> bool {
+    let Some(previous) = character.checked_sub(1) else {
+        return offsets.first() == Some(&0);
+    };
+    offsets
+        .get(previous)
+        .zip(offsets.get(character))
+        .is_some_and(|(before, after)| before + 1 == *after)
+}
+
+/// The source byte of output `byte`; splice edges map to source edges, inner bytes to `None`.
 fn spliced_back(byte: usize, splices: &[TextSplice]) -> Option<usize> {
     let (mut source_end, mut output_end): (usize, usize) = (0, 0);
     for splice in splices {