Skip to content

feat: return column expression ranges with the node SQL read - #1426

Merged
kvlonge merged 7 commits into
mainfrom
feat/column-expression-ranges
Nov 12, 2026
Merged

kvlonge merged 7 commits into
mainfrom
feat/column-expression-ranges

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Why

SQLQuack gives each output column's defining expression as a byte range of the SQL it analysed. The UI needs them in the compiled SQL the node SQL read returns, which differs from the analysed text.

Changes

  • Lineage rows keep the range (analysis cache v5); shared analyses rebase it per member.
  • At snapshot build each range is mapped back through cleaning, typed-macro markers and cursor literals into compiled_sql. An edge beside removed or generated text (a __ref(...) call, a filled placeholder) is dropped, never moved.
  • Ranges go in the existing columns snapshot document, {node_key: {column: {"expression_range": [start, end]}}}, documented with the document kinds; out of node hashes, so a SQLQuack span change never marks a model changed. No schema change.
  • NodeSql returns expression_ranges. Readers ignore unknown per-column keys; a missing or malformed range means no highlight. The OpenAPI client is regenerated.
  • DuckShop: int_edge_expression_spans and int_edge_reference_qualified_spans, with unit tests. Expectations captured on the Snowflake edge target; they equal the hand derivation, with only literal spelling changed.

Verification

  • DuckDB E2E: macros, a variable, __ref and cursor bounds give byte-exact ranges; a reference-qualified column gets none; node JSON has no ranges. The installed CLI's person SQL read returns them.
  • Strict xfails until SQLQuack 0.8.0: Snowflake aliased items and qualified DuckDB columns.
  • Snapshot contract: a ranges-only republish keeps node hashes and records no change.
  • Rust tests: mapping, drops, rebasing.
  • DuckShop columns document: 397 B (184 B compressed); every column ranged: 208,377 B (21,851 B).
  • Diagnostics by code unchanged, base vs head, on DuckShop and 7 public projects.
  • Dense 3,000 models, same machine, 3 alternating runs, head/base medians: cold 1.048 wall / 1.014 CPU, warm 1.068 / 1.036, edit 0.901 / 0.939.
  • make check-ci passes.

Lineage rows keep the output expression's byte range in the cleaned
analysis SQL, the analysis cache stores it, shared analyses rebase it
onto each member's relation names, and query_expression_ranges maps it
back to the model's query SQL. Not yet exposed by the snapshot or UI.
Snapshot model nodes carry each output column's expression range as a
byte range of compiled_sql, mapped back through typed SQL macro markers
and cursor bounds; the node SQL read returns them. DuckDB e2e passes;
the Snowflake DuckShop edge model and DuckDB qualified columns are
strict xfails until SQLQuack's strict parsers record projection spans.
Ranges move out of the node definition into the per-snapshot columns
document, so they never change a node hash or record a node change;
the node SQL read ignores unknown per-column keys and drops malformed
or misplaced ranges.
Map a cleaned range edge only between source-adjacent characters, so a
range beside a removed reference call or inside generated text is
dropped instead of moved. Skip unnamed output columns. Split helpers
into single-function entry modules, make the tests table-driven, and
add a DuckShop edge model qualified by its reference call.
@kvlonge
kvlonge marked this pull request as ready for review November 12, 2026 12:54
@kvlonge
kvlonge enabled auto-merge November 12, 2026 12:56
@kvlonge kvlonge added the full-verify Run the full Verify matrix on this pull request label Nov 12, 2026 — with Amp for GitHub
@kvlonge
kvlonge added this pull request to the merge queue Nov 12, 2026
Merged via the queue into main with commit 5e309cc Nov 12, 2026
98 of 100 checks passed
@kvlonge
kvlonge deleted the feat/column-expression-ranges branch November 12, 2026 15:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-verify Run the full Verify matrix on this pull request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant