Repository navigation
feat: support quoted mixed-case columns and read SQL with adapter lexical rules - #1270
Merged
Merged
Conversation
…every scanner Snowflake strings escape quotes with a backslash, but the scanners that find declarations, Python and SQL macro calls and @@ variables read SQL under generic rules, so an escaped quote hid later markers or reported an unclosed string. Local scenario replay also compiled with DuckDB's rules, so the reference scanner failed on Snowflake SQL. Every scanner on the compile path now reads quoted text and comments with the project's lexical rules, and local replay compiles with the project adapter's rules.
…self Local replay now compiles the edge-case area, so the runbook replays the core scenarios in place and the README drops the escaped-quote limitation.
stg_edge_ticket_notes now normalises its SKU with a SQL macro after the backslash-escaped quote, so the edge area covers a compile marker that a scanner reading generic quote rules would hide.
int_cart_funnel sorts its distinct added SKUs, int_delivery_slot_utilisation takes the weekday average as an exact decimal sum over a count and the earliest of equally busy slots, and rpt_delivery_exceptions_monthly breaks ties for the top exception code alphabetically instead of using MODE. The core-copy compile guard records the new checksum and the README notes the change for benchmark comparisons.
SET search_path and SET schema change how unqualified names resolve, but were classified as data statements, so the run's relation metadata cache kept answers from before them. They now invalidate every cached lookup. DuckDB's unqualified existence check also looks only in the schemas on the session's search path, as name resolution does.
A first build listed each schema during planning and again when the run's relation cache first needed it. The build now keeps one relation metadata cache from planning to the end of the run, and the planner's Snowflake schema listings go into it. Statements the build issues after a listing still invalidate it, and a planner listing older than a minute is read again.
A source or model column created with a quoted mixed-case name, such as "listingId", could not be used: a MODEL header could not declare it, unit-test and scenario fixtures rejected it, and once the table existed Snowflake planning lower-cased it and failed to bind. Header column keys may now be double-quoted identifiers, fixtures and contract checks match a quoted name to its unquoted spelling, Snowflake keeps stored mixed-case column names quoted, adapters render quoted names as they are, and quoted aliases a final SELECT * reads keep their exact case. DuckShop's partner-listings edge models now build, test and replay without the exclusion or the rule ignores.
The new wheel writes DuckDB lambdas as LAMBDA x : body, so the local scenario replay of a lambda now passes and its strict xfail goes.
sqb query no longer prints a Project discovery lifecycle for reading two configuration files. Connecting and running the query show a transient stderr line on a terminal only once the step has taken 300 ms, named for the step; piped and JSON output hold only the result. The user-code import hook first searches only the search path entries under a project root, so installed and standard-library imports skip the full path search they used to repeat. The Snowflake adapter sets SNOWFLAKE_DISABLE_PLATFORM_DETECTION=true unless the user set it, which removes the connector's cloud metadata probes, its AWS identity call and the botocore warnings it printed. The adapter page documents how to turn detection back on.
A rule exception naming a missing file already fails compilation, but a rule ignore did not, so an ignore left behind by a renamed or deleted model stayed in the project silently. A rule ignore path without glob characters must now exist; globs and folders are unaffected.
…lexical rules The SQL macro lexer took only the adapter's backslash-escape quotes, so an E'...' string with an escaped quote on DuckDB or Postgres read as unclosed and the model failed to compile. With an adapter's rules it now ends quoted text and comments exactly as the shared dialect scanner does, including E-string and raw-string prefixes and dialect line comments.
The macro call scanner read @py arguments with the adapter's SQL rules, so on Snowflake an invalid argument such as 10 // 3 read as a line comment and reported an unclosed parenthesis instead of the clear not-a-literal error. Inside @py arguments it now uses Python's string rules with only the generic -- line comment, so dialect comment markers such as // and # read as code, as they did before the lexical-rules change.
…name key Snowflake listings now keep a mixed-case column quoted, as in "listingId", but the runtime contract, type enforcement, incremental schema, load, migration and raw-query diff checks compared names with lower(). A quoted alias declared unquoted in the header then failed the runtime contract and skipped type enforcement, although compile accepted it. These checks now key names as the compiler does: case-folded and without identifier quotes.
…he planner's A listing read through the cache's refresh path, including one made while planning, had no age limit, so a build's run could trust it for as long as no statement invalidated it. Every kept schema listing now expires after a minute, like the planner listings the run reuses.
DuckDB resolves an unqualified name through a search path schema written in any case, but relation existence compared the search path's spelling exactly, so after SET search_path = 'STAGING' a table DuckDB could read was reported missing.
…ectory The import hook cached whether a relative search path entry such as "" lay under a project root, resolved against the working directory of its first lookup, so after a chdir it kept answering for the old directory. Relative entries are now keyed by the current working directory.
…iables With the adapter's lexical rules, @@ interpolation read a doubled backtick as an escape, so audit SQL holding `@@name`` reported an unclosed string instead of the unknown variable it names. Backtick text is read as before, ending at the next backtick; quotes and comments still follow the adapter's rules.
…ctory is gone The import hook read the working directory outside its error handling, so after the directory was deleted every import through a relative search path entry raised FileNotFoundError. The entry is skipped again, as before the hook keyed entries by directory. A SQL macro lexer test now names the raw-string prefix rules it uses.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The DuckShop edge-case area and a bounded review found several defects: escaped quotes and dialect comments hid or broke compile-time scanners, quoted mixed-case column names could not be declared, tested or planned, DuckDB ignored
SET search_pathin existence checks, builds listed Snowflake schemas twice, andsqb queryshowed misleading progress.Changes
@@interpolation) reads strings and comments with the adapter's lexical rules, including E-strings;@pyarguments use Python string rules."listingId"work in MODEL headers, fixtures, contracts, Snowflake planning and builds, and runtime contract, type-enforcement, incremental and diff checks match them as the compiler does.SET search_path, case-insensitively.rule_ignorespath without glob characters that does not exist now fails compilation, asrule_exceptionsalready did.sqb queryshows transient progress only for slow steps; the Snowflake adapter turns off connector platform detection unless the user setSNOWFLAKE_DISABLE_PLATFORM_DETECTION; the import hook skips installed packages.polyglot-sql-chio==0.13.10, makes three DuckShop core models deterministic, and moves repository links to the sqlstack-io organization.Verification
make test, andmake check-ci.