fix(dataset): derive virtual dataset schema from SQL Lab query - #43313
Draft
sadpandajoe wants to merge 1 commit into
Draft
fix(dataset): derive virtual dataset schema from SQL Lab query#43313sadpandajoe wants to merge 1 commit into
sadpandajoe wants to merge 1 commit into
Conversation
Derive an unambiguous catalog and schema from the Jinja-aware parsed query while preserving submitted defaults for ambiguous, mutating, metadata, or unparseable SQL. Include template parameters and conservative guards for quoted identifiers and incomplete parser results. Addresses #16791.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #43313 +/- ##
==========================================
- Coverage 66.67% 66.09% -0.58%
==========================================
Files 2876 2876
Lines 164007 164056 +49
Branches 37834 37848 +14
==========================================
- Hits 109347 108436 -911
- Misses 52514 53464 +950
- Partials 2146 2156 +10
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SUMMARY
When a virtual dataset is created from SQL Lab's "Save & Explore" action, the saved
dataset's
schemais always set to whatever schema is selected in the SQL Labdropdown — even when the query itself explicitly references a different schema. For
example, selecting
information_schemain the dropdown and then runningselect * from public."Vehicle Sales"creates a dataset withschema=information_schema,not
schema=public. This has been reported since 2021 (#16791) and affects datasetmetadata correctness, RLS resolution, and permission checks that read the dataset's
stored
schema.This PR derives the schema from the query itself when it is unambiguous, and falls
back to today's dropdown-based behavior in every other case:
process_jinja_sql,the same one already used by the access-control check on this path) — no new SQL
parsing is introduced.
references exactly one
(catalog, schema)location, every referenced table isexplicitly schema-qualified (zero unqualified references), the statement doesn't
mutate data, doesn't change the default schema mid-script, doesn't contain a
metadata (
SHOW ...) statement, and doesn't reference any quoted catalog/schemaidentifier (conservative, since quoting can carry case-sensitive identity on some
engines).
behavior unchanged. This is an intentional, documented limitation, not an oversight.
template_paramsor a Jinja rendering error) safely falls back to the submittedschema rather than blocking dataset creation.
Out of scope for this PR (see
RCA.mdfor details): the broader multi-schema datasetmodel proposed in community discussion, and permission/RLS enforcement based on the
derived schema — both would be separate, larger changes.
RCA.mdin this PR documents the full investigation: root cause, why it wasn'tcaught, the exact safety conditions and why each is needed, a verified matrix of
parser behavior across ~20 query shapes, and known limitations of this fix.
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
Backend-only change (dataset creation logic); no UI change. Behavior before/after for
the reported repro:
information_schema+ queryselect * from public."Vehicle Sales"→ saved datasetschema=information_schema(wrong)schema=public(matches the query)TESTING INSTRUCTIONS
pytest tests/unit_tests/commands/dataset/test_create.py— covers the primaryrepro case plus every fallback condition (zero-schema, multi-schema, unqualified
reference, mutating statement, metadata statement, quoted identifier, unparseable/
template-error SQL, Jinja partition-macro tables, template-parameter-driven schema
selection, and positive catalog+schema derivation).
query that explicitly references a different schema (e.g.
select * from <other_schema>.<table>), click Explore → name the dataset → Save &Explore. Confirm the saved dataset's schema matches the query, not the dropdown.
ADDITIONAL INFORMATION