Repository navigation
fix(planner): observe source freshness per source and fail plans on missing source tables - #908
Merged
Merged
Conversation
… source tables Adapter freshness metadata is still fetched in one batched lookup, but each table now gets its own outcome (observed, missing, or unavailable). A missing table, a view, or a null last-modified time no longer makes every source in the batch unknown. Planning fails with S405 when a selected resource reads an external source table that does not exist, naming the source, its table, the reading models, and the declaration file and line. Managed, loader, and expression sources are exempt. The Freshness metadata progress line reports source and unknown counts, a failed freshness query explains itself, and the plan carries a warning and JSON details for each unknown-freshness reason. sqb freshness reports each source on its own. Refs: CHI-464
S405 now treats the relation listing as a first pass only. It reuses the listing already gathered for source columns, then probes each source the listing did not find with the exact relation name the model SQL reads. Only the adapter's object-not-found error counts as missing; other probe errors fail planning as themselves. Each first-class adapter classifies its own not-found errors. Adapter freshness metadata no longer reports "missing": a table the metadata does not describe is unavailable, since only the probe can decide existence. Databricks again falls back to Unity Catalog when Delta history has no timestamp. Freshness warnings group by reason, every unknown source has a detail entry, and the S405 docs cover the probe and pre_hook tables. Refs: CHI-464
The relation read probe now returns readable, missing, or missing-or- unreadable. Snowflake's 2003/2043 errors cannot tell a missing table from one the role cannot read, so S405 says "does not exist or is not readable", names the role when the connection reports it, and suggests granting SELECT as well as creating the table. Databricks schema and catalog not-found errors and DuckDB missing attached catalogs now count as missing. Probes for sources the listing did not find run through the shared bounded inspection pool. The SQL test build parity fixture plans against a warehouse that holds its source table. Refs: CHI-464
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
During
sqb plan, adapter source freshness was fetched as one batch. If any one source table had no metadata, every source in the batch became "freshness unknown", and the only output wasFreshness metadata FAILwhile the plan still succeeded. On a real ~260-source Snowflake project, three missing tables hid freshness for every source on every plan (CHI-464).Changes
sqb plan --jsonaddsunknown_source_details;sqb freshnessreports per source.LAST_ALTEREDwhen Delta history has no timestamp.Verification
sqb freshness.make test,make check-ci; one independent review plus a follow-up.