fix: add the two missing @reflection.cache decorators (get_foreign_keys, get_columns) - #74
Open
TangoEnSkai wants to merge 2 commits into
Open
fix: add the two missing @reflection.cache decorators (get_foreign_keys, get_columns)#74TangoEnSkai wants to merge 2 commits into
TangoEnSkai wants to merge 2 commits into
Conversation
TangoEnSkai
requested review from
deeksha-db,
gopalldb,
jackyhu-db,
jayantsing-db,
jprakash-db,
madhav-db,
samikshya-db,
shivam2680 and
vikrantpuppala
as code owners
August 23, 2026 03:57
get_foreign_keys() was the only _describe_table_extended()-backed
reflection method without @reflection.cache. SQLAlchemy threads an
info_cache dict through every Inspector call so that repeated reflection
of the same table costs one round-trip rather than one per call, but a
dialect method only participates when it is decorated.
Undecorated, every get_foreign_keys() call issued a fresh
DESCRIBE TABLE EXTENDED `catalog`.`schema`.`table`
against the warehouse, however many times the same table was reflected
through the same Inspector. For an application that reflects on each
request, that is a warehouse query per table per request.
get_pk_constraint() sits directly above it, uses the same helper, and is
decorated; get_table_names(), get_view_names(), has_table(),
get_schema_names() and get_table_comment() are decorated too. This was an
omission rather than a deliberate exclusion.
Note the cache key includes fn.__name__, so this does not make
get_pk_constraint() and get_foreign_keys() share one DESCRIBE within a
single pass — that duplication is separate and would need the helper
itself to be cached.
Add unit tests covering the cache hit, that the cache is keyed per table,
that a direct call passing no info_cache is unaffected, and a regression
guard on the already-correct get_pk_constraint().
Resolves databricks#72
Signed-off-by: TangoEnSkai <21152231+TangoEnSkai@users.noreply.github.com>
TangoEnSkai
force-pushed
the
fix/cache-get-foreign-keys
branch
from
August 23, 2026 03:58
a93872a to
2c0360b
Compare
get_columns() was the second dialect method missing @reflection.cache,
so it never participated in the cache SQLAlchemy threads through
Inspector.get_columns():
col_defs = self.dialect.get_columns(
conn, table_name, schema, info_cache=self.info_cache, **kw
)
The info_cache kwarg arrived, landed in **kwargs, and was discarded.
Measured against main, three calls sharing one info_cache produced three
GetColumns round-trips and left info_cache empty, where the decorated
get_pk_constraint() produced one.
The cost is not always a single statement: when cur.columns() returns an
empty list, get_columns() follows up with DESCRIBE TABLE EXTENDED to tell
a column-less table from a missing one, so an uncached call on such a
table is two round-trips, repeated every time.
SQLAlchemy's own SQLite, PostgreSQL and MySQL dialects all decorate
get_columns(), so this matches upstream practice rather than inventing
one. Caching is safe with respect to Inspector._instantiate_types(),
which mutates the returned column dicts in place but is guarded by
'if not isinstance(coltype, TypeEngine)' and is therefore a no-op on an
already-instantiated cached list.
get_indexes() remains undecorated deliberately: it returns the
EMPTY_INDEX constant without touching the server.
Resolves databricks#75
Signed-off-by: TangoEnSkai <21152231+TangoEnSkai@users.noreply.github.com>
9 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
SQLAlchemy threads an
info_cachedict through everyInspectorcall so thatreflecting the same table repeatedly costs one round-trip rather than one per
call. A dialect method only participates in that cache when it is decorated with
@reflection.cache.Two methods in
DatabricksDialectwere missing it —get_foreign_keys()(#72)and
get_columns()(#75). Both issue a warehouse round-trip on every singlecall. Their neighbours
get_pk_constraint(),has_table(),get_table_names(),get_view_names(),get_materialized_view_names(),get_temp_view_names(),get_schema_names()andget_table_comment()are all decorated, which makesthese look like omissions rather than deliberate exclusions.
Inspector.get_columns()shows the kwarg being handed over explicitly:It arrived, landed in
**kwargs, and was discarded.Measured against
mainwith the transport stubbed out, three calls sharing oneinfo_cache:info_cachekeysget_columns()[]— never populatedget_foreign_keys()[]— never populatedget_pk_constraint()For
get_columns()the cost is not always a single statement: whencur.columns()returns an empty list, it follows up withDESCRIBE TABLE EXTENDEDto tell a column-less table from a missing one (base.py:156), so anuncached call on such a table is two round-trips, repeated every time.
Callers that reuse one
Inspectoracross many lookups feel this directly —Alembic's autogenerate is the common case, as is any long-lived application that
reflects per request.
What
@reflection.cachetoget_foreign_keys()andget_columns().tests/test_local/test_reflection_cache.pycovering, for both methods: thecache hit, that a direct call passing no
info_cacheis unaffected (thedecorator falls straight through when the kwarg is absent, so callers invoking
the dialect method directly keep today's behaviour), per-table cache keying for
get_foreign_keys(), and a regression guard on the already-correctget_pk_constraint().get_indexes()is left undecorated on purpose: it returns theEMPTY_INDEXconstant without touching the server, so a cache would buy nothing.
Why
An uncached reflection method is a warehouse query per table per call. The change
is the same mechanism the neighbouring methods already use, and it matches
upstream practice — SQLAlchemy's own SQLite, PostgreSQL and MySQL dialects all
decorate
get_columns().It cannot change results:
@reflection.cacheis a no-op unless SQLAlchemy itselfsupplies
info_cache, and the calls it collapses are identical by construction.Caching is also safe with respect to
Inspector._instantiate_types(), whichmutates the returned column dicts in place — it is guarded by
if not isinstance(coltype, TypeEngine), so re-running it over analready-instantiated cached list is a no-op. This is the same situation the
upstream dialects are in.
Scope note
reflection.cachebuilds its key fromfn.__name__, so this does not makeget_pk_constraint()andget_foreign_keys()share a singleDESCRIBE TABLE EXTENDEDwithin one reflection pass — those two still issue the same statementonce each. Removing that remaining duplication would mean caching
_describe_table_extended()itself, which is a larger change and a differentdiscussion; happy to open a follow-up issue if you'd like it pursued.
Note on scope of this PR
This started as the
get_foreign_keys()fix for #72. While verifying it I foundget_columns()had the identical omission and filed #75. Since both are the sameone-line change to the same file and share one test module, keeping them apart
would have meant two PRs conflicting on all three files — so they are together
here. Happy to split them if you'd rather review them separately.
Completion Criteria
@reflection.cacheadded toget_foreign_keys()andget_columns()info_cachepathpytest tests/test_local(offline modules) — 300 passedblack --checkclean on both changed files (_parse.pyis flagged only by anewer local black than the pinned
^22.3.0, is untouched here, and isalready flagged on
main)# Unreleasedsectionclose #72
close #75