Repository navigation
Conversation
A `Partitioned` hash join merges its partitions' build maps into one routed `hash_lookup` filter. The routed lookup accepted only hash tables, so a join whose partitions built an `ArrayMap` (dense integer keys) fell back to a `CASE` over one filter per partition. The routed lookup now probes an `ArrayMap` partition with the key values of the rows routed to it, because an `ArrayMap` is keyed by value, not hash. Every non-empty partition can now merge. The `CASE` fallback remains only for canceled partitions. The routed lookup fuzz test now builds an `ArrayMap` for about half of the partitions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ed-lookup-array-map # Conflicts: # datafusion/physical-plan/src/joins/hash_join/partitioned_hash_eval.rs # datafusion/physical-plan/src/joins/hash_join/shared_bounds.rs
Array map partitions now join the merged lookup, and a dense partition usually has no pruning bitmap, so it contributes its whole bounds. The merged pruning bitmap returned `None` when every partition contributed only bounds, so a Range-partitioned join stopped pruning the probe containers that fall in the gaps between partitions. `union` already returns `None` when no bucket can be excluded, so the early return goes. This also adds tests that array map partitions reach the merged filter under hash and range routing. The fuzz test keeps hash tables under `force_hash_collisions`, where a hash table matches every key and an array map would disagree with the reference. The remaining changes are doc wording for array maps, `take_arrays`, and a simpler test helper. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ray map An array map probes by key value, so when no partition holds a hash map the routed lookup computed per-row hashes that nothing read. The lookup now hashes the probe keys only when some partition is a hash map. The counting sort by partition runs either way, and the sorted hashes are gathered from the sorted rows only when hashes exist. A new test builds every partition as an array map and checks exact matches, including a null probe row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ed-lookup-skip-hashes # Conflicts: # datafusion/physical-plan/src/joins/hash_join/partitioned_hash_eval.rs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
A
Partitionedhash join with dense integer keys builds anArrayMapin every partition. Its dynamic filter then computes a hash for every probe row, but anArrayMaplooks keys up by value, so nothing reads those hashes. Skipping them makes the filter cheaper on every probe batch.What changes are included in this PR?
HashTableLookupExpr::evaluate_routedhashes the probe keys only when at least one partition is a hash table. The counting sort by partition runs either way.What is the testing strategy for this PR?
ArrayMapand checks exact matches, including a null probe row.force_hash_collisions.Are there any user-facing changes?
No.
🤖 Generated with Claude Code