Skip to content

feat(pwmj): support RightSemi/RightAnti existence joins - #24457

Open
SubhamSinghal wants to merge 1 commit into
apache:mainfrom
SubhamSinghal:pwmj-right-existence-joins
Open

feat(pwmj): support RightSemi/RightAnti existence joins#24457
SubhamSinghal wants to merge 1 commit into
apache:mainfrom
SubhamSinghal:pwmj-right-existence-joins

Conversation

@SubhamSinghal

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Part of #17427 (the PiecewiseMergeJoin epic). This is the second half of its existence-join item: #23870 added LeftSemi/LeftAnti, this adds RightSemi/RightAnti. Only the Mark joins remain unsupported.

Rationale for this change

A range-predicate right semi/anti join — ... RIGHT SEMI JOIN r ON l.v > r.v, or any query that decorrelates to one — currently falls back to NestedLoopJoinExec.

What changes are included in this PR?

New RightExistencePWMJStream (right_existence_join.rs), a 3-state machine: await the buffered reduction, then per streamed batch evaluate the key, compare it against the reduced key with a vectorized cmp kernel, and filter. Plus:

  • exec.rs — dispatch; build_buffered_extreme, which folds every buffered partition on its own SpawnedTask; BufferedExtreme as the shared state; relaxed ordering, distribution and partitioning declarations for these two join types; removal of the dead sort_options flip and the unimplemented!() placeholder that the swap design had left behind.
  • physical_planner.rs — the gate now excludes only the Mark joins.
  • utils.rsis_supported_existence_join widened; new is_supported_right_existence_join (which deliberately excludes RightMark, so a Mark join cannot inherit the relaxed input requirements if that gate is ever loosened); no visited-indices bitmap allocated for these join types.
  • extreme_key made pub(super) so both existence streams share one implementation.

Are these changes tested?

Yes

Benchmark

Microbenchmark, 20K × 20K rows, single inequality, enable_piecewise_merge_join on vs off

Case PWMJ NestedLoopJoin Speedup
RightSemi, all match ~0.023 ms ~77 ms ~3,300×
RightAnti, all match ~0.024 ms ~77 ms ~3,200×
RightSemi, no match ~0.023 ms ~78 ms ~3,300×
RightAnti, no match ~0.024 ms ~78 ms ~3,300×
RightSemi, half match ~0.036 ms ~78 ms ~2,200×
RightAnti, half match ~0.036 ms ~77 ms ~2,100×

Are there any user-facing changes?

No

@SubhamSinghal
SubhamSinghal marked this pull request as ready for review August 18, 2026 12:57
@github-actions github-actions Bot added core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) physical-plan Changes to the physical-plan crate labels Aug 18, 2026
@SubhamSinghal

Copy link
Copy Markdown
Contributor Author

benchmark PR: #24458

@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 87.98220% with 81 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.26%. Comparing base (bb038a6) to head (5116d05).
⚠️ Report is 4 commits behind head on main.

Files with missing lines Patch % Lines
...joins/piecewise_merge_join/right_existence_join.rs 88.08% 20 Missing and 36 partials ⚠️
...ysical-plan/src/joins/piecewise_merge_join/exec.rs 86.97% 6 Missing and 19 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #24457      +/-   ##
==========================================
+ Coverage   81.23%   81.26%   +0.03%     
==========================================
  Files        1112     1113       +1     
  Lines      390635   392183    +1548     
  Branches   390635   392183    +1548     
==========================================
+ Hits       317350   318727    +1377     
- Misses      54650    54695      +45     
- Partials    18635    18761     +126     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@SubhamSinghal

Copy link
Copy Markdown
Contributor Author

@comphead @kumarUjjawal PR for right semi/anti join. Please help in reviewing this PR. Thanks.

@comphead

Copy link
Copy Markdown
Contributor

Thanks @SubhamSinghal I'll check this today

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate physical-plan Changes to the physical-plan crate sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants