[mohammadnaqvi04]

Which issue does this PR close?

Part of #5492.

Rationale for this change

Many correlated subqueries fail to plan in DataFusion today, and some return wrong rows. #5492 tracks a general decorrelation rule to replace the current rules, and the plan proposed there starts with a tests-only PR, so every later PR has a fixed set of queries to check its fixes against.

What changes are included in this PR?

subquery_decorrelation.slt containing 57 queries.

Includes the following:

• The 12 shapes of the shape catalog linked in #5492. Three fail at planning with a schema error, six are left in the plan and fail at physical planning, one is a control that works today, and two are not valid SQL, because an aggregate over outer columns only belongs to the outer query. DuckDB rejects those two, and DataFusion should reject them while planning.
• All 44 queries from the repros in @jayzhan211's prototype branch, mostly one per issue, including the dbolympics shapes (#26704, #26710, #14554, #16137, #15558) and the wrong-result issues (#26014, #26032, #25808, #26503, #26692).
• The repro for #26572 from the list on #5492.

#26569 from that list isn't included, because it only affects joins built through the Rust API. DataFusion's own decorrelation sets the value key count (null_aware(value_keys=2) for a two-column NOT IN), so no SQL query shows it.

The cases @jayzhan211's fuzzer found will follow in a second PR.

Each query records what main returns today. The 42 that aren't right yet have DuckDB's answer in a comment above them, so a PR that fixes one replaces the recorded error or rows with that answer. Cases taken from an issue link to it. Today the 57 queries break down as follows.

Today	Queries
Fails to plan or execute	34
Returns wrong rows	8
Returns the right rows	11
Fails, as it should	4

What is the testing strategy for this PR?

The file passes on main, with output recorded by --complete, errors trimmed to stable patterns, and rowsort for queries without an ORDER BY. Answers were checked against DuckDB 2.0.0, and SQLite 3.51 agrees on 51 of the 57. @jayzhan211's harness agrees with the file on the 40 of 44 repro queries it can check.

Are there any user-facing changes?

No, this only adds tests.

AI Usage

AI was used during development of this PR, primarily to help review the test suite and verify test cases against the DB binaries.

[jayzhan211]

@mohammadnaqvi04 thanks for putting this catalog together! Please add the reference answer as a comment above each query whose recorded result isn't the right one: 11 queries record wrong rows and 34 record errors, and nothing in the file tells them apart from the 8 that are right. Without it, a later PR that changes a recorded result has nothing in the file to check its new rows against, which is what step 1 of the plan on #5492 asks for ("each with its reference answer"). The DuckDB answers you already have are enough, for example:

 # correlated_lossy_cast_below_count
+# Wrong today. Expected (DuckDB):
+# 2024-01-01 2
+# 2024-01-02 1
 query DI

The 11 recorded wrong results, with DuckDB 1.1.3's answer

Line	Issue	Recorded	DuckDB
162	#16134 (config off)	4 rows	error: more than one row
322	#26234	(empty)	1
337	#26014	01-01 1, 01-01 1, 01-02 1	01-01 2, 01-02 1
358	#26032	1	1, 2
377	#25808	0 1, 0 1, 0 2	0 1, 0 2
415	#26503	1 1 1, 2 1 0, 1 9 0	1 1 1, 2 1 NULL, 1 9 NULL
424	#26505	1 10, 2 NULL	1 NULL, 2 NULL
443	#26695	-1, 1, 2	1, 2
464	#26629	1	(empty)
552	#21198	1 2, 2 NULL	1 2
722	#26692	1	1, 1

This file also fails 5 queries on current main (I ran it on 3f4a37f, one commit behind main), so it needs a rebase and a re-record before merge. The #26503 (line 416) and #26629 (line 465) queries now error, after #26723 and #26640. The #26285 (line 530), #26622 (line 607) and #26572 (line 867) queries now return rows, and those rows match DuckDB.

What main returns now for the three that return rows

• line 530: 1, 3, 100000
• line 607: false, true
• line 867: 1 2, 2 2

[jayzhan211]

Optional cleanup: on current main this query also returns the right rows (1, 3, 100000) with the default hash join, so after the rebase this override and the RESET at line 543 can go, and the test runs the default join path.

-statement ok
-set datafusion.optimizer.prefer_hash_join = false;
-
 statement ok
 CREATE TABLE a(k INT) AS VALUES (1), (100000);
@@
-statement ok
-RESET datafusion.optimizer.prefer_hash_join;
-

[mohammadnaqvi04]

Done. The override and the RESET are gone.

[mohammadnaqvi04]

Thanks @jayzhan211! I added DuckDB's answer above the 42 queries that aren't correct yet and rebased and re-recorded on current main. From what I can tell, six results changed, one more than you saw. #26503, #26629 and #21198 now error, #21198 after #26874, which merged after your review. #26285, #26622 and #26572 now return the right rows, though #26285 and #26622 are still open. I've also updated the counts in the description.