[DISCUSSION] How should we test datafusion-spark against Spark?

@alamb

At the DataFusion community call on 2026-11-04 there was renewed interest in datafusion-spark. This issue lists the challenges and some options for making it reliably Spark-compatible.

Challenges

Development has stalled. In my opinion, most DataFusion reviewers aren't interested in Spark functionality, and the people who are, in Comet and Sail, rarely review or contribute because of the challenges below. Since November 2025, 79% of spark PRs came from outside the Comet and Sail regulars. Comet regulars reviewed 89 of the 347 PRs, 56 of them by one reviewer, and Sail contributors reviewed 5. Comet runs about 34 Spark expressions on this crate and about 96 on its own code, including at least 21 functions this crate also implements. It would like to upstream more, such as CAST (Spark-compatible CAST operation #11201), but that isn't practical without real tests that ensure Spark compatibility.
We don't test against Spark. Nothing checks the expected results in test_files/spark/*.slt against Spark, and reviewers can't verify them without running Spark. Audits against a live Spark (docs: add an agent skill for auditing datafusion-spark expressions #23893) keep finding divergences, including tests that assert the wrong value ([Bug] atan2 does not normalize -0.0 the way Spark does #26333, [Bug] trunc and date_trunc handle unsupported, NULL and non-literal formats differently from Spark #26338). I raised this risk when the crate moved here ([DISCUSSION] Add separate crate to cover spark builtin functions #5600).
Spark versions differ. Behavior changes between Spark releases (ANSI mode became the default in 4.0, for example), but the crate has no notion of a Spark version (Introduce version-specific behavior in Spark expressions #21698) and an .slt query can only have one expected result (Support Spark-version-specific expectations in datafusion-spark sqllogictest files #23887). Comet supports five Spark versions (3.4 to 4.2), and even Comet runs only one of them before merge, the rest nightly or on demand.
Two kinds of users, one test path. Comet builds physical plans directly, so it needs invoke_with_args(). Projects that use DataFusion's planner and optimizer can use simplify() rewrites instead. SLT always goes through the planner and optimizer, so it tests the rewrite rather than the physical path, and functions with no physical implementation pass every test (Some datafusion-spark expressions are missing the physical implementation #21959).

Options

Run Spark in this repo's CI. feat: cross-engine Spark-dialect SLT tests #21508 was an earlier attempt. This catches divergences before merge, but a DataFusion PR could then fail a Spark test its author can't debug, and it adds a JVM and a Spark version matrix to CI whose usage we are already trying to reduce (Reduce GitHub Actions usage (DataFusion is currently the top ASF consumer of GHA minutes) #25148).
Generate expected results with Spark and check them in per Spark version. A script runs the .slt queries on each supported Spark version and records the results, and CI runs only DataFusion against them. Nobody needs Spark to debug a failure, since it is an ordinary SLT diff against a value Spark produced. Whoever adds a test has to run the script (or ask a bot to), the queries must be valid Spark SQL (DataFusion now has a Spark dialect, feat: Add Spark SQL parser dialect config #22529), and passing on more than one Spark version needs Introduce version-specific behavior in Spark expressions #21698.
Move the crate to a new apache/datafusion-spark repository under the DataFusion PMC, like Comet, Ballista and datafusion-iceberg. Its CI can run real Spark across versions, and its contributors and reviewers would be people who can debug Spark. The pieces of core that only this crate uses would move with it: enable_ansi_mode, datafusion.spark.*, datafusion-cli --spark and the Spark handling in the sqllogictest runner.

These can be combined, and option 2 works in either repository. Each option also needs a way to test the physical path, for example by running the Spark .slt files a second time with simplify() disabled.

Impact on downstream projects

Comet and Sail depend on datafusion-spark (55.2.0), which today ships with each DataFusion release. Options 1 and 2 keep that. Option 3 would mean:

A separate release, with its own vote, after each DataFusion release before Comet and Sail can upgrade. In [DISCUSSION] Add separate crate to cover spark builtin functions #5600 Sail said waiting on out-of-tree crates slowed its upgrades, but that this goes away if Comet and Sail maintainers help keep the crate in sync.
DataFusion API changes absorbed when the new repo upgrades, as Comet does today, rather than in the same PR.
Public APIs for code shared with datafusion-functions (Deduplicate Spark function code with native/default datafusion function code #17964).

Which options, or combination, should we pursue? Are there others?

cc @alamb @comphead @mbutrovich @shehabgamin @linhr @Jefffrey @coderfender @xanderbailey

Move the crate to a new apache/datafusion-spark repository under the DataFusion PMC, like Comet, Ballista and datafusion-iceberg. Its CI can run real Spark across versions, and its contributors and reviewers would be people who can debug Spark. The pieces of core that only this crate uses would move with it: enable_ansi_mode, datafusion.spark.*, datafusion-cli --spark and the Spark handling in the sqllogictest runner.

I am in favor of this option personally as I think it is cleaner and fits better with the DataFusion modular design

Also, it would then set a clear precedent that we shouldn't add datafusion-json / datafusion-variant into the main repo (we can potentially bring them into apache governance, but they would live in separate repos)

Totally for option 3, I was chasing this option for a while. datafusion-spark can have its own release cadence and release maintainers.

I am also +1 for this option, even though it creates more work for Comet release process.

Totally in favor of option 3. Volunteering to help out on that effort

+1 for moving the crate to a new apache/datafusion-spark. I think it is best for the people interested in Spark functionality and doesn't affect the people in DataFusion community who aren't interested in it.

+1 to option 3

Moving into a new crate will make pre-release testing for DataFusion a lot harder. So that's one thing to keep in mind

+1 for option 3 from my point of view

option 3 does seem the natural choice. do we have enough datafusion maintainers to lead the charge on solving the other issues though? such as setting up spark CI, having capacity for reviewing the contributions to the new repo, determining how to support spark versions, etc.

Moving into a new crate will make pre-release testing for DataFusion a lot harder. So that's one thing to keep in mind

Just to clarify, I don't oppose option 3. My main concern is that both Comet and Sail test DataFusion ahead of its releases, helping catch regressions before they ship. If datafusion-spark moves to a separate repository, we should consider how that would affect the existing pre-release testing workflow.

If datafusion-spark moves to a separate repository, we should consider how that would affect the existing pre-release testing workflow.

The datafusion-spark repo would probably need to create a branch/fork that pins to the pre-release datafusion branch and then comet/sail pin to that as well

It is an extra step, but something that is probably reasonably automatable (esp in the age of agents)

yes, having a long-lived branch in the apache repo that pins to latest datafusion core makes sense. We do the same in Comet and Ballista already.

@andygrove

+1 for option 3, thanks for documenting this discussion, @andygrove!

ok, looks like we have concencus for option 3. On timing, I propose that we create it once the 56.0.0 vote passes. I'd be happy to do that work.

By the way, I opened PR #27165 to add a manual PySpark runner that can run the current slt files in datafusion-spark and report on differences between datafusion-spark and Spark.

Sounds like a great plan to me.

I created https://github.com/apache/datafusion-spark

Now that we are close to releasing 56, seems like a good time to start on this. We'll need to decide how to move the code over, looking into that next.

I created https://github.com/apache/datafusion-spark

Now that we are close to releasing 56, seems like a good time to start on this. We'll need to decide how to move the code over, looking into that next.

we would need to

make an announcement
remove datafusion-spark from release notes
IMO crates.io would remain intact as long as we keep the same name
good question is: how we gonna version the crate, the latest release crate would be 56.0.0, and what are our versioning strategy plans be later on?
new ASF slack channel?

Claude suggests running these commands to populate the new repo.

git clone --single-branch --no-tags https://github.com/apache/datafusion.git df-spark-import
cd df-spark-import
git filter-repo \
  --path datafusion/spark/ \
  --path datafusion/sqllogictest/test_files/spark/ \
  --path-rename datafusion/spark/:crates/spark/ \
  --path-rename datafusion/sqllogictest/test_files/spark/:crates/sqllogictest/test_files/ \
  --commit-callback '
commit.message = commit.message.rstrip(b"\n") + b"\n\nOriginal-commit: apache/datafusion@" + commit.original_id + b"\n"'
git push git@github.com:apache/datafusion-spark.git main

IMO crates.io would remain intact as long as we keep the same name

good question is: how we gonna version the crate, the latest release crate would be 56.0.0, and what are our versioning strategy plans be later on?

Yes, I propose that we keep the same versioning. datafusion-spark 57 would just be released from the new repo instead of from the datafusion repo.

We will need to add all the release scripts to this repo once the code has been imported

Navigation Menu

Repository navigation

[DISCUSSION] How should we test datafusion-spark against Spark? #27103

Challenges

Options

Impact on downstream projects

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[DISCUSSION] How should we test datafusion-spark against Spark? #27103

Description

Challenges

Options

Impact on downstream projects

Activity

alamb commented on Nov 4, 2026

comphead commented on Nov 4, 2026

andygrove commented on Nov 4, 2026

coderfender commented on Nov 4, 2026

viirya commented on Nov 4, 2026

hsiang-c commented on Nov 4, 2026

shehabgamin commented on Nov 5, 2026

xanderbailey commented on Nov 5, 2026

Jefffrey commented on Nov 5, 2026

shehabgamin commented on Nov 5, 2026

alamb commented on Nov 5, 2026

andygrove commented on Nov 5, 2026

mbutrovich commented on Nov 5, 2026

andygrove commented on Nov 6, 2026

alamb commented on Nov 8, 2026

andygrove commented on Nov 11, 2026

comphead commented on Nov 11, 2026

andygrove commented on Nov 11, 2026

andygrove commented on Nov 11, 2026

andygrove commented on Nov 11, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions