detect errors in datafusion and fix datafusion vortex on c8g.metal-48xl - #1463
Open
toschmidt wants to merge 2 commits into
Open
detect errors in datafusion and fix datafusion vortex on c8g.metal-48xl#1463toschmidt wants to merge 2 commits into
toschmidt wants to merge 2 commits into
Conversation
datafusion-cli and vortex-datafusion-cli exit 0 even when a query errors
mid-batch: they print the failure to stderr and carry on, leaving only
create.sql's own ~0.01s Elapsed line on stdout. The query scripts merged
stderr into stdout (2>&1) and reported the last Elapsed line, so a failed
query was silently recorded as a ~0.011s runtime.
On c8g.metal-48xl this masked real failures: wide scans over the 100
partitioned .vortex files exhaust the open-file limit ('Too many open
files (os error 24)') and were published as bogus 40-200x speedups.
Capture stderr separately and treat any stderr output (or a nonzero exit)
as a failed run, so the driver records null and surfaces the error instead
of a fake near-zero time. A successful query writes nothing to stderr.
vortex-io opens all partition files concurrently, so on many-core hosts (e.g. c8g.metal-48xl, 192 vCPU) a wide scan over the 100 .vortex files exceeds the default open-file soft limit of 1024 and fails with 'Too many open files (os error 24)'. The failure is a flaky race — a different subset of queries fails each run. Raise the soft limit to the hard cap (no privilege required) at the top of the query script; the spawned CLI inherits it. Also add the c8g.metal-48xl results (partitioned and single) produced with the fix in place: the full 43-query sweep completes with zero failures and real times instead of the earlier bogus ~0.011s.
toschmidt
requested a deployment
to
benchmark-approval
August 18, 2026 22:05 — with 
GitHub Actions
Waiting
GitHub Actions
Waiting
Contributor
Author
|
@AdamGS do you agree with the change? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Some queries in datafusion vortex partitioned failed due to the open file limit. The runner incorrectly reported the execution time of create table as query runtime leading to incorrect runtimes for some queries.
Errors are now detected correctly and the open file limit is increased before starting datafusion.