Repository navigation
ci(oracles): check that the e2e gap tests partition into CI shards - #34
Merged
Merged
Conversation
… shards scripts/e2e-shards.mjs names five shards of the e2e oracle's tests, balanced by the costs measured on main. Its check lists the file's tests with vitest list and fails when a test is in no shard or in several, or when a shard names no test. Its pattern command prints a shard's -t pattern, which selects the test under Vitest 4's and Vitest 5's name joins. The Test job runs the check (3 s). CI keeps its one fixed-seed e2e job: until the property registers its parts as units, the determinism test alone would be a shard of about 14 minutes. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
|
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars.
 |
commit: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This change adds a checked partition of the end-to-end gap oracle's tests into five CI shards. CI does not run the shards yet. The partition waits until the property registers its parts as units. Until then, the check keeps the partition in step with the tests.
Why the shards wait
The end-to-end gap oracle (
oracle-gaps/gap-report-oracle.property.test.mjs) registers 13 tests: the fixed-seed campaign, the random campaign, 10 calibrations, and the determinism test. A shard can only hold whole tests. The determinism test runs all 30 named histories twice, about 345 s locally and about 800 s on CI. Any 5-way split therefore has one job of about 14 minutes, past the limit of 10 minutes for CI. So CI keeps its one fixed-seed job for now. When the property registers its campaigns and determinism histories as separate tests, the partition grows to those units and CI runs it as a matrix.What the partition promises
Every test of the file is in exactly one shard. A test in no shard would leave its law out of CI. A test in two shards would run twice. The check script enforces both rules against what
vitest listreports:The check fails in three cases:
The script anchors each shard's
-tpattern, sofixed seed 20261105does not also select a test whose name extends it. The pattern accepts both name joins: Vitest 4 joins a suite and a test with a space, and Vitest 5 joins them with>. The check simulates the Vitest 5 join on the names thatvitest listprints. No shard ran under Vitest 5.The shards balance the costs measured on
mainat 84cd34a, longest first:Where it runs
The Test job runs
pnpm --filter @tanstack/oracles run test:oracle-gaps:shards, which takes about 3 s. It checks the partition ahead of the shard jobs, which CI does not run yet. The script lives inpackages/oracles/scripts/, which the package does not publish.Verification
npm pack --dry-runlists no file underscripts/.This pull request and its description were written by Isaac.