Repository navigation
ci: burn-in groups all report, known flakes quarantined - #66
Merged
Merged
Conversation
fail-fast is off for the burn-in matrix, so one failing group no longer cancels the others. Known flakes are listed in scripts/ci/quarantine.json with their BACKLOG entry: a run whose only failures are quarantined tests is reported in the job summary as a known flake and does not fail the burn-in. Any other failure still fails it, and nothing is skipped. Quarantined: watcher-new-folder, removed and made again at the same path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The burn-in matrix no longer cancels itself on one failure.
fail-fast: falseon the burn-in matrix: every group runs its repetitions to the end and reports. One failing group still fails the pull request.scripts/ci/quarantine.jsonlists known flakes (spec file, test name asdescribe > title, BACKLOG entry). A vitest run whose only failures are quarantined tests is reported in the job summary as "known flake, see BACKLOG" and the repetitions go on. Any other failure still fails the run (also a file that fails outside a test, or a run with no report). Nothing is skipped: a quarantined test runs every time and passes when it passes.watcher-new-folder.spec.ts> "removed and made again at the same path" (5 events where 4 were expected; failed in all 3 CI burn-ins of dev-server 2/3, 0 of 60 container runs). No watcher change.Checked locally with the quarantined test forced to fail: exit 0 and the known-flake summary; with a second test forced to fail as well: exit 1, the other test named as the failure. New cases in
scripts/ci/burn-in.test.mjs, shown red against a mutation that quarantines every failure.