Skip to content

Commit 0a2ecc9

Browse files
DavertMikDavertMikclaude
authored
feat(workers): run serial tests after parallel workers (#5728)
* feat(workers): run serial tests after parallel workers Tests matching the new `serial` config pattern are kept out of the parallel workers. Once those exit, one more worker starts and runs the serial tests one by one, inside the same run: bootstrapAll, teardownAll and workers.result fire once and the exit code covers both parts. Works with every --by strategy and combines with --grep. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(workers): add --serial option to run-workers Sets or overrides the serial config pattern for a single run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: DavertMik <davert@testomat.io> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 37a2093 commit 0a2ecc9

13 files changed

Lines changed: 244 additions & 20 deletions

File tree

‎bin/codecept.js‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -209,6 +209,7 @@ program
209209
.option('-i, --invert', 'inverts --grep matches')
210210
.option('-o, --override [value]', 'override current config options')
211211
.option('--suites', 'parallel execution of suites not single tests')
212+
.option('--serial <pattern>', 'run tests matching <pattern> one by one after parallel workers finish')
212213
.option('--by <strategy>', 'test distribution strategy: "test" (pre-assign individual tests), "suite" (pre-assign test suites), or "pool" (dynamic distribution for optimal load balancing, recommended)')
213214
.option(commandFlags.debug.flag, commandFlags.debug.description)
214215
.option(commandFlags.verbose.flag, commandFlags.verbose.description)

‎docs/commands.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -157,6 +157,9 @@ npx codeceptjs run-workers 3 --by suite
157157

158158
# Pool mode with filtering
159159
npx codeceptjs run-workers 4 --by pool --grep "@smoke"
160+
161+
# Run tests tagged @serial one by one after parallel workers finish
162+
npx codeceptjs run-workers 3 --serial "@serial"
160163
```
161164

162165
**Test Distribution Strategies:**
@@ -165,6 +168,8 @@ npx codeceptjs run-workers 4 --by pool --grep "@smoke"
165168
- `--by suite`: Pre-assigns entire test suites to workers
166169
- `--by pool`: Dynamic distribution for optimal load balancing (recommended for best performance)
167170

171+
`--serial <pattern>` keeps tests matching the pattern out of parallel workers and runs them one by one in an extra worker once the others finish, within the same run. It overrides the `serial` config option. See [Serial tests](/parallel#serial-tests).
172+
168173
The pool mode provides the best load balancing by maintaining tests in a shared pool and distributing them dynamically as workers become available. This prevents workers from sitting idle and ensures optimal CPU utilization, especially when tests have varying execution times.
169174

170175
See [Parallel Execution](/parallel) documentation for more details.

‎docs/configuration.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,7 @@ export const config = {
4444

4545
- `timeout` — default per-test timeout in seconds; a test is killed if it stops responding.
4646
- `mocha` — [Mocha options](https://mochajs.org/#configuring-mocha-nodejs), including extra reporters. See [Reporters](/reports).
47+
- `serial` — pattern for tests that must not run in parallel, e.g. `serial: '@serial'`. With `run-workers` they run one by one after all parallel workers finish. `--serial` overrides it. See [Serial tests](/parallel#serial-tests).
4748
- `workerInitializationDelay` — delay in milliseconds between spinning up parallel workers to prevent CPU spikes and stagger browser startup. Defaults to `200`. Set to `0` to disable.
4849
- `workerInitializationMaxDelay` — maximum total delay (in milliseconds) for worker initialization staggering. Defaults to `10000` (10 s). Set to `0` to disable capping.
4950

‎docs/parallel.md‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,40 @@ npx codeceptjs run-workers 4 --by pool
4040

4141
`--suites` is shorthand for `--by suite`.
4242

43+
### Serial tests
44+
45+
Some tests can't share the stage: they change global settings, reset data, or toggle a feature flag that other tests read. Mark them with a tag and set the `serial` pattern in config or pass it with `--serial`:
46+
47+
```js
48+
// codecept.conf.js
49+
export const config = {
50+
serial: '@serial',
51+
}
52+
```
53+
54+
```js
55+
Scenario('change account timezone @serial', ({ I }) => {
56+
// ...
57+
})
58+
```
59+
60+
`run-workers` keeps matching tests out of the parallel workers. When those workers finish, one more worker starts and runs the serial tests one by one. It is still a single run: `bootstrapAll`, `teardownAll` and `event.workers.result` fire once, and the exit code covers both parts.
61+
62+
```
63+
Test distribution:
64+
Worker 1: 14 tests
65+
Worker 2: 13 tests
66+
Worker 3: 4 tests (serial)
67+
```
68+
69+
Pass `--serial` to set or override the pattern for a single run:
70+
71+
```sh
72+
npx codeceptjs run-workers 3 --serial "@serial"
73+
```
74+
75+
`serial` is matched against the full test title the same way `--grep` is, so a tag on a `Feature` makes all its scenarios serial. It works with every `--by` strategy and combines with `--grep`: only serial tests that also match grep are run. With [multiple browsers](#multiple-browsers) each profile gets its own serial worker, started one after another.
76+
4377
### Multiple browsers
4478

4579
Define browser profiles under `multiple` in `codecept.conf.js`:

‎lib/command/run-workers.js‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -55,7 +55,7 @@ export default async function (workerCount, selectedRuns, options) {
5555
output.print('Test distribution:')
5656
workerObjects.forEach((worker, index) => {
5757
const testCount = worker.tests.length
58-
output.print(` Worker ${index + 1}: ${testCount} test${testCount !== 1 ? 's' : ''}`)
58+
output.print(` Worker ${index + 1}: ${testCount} test${testCount !== 1 ? 's' : ''}${worker.serial ? ' (serial)' : ''}`)
5959
})
6060
output.print()
6161

‎lib/command/workers/runTests.js‎

Lines changed: 9 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ let stdout = ''
1717

1818
const stderr = ''
1919

20-
const { options, tests, testRoot, workerIndex, poolMode } = workerData
20+
const { options, tests, skipTests = [], testRoot, workerIndex, poolMode } = workerData
2121

2222
// Global error handlers to catch critical errors but not test failures
2323
process.on('uncaughtException', (err) => {
@@ -287,6 +287,7 @@ async function runPoolTests() {
287287
// Load only the assigned test file
288288
mocha.files = [testIdentifier]
289289
await loadTests(mocha)
290+
filterSuites(mocha.suite.suites, test => !skipTests.includes(test.uid))
290291

291292
if (mocha.suite.total() > 0) {
292293
// Run only the tests in the current mocha suite
@@ -423,16 +424,14 @@ async function filterTests() {
423424
mocha.files = files
424425
await loadTests(mocha)
425426

426-
// Recursively filter tests in all suites (including nested ones)
427-
const filterSuiteTests = (suite) => {
428-
suite.tests = suite.tests.filter(test => tests.indexOf(test.uid) >= 0)
429-
for (const childSuite of suite.suites) {
430-
filterSuiteTests(childSuite)
431-
}
432-
}
427+
filterSuites(mocha.suite.suites, test => tests.indexOf(test.uid) >= 0)
428+
}
433429

434-
for (const suite of mocha.suite.suites) {
435-
filterSuiteTests(suite)
430+
// Recursively filter tests in all suites (including nested ones)
431+
function filterSuites(suites, keep) {
432+
for (const suite of suites) {
433+
suite.tests = suite.tests.filter(keep)
434+
filterSuites(suite.suites, keep)
436435
}
437436
}
438437

‎lib/workers.js‎

Lines changed: 57 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -68,11 +68,12 @@ const withoutEmptyGroups = groups => {
6868
return nonEmpty.length ? nonEmpty : groups.slice(0, 1)
6969
}
7070

71-
const createWorker = (workerObject, isPoolMode = false) => {
71+
const createWorker = (workerObject, isPoolMode = false, skipTests = []) => {
7272
const worker = new Worker(pathToWorker, {
7373
workerData: {
7474
options: simplifyObject(workerObject.options),
7575
tests: workerObject.tests,
76+
skipTests,
7677
testRoot: workerObject.testRoot,
7778
workerIndex: workerObject.workerIndex + 1,
7879
poolMode: isPoolMode,
@@ -224,6 +225,7 @@ class WorkerObject {
224225
*/
225226
constructor(workerIndex) {
226227
this.workerIndex = workerIndex
228+
this.serial = false
227229
this.options = {}
228230
this.tests = []
229231
this.testRoot = getTestRoot()
@@ -299,6 +301,7 @@ class Workers extends EventEmitter {
299301
this.testGroups = []
300302
this.testPool = []
301303
this.testPoolInitialized = false
304+
this.serialTests = []
302305
this.isPoolMode = config.by === 'pool'
303306
this.activeWorkers = new Map()
304307
this.maxWorkers = numberOfWorkers // Track original worker count for pool mode
@@ -323,9 +326,21 @@ class Workers extends EventEmitter {
323326

324327
async _initWorkers(numberOfWorkers, config) {
325328
await this.splitTestsByGroups(numberOfWorkers, config)
329+
this.serialTests = await this.createGroupOfSerialTests()
330+
if (this.serialTests.length && !this.isPoolMode) {
331+
this.testGroups = this.testGroups.map(group => group.filter(uid => !this.serialTests.includes(uid))).filter(group => group.length)
332+
}
326333
// For function-based grouping, use the actual number of test groups created
327334
const actualNumberOfWorkers = isFunction(config.by) ? this.testGroups.length : numberOfWorkers
328335
this.workers = createWorkerObjects(this.testGroups, this.codecept.config, getTestRoot(config.testConfig), config.options, config.selectedRuns)
336+
if (this.serialTests.length) {
337+
const serialWorkers = createWorkerObjects([this.serialTests], this.codecept.config, getTestRoot(config.testConfig), config.options, config.selectedRuns)
338+
for (const worker of serialWorkers) {
339+
worker.workerIndex = this.workers.length
340+
worker.serial = true
341+
this.workers.push(worker)
342+
}
343+
}
329344
this.numberOfWorkers = this.workers.length
330345
}
331346

@@ -408,6 +423,32 @@ class Workers extends EventEmitter {
408423
return groups
409424
}
410425

426+
/**
427+
* Collects tests matching `--serial` option or `serial` config pattern.
428+
* These tests are excluded from parallel workers and run one by one after them.
429+
*
430+
* @returns {Promise<string[]>}
431+
*/
432+
async createGroupOfSerialTests() {
433+
const pattern = this.options.serial || this.codecept?.config.serial
434+
if (!pattern) return []
435+
const serial = new RegExp(pattern)
436+
437+
Container.createMocha(this.codecept.config.mocha || {}, this.options)
438+
const mocha = Container.mocha()
439+
mocha.files = this.codecept.testFiles
440+
await loadTests(mocha, { reload: true })
441+
442+
const group = []
443+
mocha.suite.eachTest(test => {
444+
if (test && serial.test(test.fullTitle()) && matchesGrep(mocha, test)) group.push(test.uid)
445+
})
446+
447+
mocha.unloadFiles()
448+
449+
return group
450+
}
451+
411452
/**
412453
* @param {Number} numberOfWorkers
413454
*/
@@ -534,7 +575,8 @@ class Workers extends EventEmitter {
534575
: 0
535576

536577
for (const worker of this.workers) {
537-
const workerThread = createWorker(worker, this.isPoolMode)
578+
if (worker.serial) continue
579+
const workerThread = createWorker(worker, this.isPoolMode, this.serialTests)
538580
this._listenWorkerEvents(workerThread)
539581
workerThreads.push(workerThread)
540582

@@ -545,6 +587,8 @@ class Workers extends EventEmitter {
545587
}
546588
}
547589

590+
if (!workerThreads.length) this._runSerialWorker()
591+
548592
recorder.add('workers started', () => {
549593
// Workers are already running, this is just a placeholder step
550594
})
@@ -556,6 +600,14 @@ class Workers extends EventEmitter {
556600
})
557601
}
558602

603+
_runSerialWorker() {
604+
const parallelWorkers = this.workers.filter(worker => !worker.serial)
605+
if (this.closedWorkers < parallelWorkers.length) return
606+
const worker = this.workers[this.closedWorkers]
607+
if (!worker) return
608+
this._listenWorkerEvents(createWorker(worker))
609+
}
610+
559611
/**
560612
* @returns {Array<WorkerObject>}
561613
*/
@@ -787,14 +839,10 @@ class Workers extends EventEmitter {
787839
process.exitCode = 1
788840
}
789841

790-
if (this.isPoolMode) {
791-
// Pool mode: finish when all workers have exited and no more tests
792-
if (this.closedWorkers === this.numberOfWorkers) {
793-
this._finishRun()
794-
}
795-
} else if (this.closedWorkers === this.numberOfWorkers) {
796-
// Regular mode: finish when all original workers have exited
842+
if (this.closedWorkers === this.numberOfWorkers) {
797843
this._finishRun()
844+
} else {
845+
this._runSerialWorker()
798846
}
799847
})
800848
}
Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
export const config = {
2+
tests: './workers-serial/*_test.js',
3+
timeout: 10000,
4+
output: './output',
5+
helpers: {
6+
FileSystem: {},
7+
},
8+
include: {},
9+
serial: '@serial',
10+
mocha: {},
11+
name: 'sandbox',
12+
};
Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
import { track } from './timeline.js'
2+
3+
Feature('Serial lane @serial')
4+
5+
Scenario('serial one', async () => {
6+
await track('serial one')
7+
})
8+
9+
Scenario('serial two', async () => {
10+
await track('serial two')
11+
})
Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
import { track } from './timeline.js'
2+
3+
Feature('Parallel lane')
4+
5+
Scenario('parallel one', async () => {
6+
await track('parallel one')
7+
})
8+
9+
Scenario('parallel two', async () => {
10+
await track('parallel two')
11+
})
12+
13+
Scenario('serial three @serial', async () => {
14+
await track('serial three')
15+
})
16+
17+
Scenario('parallel three', async () => {
18+
await track('parallel three')
19+
})

0 commit comments

Comments
 (0)