diff --git a/CLAUDE.md b/CLAUDE.md index e364b8c8..4ffe3ee7 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -241,6 +241,17 @@ Documented divergences from the conventions above. They exist today as debt to b - **Nightwatch: `retain-on-failure` works; the other retry-aware policies degrade.** Its `--retries` re-runs the testcase *internally* without re-firing the plugin's per-test hooks, and the per-testcase results carry no attempt/retry field (retries live only in undocumented, version-varying Nightwatch internals — `suiteRetries.testRetriesCount` / `reporter.testResults.retryTest`), so the ledger sees only the final attempt for the `describe/it` and exports-object interfaces. Cucumber scenarios expose per-scenario hooks, so the feed captures their attempts. Not cleanly fixable without depending on those internals. - WDIO `specFileRetries` spawns a fresh worker per retry, so cross-process attempts aren't in the (process-scoped) ledger. - Run identity across worker sockets is env-propagated. `core/run-id.ts` `resolveRunId()` publishes `DEVTOOLS_RUN_ID` (`RUNNER_ENV.RUN_ID`) and every worker socket carries it as `?runId=` (`WORKER_WS_QUERY`), so the backend keeps accumulated run state when the *next spec's* worker connects and wipes it only for a genuinely new run. Without it every connect read as a new run: Preserve & Rerun 409'd for every spec except the last one that ran, and a dashboard opened mid-run replayed only the current spec. The WDIO service stamps it in the launcher's `onPrepare`, before workers fork, so all workers of one run agree; single-process adapters self-stamp on first use. **Gap: multi-process parallel runs in Selenium/Nightwatch** (jest/vitest workers, nightwatch `test_workers`) load the plugin per worker with no launcher-side hook to stamp first, so each worker generates its own id and still reads as a new run — the pre-fix behaviour, not a regression. Deriving the fallback from `process.ppid` would group those siblings, but would also make two sequential single-process runs share an id and inherit each other's state against a standalone dashboard, so the per-process fallback stands. +- **A rerun template selects EITHER by name pattern or by exact id, and the two cannot share a slot.** `shared/src/runner.ts` `RERUN_SLOT` names both, and `backend/src/runner.ts` `#resolveGenericCommand` branches on which one the adapter's template carries: `{{testName}}` is filled from `label`/`fullTitle` through `escapeFilterRegex` because mocha `--grep`, jest `--testNamePattern` and cucumber `--name` all match by regex; `{{testId}}` is filled from `uid`, shell-quoted and **never escaped**, because pytest selects by nodeid and matches it literally. Measured: `pytest 'test_thing\.py::test_a'` collects nothing and exits 0, so the escaped form fails as a rerun that appears to have run and passed. Which slot a payload can service also decides `isTargetedRerun` — an id template needs a `uid`, not a label. + - A pytest nodeid addresses a file, a class or one test in one syntax (`file.py`, `file.py::Class`, `file.py::Class::test`), and the Python adapter's uids already *are* nodeids at all three levels, so one slot covers every row the tree offers — no per-level filter flag and no cucumber-style feature special case. Verified end-to-end: substituting a test nodeid collects 1, a file nodeid collects 3. + - `selenium-devtools-py/src/selenium_devtools/rerun.py` derives both commands from pytest's own view of its invocation (`config.invocation_params.args` plus `config.args` for which of them were positional) rather than parsing argv itself: inferring positionals needs a table of every option that takes a value, and dropping a value while keeping its option makes that option swallow the appended id. Capabilities are **derived from which commands got built**, never declared — the backend's fallback for a rerun it was given no command for is the wdio binary, so an advertised-but-unserviceable control is worse than an absent one. A plain script publishes a launch command only and advertises Run-all alone. + - Selectors are stripped from a targeted rerun (`-k`, `-m`, `--deselect`, `--lf`/`--ff`/`--sw` family, `-n`/`--numprocesses`/`--dist`): the rerun already names its test, so a surviving filter can only narrow further — usually to nothing, which pytest reports as a clean exit. Positionals go too, or a rerun's own child would union the inherited nodeid with the next one and each generation would run one test more. The xdist flags also go because each worker would connect under its own run id. + - The rerun spawns in pytest's **rootdir** (`RUNNER_ENV.RUNNER_CWD`, stamped before the backend is launched so its process inherits it): a nodeid is reported relative to rootdir while a positional path resolves against the process's cwd, so anywhere else makes every nodeid a path that does not exist. The launch command's positionals are absolutised for the same reason. The variable is *replaced* on a second `enable()` in one process but only while it still holds **the value we wrote** (tracked, and deliberately surviving `reset()`): our leftover would otherwise spawn the next run's reruns in the previous project, while a value someone exported since is an instruction. A boolean "we wrote it once" cannot serve both — it says nothing about whether the current value is still ours. The remaining ambiguity is accepted and untouchable: a caller who exports the *same* path we already stamped is byte-identical to our leftover in the only channel there is, so that override is replaced; pinning a directory across runs works by exporting it before the first `enable()`, which is never claimed as ours. Residual: an option carrying a *relative* path (`-c`, `--junitxml`) resolves against rootdir on a rerun, and an already-running dashboard keeps the directory it was started in. + - **A rerun does not travel down the worker socket.** `POST /api/tests/run` spawns a fresh process; the socket carries only `clientConnected`/`clientDisconnected`. So the single-`workerSocket` limitation is about which process the *dashboard state* belongs to under `pytest -n`, not about routing the rerun. + - **A spawned rerun must be pointed back at the backend that asked for it, or it reports into a dashboard nobody is looking at.** `REUSE_ENV` (`DEVTOOLS_APP_REUSE`/`_HOST`/`_PORT`) is how the backend does that, and an adapter that ignores it launches a *second* backend and a *second* window: measured on the Python adapter, a rerun opened a new dashboard carrying the rerun's data while the window the user pressed Rerun in stayed as it was — which reads as a rerun that captured nothing. `backend.py` `reuse_target()` now attaches to it **ahead of `DEVTOOLS_PORT`** (that variable is an ambient preference inherited from the parent; the handshake names the backend that requested *this* run), and the window gate lives in `lifecycle.auto_open_enabled()` rather than at the `enable()` call site so it is directly testable. An incomplete handshake deliberately still opens a window — no usable target means the child launched its own backend, and then the window is the only way to see it. + - A plain script's tree is one synthetic suite holding one synthetic test, and both denote the whole run, so its launch command doubles as its rerun template (no slot — the backend substitutes nothing) and all three controls are honest. Refusing the row-scoped ones instead would disable the button beside the only row the tree has. + - **Two unrelated events share the `clearExecutionData` scope, and the receiver cannot tell them apart from the uid.** A run STARTING (`backend/src/index.ts` `handleTestRun`, one per `POST /api/tests/run`) and ONE ENTRY resetting inside a run already in flight (`nightwatch-devtools/src/cucumber-lifecycle.ts`, which re-emits a scenario suite and must not wipe its siblings) arrive under the same scope with the same shape. The app inferred the difference by comparing the uid against `rerunState.activeRerunSuiteUid` — a latch that outlived its rerun, so the *next* run start at a different scope read as a child clear of the last one and **skipped its wipe entirely**: rerun a suite, then the file or Tests, and the Actions/Console/Network tabs kept the previous run's rows and grew with each rerun. `ClearExecutionDataWsPayload.runStart` now states it on the wire (it has to be on the wire, not local to the clicking window — popouts see only WS events), and the app clears both latches when it is set. A backend test asserts the flag actually ships: the app-side fix reads it, so dropping it would restore the bug with every app test still green. + - Still open, same class: `app/src/components/browser/snapshot.ts` `#videos` is only ever pushed to, so the screencast "Recording N" dropdown accumulates every session of every run for the life of the page (observed at 17). That component listens only to the `screencast-ready` window event and never learns a run started. + - **A rerun's process collects a SUBSET, so anything it derives from "this collection" is wrong for the tree it merges into.** Two bugs of that one shape, both found by rerunning a single pytest test: (a) `SuiteStats.order` — which `test-entry-state.ts` `orderedChildren` sorts a suite's tests and child suites by — was pytest's `enumerate(session.items)` index, so a rerun restamped its one test as position 0 and the row jumped above the class it was written below. It is now the item's **source line**, a property of the test rather than of the collection; within a module pytest collects in definition order, so the two agree wherever both are meaningful (a plugin that reorders collection is the exception, and there the line is the more stable answer anyway). (b) `suite-merge.ts` `resetStaleChildrenOnRerun` flipped every settled child *suite* to `pending` whenever an incoming suite arrived `pending` — but a single-test rerun re-emits the parent as `pending` carrying only the one test it collected, so a sibling class suite was set spinning and never reported again, keeping the spinner for the rest of the session with all of its own tests still green. `mergeTests` already froze sibling *tests* on `activeRerunTestUid`; that guard now covers child suites too. A suite on the path to the target is unaffected either way — it re-reports its own state. - **Chrome discards all WebDriver-synthesized input to a tab after a breached credential is submitted.** The first time a test types a `(username, password)` pair that Chrome's password-leak check finds in a breach corpus into an `` and submits a form whose destination no longer shows that login form, Chrome queries `passwordsleakcheck-pa.googleapis.com` and ~0.3-0.9 s later stops delivering **all** synthesized input — mouse *and* keyboard — to that tab. chromedriver returns HTTP 200 for every subsequent Element Click / Send Keys; nothing reaches the page. Untrusted JS (`element.click()`) still works and direct CDP `Input.dispatchMouseEvent`/`dispatchKeyEvent` are equally dead, so this is Chrome, not chromedriver and not our capture. `tomsmith` / `SuperSecretPassword!` — the-internet's demo credential — triggers it; changing only the *username* does not, nor does a random password. - **Workaround: add `--host-resolver-rules=MAP passwordsleakcheck-pa.googleapis.com 127.0.0.1` to the browser args.** Both examples do. Verified 3/3 on the WDIO mocha example and on the Nightwatch example, where it also fixes the **within-one-test** logout click that a session reset never could. `--guest` also works (3/3); `--incognito` works at the raw-WebDriver level but WebdriverIO rejects it at session creation; disabling the password manager via `prefs` does **not** (6/6 still fail). - **Not a version regression, not headless-specific, not the site, not "the Nth navigation".** Measured identically on Chrome 149.0.7827.155 / 150.0.7871.124 / 151.0.7922.77 / 152.0.7977.30 with matched chromedrivers (5/5 each), headless and headed, and on a purely local two-page static form. It fires **once per browser profile** on a wall clock — a liveness probe that never navigates again goes dead 904 ms after the submit — so the historical ~25% intermittency was the race between the next input command and that round trip. Do **not** pin `browserVersion` to 149; every part of the earlier "Chrome 150 regression, fixed in 151" attribution is contradicted. diff --git a/packages/app/src/components/sidebar/test-entry-state.ts b/packages/app/src/components/sidebar/test-entry-state.ts index c27750f8..43e16bca 100644 --- a/packages/app/src/components/sidebar/test-entry-state.ts +++ b/packages/app/src/components/sidebar/test-entry-state.ts @@ -28,8 +28,10 @@ function isSuiteFragment(entry: Fragment): entry is SuiteStatsFragment { * still executes first. jasmine matches it. Those runners set no `order` and * keep this shape. * - * pytest instead runs in collection order and interleaves module-level tests - * with classes, so it stamps `order` and the two buckets are merged by it. Only + * pytest instead interleaves module-level tests with classes, so it stamps + * `order` and the two buckets are merged by it. That stamp is the child's + * source line, not its position in the run: a rerun collects one test, and an + * index from that collection would say 0 and move the row to the top. Only * applied when EVERY child carries one — a partially stamped suite would sort * the unstamped children into a position nothing asked for. */ diff --git a/packages/app/src/controller/DataManager.ts b/packages/app/src/controller/DataManager.ts index 42a13da2..3748b64f 100644 --- a/packages/app/src/controller/DataManager.ts +++ b/packages/app/src/controller/DataManager.ts @@ -260,8 +260,18 @@ export class DataManagerController implements ReactiveController { } #handleClearExecutionScope(data: unknown): void { - const { uid, entryType, clearSuiteTree } = + const { uid, entryType, clearSuiteTree, runStart } = data as SocketMessage<'clearExecutionData'>['data'] + // A run is starting, so nothing is in flight for the child-clear rules + // below to apply to. Those latches track ONE rerun; left standing they + // made the next rerun at a different scope look like a child clear of the + // last one, and its wipe was skipped — so pressing Rerun on a suite and + // then on the file (or on Tests) kept the previous run's actions, console + // and network rows and grew them run after run. + if (runStart) { + rerunState.activeRerunSuiteUid = undefined + this.#activeRerunTestUid = undefined + } this.clearExecutionData(uid, entryType) if (clearSuiteTree) { this.suitesContextProvider.setValue([]) diff --git a/packages/app/src/controller/suite-merge.ts b/packages/app/src/controller/suite-merge.ts index 2904d625..809884a6 100644 --- a/packages/app/src/controller/suite-merge.ts +++ b/packages/app/src/controller/suite-merge.ts @@ -166,6 +166,14 @@ export function mergeChildSuites( // mark them 'pending' so they render as a spinner instead of a stale check. // Exception: child-scope rerun (activeRerunSuiteUid differs from the // incoming feature suite's uid) — sibling scenarios keep terminal states. +// +// Also skipped during a SINGLE-TEST rerun, which `mergeTests` already freezes +// siblings for — this is the same rule for the other kind of sibling. A child +// suite on the path to the target re-reports its own state anyway, while one +// off the path never reports again, so flipping it to 'pending' left it +// spinning for the rest of the session. Reproduced on pytest, whose tree puts +// a class suite beside a module-level test: rerunning the module-level test +// left the class in flight with all its tests still showing green. function resetStaleChildrenOnRerun( mergedSuites: SuiteStatsFragment['suites'] | undefined, incoming: SuiteStatsFragment, @@ -173,7 +181,12 @@ function resetStaleChildrenOnRerun( ): SuiteStatsFragment['suites'] | undefined { const isChildRerun = !!ctx.activeRerunSuiteUid && ctx.activeRerunSuiteUid !== incoming.uid - if (incoming.state !== 'pending' || !mergedSuites || isChildRerun) { + if ( + incoming.state !== 'pending' || + !mergedSuites || + isChildRerun || + ctx.activeRerunTestUid + ) { return mergedSuites } return mergedSuites.map((s) => diff --git a/packages/app/tests/data-manager.test.ts b/packages/app/tests/data-manager.test.ts index fe666825..7db7bc8a 100644 --- a/packages/app/tests/data-manager.test.ts +++ b/packages/app/tests/data-manager.test.ts @@ -830,6 +830,56 @@ describe('DataManagerController', () => { expect(manager.suitesContextProvider.value).toEqual([]) }) + it('wipes execution data on every run start, not just the first', async () => { + // The second rerun of a session used to keep the first one's rows: a + // suite rerun latches its uid to recognise Nightwatch's mid-run child + // clears, and the latch outlived the run — so the next run start, at a + // different scope, was misread as a child clear and skipped its wipe. + const { manager, deliver } = await boot() + deliver('suites', suitesFrame(suite('login-suite'))) + + deliver(WS_SCOPE.clearExecutionData, { + uid: 'login-suite', + entryType: 'suite', + runStart: true + }) + deliver('commands', [command()]) + deliver('consoleLogs', [{ type: 'log', args: ['from the first rerun'] }]) + deliver('networkRequests', [request()]) + + // A second run at a DIFFERENT scope — the file, or Tests. + deliver(WS_SCOPE.clearExecutionData, { + uid: RUN_ALL_UID, + entryType: 'suite', + runStart: true + }) + + expect(manager.commandsContextProvider.value).toEqual([]) + expect(manager.consoleLogsContextProvider.value).toEqual([]) + expect(manager.networkRequestsContextProvider.value).toEqual([]) + }) + + it('still spares a sibling when one entry resets mid-run', async () => { + // Nightwatch re-emits a cucumber scenario suite while the run is in + // flight, and only that scenario's data may go. That clear carries no + // `runStart`, which is what distinguishes it. + const { manager, deliver } = await boot() + deliver('suites', suitesFrame(suite('feature'))) + deliver(WS_SCOPE.clearExecutionData, { + uid: 'feature', + entryType: 'suite', + runStart: true + }) + deliver('commands', [command()]) + + deliver(WS_SCOPE.clearExecutionData, { + uid: 'feature/scenario-2', + entryType: 'suite' + }) + + expect(manager.commandsContextProvider.value).toHaveLength(1) + }) + it('fails the tests still in flight when the run is stopped', async () => { const { manager, deliver } = await boot() deliver( diff --git a/packages/app/tests/suite-merge.test.ts b/packages/app/tests/suite-merge.test.ts index 3fe23b9b..1e440879 100644 --- a/packages/app/tests/suite-merge.test.ts +++ b/packages/app/tests/suite-merge.test.ts @@ -223,6 +223,39 @@ describe('mergeSuite', () => { ) }) + it('keeps a sibling child suite settled during a single-test rerun', () => { + // pytest's tree puts a class suite beside a module-level test. Rerunning + // that test re-emits the file suite as 'pending' with only the one test it + // collected — the class is never mentioned again, so flipping it to + // 'pending' left it spinning for the rest of the session with all of its + // own tests still showing green. Sibling TESTS are already frozen by + // mergeTests; this is the same rule for the other kind of sibling. + const existing = suite('file.py', { + tests: [test('file.py::test_module_level')], + suites: [ + suite('file.py::TestLogin', { + state: 'passed', + tests: [test('file.py::TestLogin::test_valid')] + }) + ] + }) + const incoming = suite('file.py', { + state: 'pending', + tests: [test('file.py::test_module_level', { state: 'pending' })], + suites: [] + }) + + const merged = mergeSuite( + existing, + incoming, + ctx({ activeRerunTestUid: 'file.py::test_module_level' }) + ) + + const cls = merged.suites?.find((s) => s.uid === 'file.py::TestLogin') + expect(cls?.state).toBe('passed') + expect(cls?.end).toBeDefined() + }) + it('strips undefined/null state from incoming to preserve existing state', () => { const existing = suite('s', { state: 'passed' }) const incoming = suite('s', { diff --git a/packages/backend/src/index.ts b/packages/backend/src/index.ts index 02a51fbd..9382eef0 100644 --- a/packages/backend/src/index.ts +++ b/packages/backend/src/index.ts @@ -146,10 +146,12 @@ async function handleTestRun( `run ${body.entryType} uid=${body.uid} framework=${body.framework} spec=${body.specFile} title=${JSON.stringify(body.fullTitle)}` ) // Broadcast a clear so popouts (which only see WS events) wipe too. + // `runStart` says this clear is a run beginning rather than one entry + // resetting mid-run, which the receiver cannot tell from the uid alone. broadcastToClients( JSON.stringify({ scope: WS_SCOPE.clearExecutionData, - data: { uid: body.uid, entryType: body.entryType } + data: { uid: body.uid, entryType: body.entryType, runStart: true } }) ) // Plain Rerun hides the Compare tab by dropping all baselines. diff --git a/packages/backend/src/runner.ts b/packages/backend/src/runner.ts index cc9b0831..281e62e1 100644 --- a/packages/backend/src/runner.ts +++ b/packages/backend/src/runner.ts @@ -6,6 +6,7 @@ import kill from 'tree-kill' import logger from '@wdio/logger' import { parse as shellParse, quote as shellQuote } from 'shell-quote' import { + RERUN_SLOT, REUSE_ENV, RUNNER_ENV, type RunnerRequestBody @@ -23,7 +24,7 @@ const log = logger('@wdio/devtools-runner') * the user-supplied rerun template can't trigger backtracking regardless of * how many spaces it contains. See CodeQL js/polynomial-redos for context. */ -const NAME_SLOT = '--name "{{testName}}"' +const NAME_SLOT = `--name "${RERUN_SLOT.testName}"` function hasNameTestNameSlot(template: string): boolean { return template.includes(NAME_SLOT) } @@ -184,13 +185,25 @@ class TestRunner { #resolveGenericCommand(payload: RunnerRequestBody): string { const template = payload.rerunCommand const fallback = payload.launchCommand || '' + // Which slot the template carries decides what selects the entry: an exact + // id comes from `uid`, a name pattern from the label. A run with no usable + // selector is not targetable and falls back to relaunching everything. + const usesTestId = Boolean(template?.includes(RERUN_SLOT.testId)) + const name = payload.label || payload.fullTitle || '' const isTargetedRerun = !payload.runAll && (payload.entryType === 'test' || payload.entryType === 'suite') && - Boolean(payload.label || payload.fullTitle) + Boolean(usesTestId ? payload.uid : name) if (!template || !isTargetedRerun) { return fallback || template || '' } + if (usesTestId) { + // Shell-quoted but never regex-escaped: the id is matched literally, and + // a parametrized pytest nodeid can carry brackets and spaces + // (`test_login.py::test_x[a b]`). Split/join rather than a regex so the + // template can't drive backtracking (see the CodeQL note above). + return template.split(RERUN_SLOT.testId).join(shellQuote([payload.uid])) + } // Cucumber's `--name` matches scenario titles, never feature titles. // Feature-level reruns must drop `--name` and pass the .feature path as a // positional arg. The dashboard tags the root suite with @@ -209,10 +222,9 @@ class TestRunner { const stripped = stripNameTestNameSlot(template) return `${stripped} ${shellQuote([featureSpec])}` } - const name = payload.label || payload.fullTitle || '' // The slot is double-quoted in the template, so the backslashes the escape // adds survive shell parsing and reach the runner as literals. - return template.replace(/\{\{testName\}\}/g, escapeFilterRegex(name)) + return template.split(RERUN_SLOT.testName).join(escapeFilterRegex(name)) } #parseGenericCommand(command: string): { file: string; args: string[] } { diff --git a/packages/backend/tests/run-start-broadcast.test.ts b/packages/backend/tests/run-start-broadcast.test.ts new file mode 100644 index 00000000..c1d61b3e --- /dev/null +++ b/packages/backend/tests/run-start-broadcast.test.ts @@ -0,0 +1,98 @@ +/** + * Two unrelated events share the `clearExecutionData` scope: a run STARTING, + * and a single entry resetting inside a run already in flight (Nightwatch + * re-emits a cucumber scenario suite that way). Only the sender can tell them + * apart, so the run route marks its own with `runStart` and the dashboard stops + * inferring intent from the uid — inferring it is what made the second rerun of + * a session keep the first one's actions, console and network rows. + * + * This asserts the flag actually leaves the backend: the app-side fix reads it, + * so dropping it here would silently restore the bug with every app test still + * green. + */ + +import os from 'node:os' +import { afterEach, describe, expect, it, vi } from 'vitest' +import { WebSocket } from 'ws' +import type { FastifyInstance } from 'fastify' +import { WS_PATHS, WS_SCOPE } from '@wdio/devtools-shared' +import { start } from '../src/index.js' +import * as utils from '../src/utils.js' + +vi.mock('../src/utils.js', () => ({ + getDevtoolsApp: vi.fn(), + getCollectorSource: vi.fn() +})) + +const WAIT_TIMEOUT_MS = 2000 + +let server: FastifyInstance | undefined + +afterEach(async () => { + await server?.close() + server = undefined + vi.restoreAllMocks() +}) + +async function bootWithClient(): Promise<{ + server: FastifyInstance + frames: Array<{ scope: string; data: Record }> +}> { + vi.mocked(utils.getDevtoolsApp).mockResolvedValue(os.tmpdir()) + vi.mocked(utils.getCollectorSource).mockResolvedValue('// collector') + const started = await start({ port: 0 }) + server = started.server + + // A real dashboard client, because the flag travels over the socket rather + // than in the POST response — popouts see nothing else. + const socket = new WebSocket( + `ws://localhost:${started.port}${WS_PATHS.client}` + ) + const frames: Array<{ scope: string; data: Record }> = [] + socket.on('message', (raw) => frames.push(JSON.parse(String(raw)))) + await new Promise((resolve, reject) => { + socket.once('open', () => resolve()) + socket.once('error', reject) + }) + return { server: started.server, frames } +} + +async function clearFrame( + frames: Array<{ scope: string; data: Record }> +): Promise> { + const deadline = Date.now() + WAIT_TIMEOUT_MS + while (Date.now() < deadline) { + const found = frames.find( + (frame) => frame.scope === WS_SCOPE.clearExecutionData + ) + if (found) { + return found.data + } + await new Promise((resolve) => setTimeout(resolve, 10)) + } + throw new Error('timed out waiting for a clearExecutionData frame') +} + +describe('a run start announces itself as one', () => { + it('marks the clear it broadcasts with runStart', async () => { + const { server: app, frames } = await bootWithClient() + const { testRunner } = await import('../src/runner.js') + vi.spyOn(testRunner, 'run').mockResolvedValue() + + const response = await app.inject({ + method: 'POST', + url: '/api/tests/run', + payload: { + uid: 'examples/test_login.py', + entryType: 'suite' + } + }) + + expect(response.statusCode).toBe(200) + expect(await clearFrame(frames)).toEqual({ + uid: 'examples/test_login.py', + entryType: 'suite', + runStart: true + }) + }) +}) diff --git a/packages/backend/tests/runner.test.ts b/packages/backend/tests/runner.test.ts index 7c3bf9b5..5e8e23ac 100644 --- a/packages/backend/tests/runner.test.ts +++ b/packages/backend/tests/runner.test.ts @@ -329,6 +329,102 @@ describe('TestRunner', () => { ) }) + // A pytest nodeid is matched LITERALLY, so the escaping the name-pattern + // runners need would turn it into an id no test has. Measured: `pytest + // 'test_thing\.py::test_a'` collects nothing and exits 0, so a broken slot + // reads as a rerun that ran and passed. + describe('exact-id rerun templates', () => { + const pythonRerun = `python3 -m pytest ${'{{testId}}'}` + const spawnedArgs = () => vi.mocked(spawn).mock.calls.at(-1)![1] as string[] + + beforeEach(() => { + vi.mocked(spawn).mockReturnValue(createMockChild()) + }) + + it('substitutes the uid verbatim, never regex-escaped', async () => { + await testRunner.run({ + uid: 'examples/test_login.py::TestLogin::test_valid', + entryType: 'test', + label: 'test_valid', + rerunCommand: pythonRerun + }) + + expect(spawnedArgs()).toEqual([ + '-m', + 'pytest', + 'examples/test_login.py::TestLogin::test_valid' + ]) + }) + + it('keeps a parametrized id with spaces as a single argument', async () => { + await testRunner.run({ + uid: 'test_login.py::test_x[a b]', + entryType: 'test', + label: 'test_x[a b]', + rerunCommand: pythonRerun + }) + + expect(spawnedArgs()).toContain('test_login.py::test_x[a b]') + }) + + it('selects a suite by its id too — a nodeid addresses a file', async () => { + await testRunner.run({ + uid: 'examples/test_login.py', + entryType: 'suite', + label: 'test_login.py', + rerunCommand: pythonRerun + }) + + expect(spawnedArgs()).toContain('examples/test_login.py') + }) + + it('prefers the uid over the label, which is only a display name', async () => { + await testRunner.run({ + uid: 'examples/test_login.py::test_valid', + entryType: 'test', + label: 'logs in with valid credentials', + rerunCommand: pythonRerun + }) + + expect(spawnedArgs()).not.toContain('logs in with valid credentials') + }) + + it('targets on the uid alone, with no display name to fall back on', async () => { + await testRunner.run({ + uid: 'examples/test_login.py::test_valid', + entryType: 'test', + rerunCommand: pythonRerun, + launchCommand: 'python3 -m pytest examples/' + }) + + expect(spawnedArgs()).toContain('examples/test_login.py::test_valid') + }) + + it('falls back to the launch command for a run-all', async () => { + await testRunner.run({ + uid: '__RUN_ALL__', + entryType: 'suite', + runAll: true, + rerunCommand: pythonRerun, + launchCommand: 'python3 -m pytest examples/' + }) + + expect(spawnedArgs()).toEqual(['-m', 'pytest', 'examples/']) + }) + + it('still regex-escapes a name-pattern template', async () => { + await testRunner.run({ + uid: 'some-uid', + entryType: 'test', + label: 'Login (failing)', + rerunCommand: `npx mocha --grep "${'{{testName}}'}"` + }) + + expect(spawnedArgs()).toContain('--grep') + expect(spawnedArgs()).toContain('Login \\(failing\\)') + }) + }) + describe('registerConfigFile', () => { it('uses a worker-registered config path ahead of the default search', async () => { const registered = '/proj/wdio.BUILD.conf.ts' diff --git a/packages/selenium-devtools-py/README.md b/packages/selenium-devtools-py/README.md index f0108258..57c6439b 100644 --- a/packages/selenium-devtools-py/README.md +++ b/packages/selenium-devtools-py/README.md @@ -138,6 +138,58 @@ What still reads as separate runs genuinely is: two independent `pytest` invocations, or a worker started without the environment. Export `DEVTOOLS_RUN_ID` yourself to join such processes into one run. +### Run controls (Run, Rerun, Run-all) + +**All three work, under pytest and for a plain script alike.** A rerun is not a +message to the running process — the backend spawns a fresh one from a command +the adapter publishes at startup, so what the buttons can do is fixed before any +test runs, and the adapter advertises exactly that (a control it cannot service +stays disabled with a reason rather than failing on click). + +Under pytest each control selects what its row names. For a plain script the +tree is one synthetic suite holding one synthetic test — both denote the whole +run, so all three controls relaunch the script, which is what that tree means. + +The rerun reports into **the dashboard you pressed the button in**: the backend +points the process it spawns back at itself (`DEVTOOLS_APP_REUSE` /`_HOST` +/`_PORT`), so the child attaches to that backend and opens no second window. + +The command is your own invocation, re-derived: + +``` +you ran: pytest examples/ -k login -n 4 +run-all: -m pytest /abs/examples -k login -n 4 +one test: -m pytest +``` + +Three things about that are deliberate: + +- **The interpreter is the one running your tests**, not whatever `python3` + resolves to on the backend's PATH — that need not be the venv holding selenium + and this adapter. +- **A single test is selected by nodeid**, so the same slot serves a test, a + class and a file (`file.py::Class::test`, `file.py::Class`, `file.py`). No + filter flag is involved, and nothing is matched by name. +- **Options that narrow the run are dropped from a targeted rerun** — `-k`, + `-m`, `--deselect`, `--lf`/`--ff`/`--sw`, and `-n`/`--dist`. A rerun already + names its test, so a surviving filter could only narrow that further, usually + to nothing — which pytest reports as a clean exit, so it would look like it + worked. The xdist flags go for a second reason: a one-test rerun has nothing + to parallelise, and each worker would connect as its own run. + +The rerun spawns in pytest's **rootdir**, because a nodeid is reported relative +to rootdir while a path argument resolves against the process's directory. If +you launch pytest from somewhere other than its rootdir, an option carrying a +*relative* path (`-c`, `--junitxml`) resolves against rootdir on the rerun; the +positional paths in the run-all command are made absolute for that reason. + +Two limits worth knowing: the directory reaches the backend through the +environment, so a dashboard that was already running when you connected keeps +the directory it was started in; and a rerun under `pytest -n` is issued to a +freshly spawned single process, which is what you want, but the backend has one +worker slot — so with several parallel workers connected the dashboard's state +belongs to whichever connected last. + ## Dashboard window lifecycle Like the JS adapters, `enable()` opens the dashboard in a dedicated, closable @@ -162,6 +214,7 @@ src/selenium_devtools/ screencast.py screenshot-polling recorder + ffmpeg webm encode backend.py launch-or-attach the Node backend + port discovery lifecycle.py dashboard window open/close + shutdown-on-disconnect + rerun.py launch/rerun commands the dashboard's run controls spawn pytest_plugin.py suite/test tree feeder (opt-in) scripts/gen_contract.py regenerate _contract.py from shared (dev-time; also a drift-guard) tests/ stdlib-unittest unit tests (no selenium/pytest needed) @@ -243,7 +296,8 @@ rejects re-uploading an existing version). - **Phase 2 (done)** — BiDi console/network, assertion rows, and screenshot-polling screencast. Not yet: a CDP `Page.startScreencast` push-mode fast-path, per-command screenshots, and performance capture. -- **Phase 3** — trace export, preserve-and-rerun, action snapshots. Per the +- **Phase 3** — trace export, preserve-and-diff, action snapshots. Run controls + (Run / Rerun / Run-all) are done — see above. Per the architecture, the heavy post-processing is a candidate to live server-side in the backend (written once) rather than re-implemented here. diff --git a/packages/selenium-devtools-py/scripts/gen_contract.py b/packages/selenium-devtools-py/scripts/gen_contract.py index ee6fa8e6..5238fb02 100644 --- a/packages/selenium-devtools-py/scripts/gen_contract.py +++ b/packages/selenium-devtools-py/scripts/gen_contract.py @@ -99,6 +99,46 @@ def _run_id_env(runner_ts: str) -> str: return m.group(1) +def _rerun_slot(runner_ts: str) -> dict[str, str]: + """Slots the backend substitutes into a ``rerunCommand``. + + Generated because the adapter WRITES the slot and the backend READS it. A + drifted name is not a type error on either side: the template keeps a + literal ``{{...}}``, the shell hands it to pytest as a file name, and the + rerun fails as a collection error naming a path nobody wrote. + """ + m = re.search( + r"export const RERUN_SLOT = \{(.*?)\n\} as const", runner_ts, re.DOTALL + ) + if not m: + raise SystemExit("could not find `RERUN_SLOT` in shared/runner.ts") + return dict(re.findall(r"(\w+):\s*'([^']+)'", m.group(1))) + + +def _reuse_env(runner_ts: str) -> dict[str, str]: + """Env vars the backend sets on a rerun child to point it at itself. + + Generated because only the backend writes them and only an adapter reads + them. A name that drifts is silent in the worst way: the child launches a + SECOND dashboard and reports into it, so the rerun looks like it worked + while the window the user clicked in stays empty. + """ + m = re.search( + r"export const REUSE_ENV = \{(.*?)\n\} as const", runner_ts, re.DOTALL + ) + if not m: + raise SystemExit("could not find `REUSE_ENV` in shared/runner.ts") + return dict(re.findall(r"(\w+):\s*'([^']+)'", m.group(1))) + + +def _runner_cwd_env(runner_ts: str) -> str: + """The env var naming the directory the backend spawns a rerun in.""" + m = re.search(r"RUNNER_CWD:\s*'([^']+)'", runner_ts) + if not m: + raise SystemExit("could not find `RUNNER_ENV.RUNNER_CWD` in shared/runner.ts") + return m.group(1) + + def _test_runner_ids(types_ts: str) -> list[str]: m = re.search(r"export const TEST_RUNNER_IDS = \[(.*?)\] as const", types_ts, re.DOTALL) if not m: @@ -117,7 +157,11 @@ def main() -> int: routes_ts = (shared / "src" / "routes.ts").read_text() control = _ws_scopes(routes_ts) worker_query = _worker_query(routes_ts) - run_id_env = _run_id_env((shared / "src" / "runner.ts").read_text()) + runner_ts = (shared / "src" / "runner.ts").read_text() + run_id_env = _run_id_env(runner_ts) + rerun_slot = _rerun_slot(runner_ts) + runner_cwd_env = _runner_cwd_env(runner_ts) + reuse_env = _reuse_env(runner_ts) # Drift-guard. missing = [v for v in REQUIRED_DATA_SCOPES.values() if v not in data_keys] @@ -134,6 +178,21 @@ def main() -> int: "socket carries it, and without it every connect reads as a new run." ) + if "testId" not in rerun_slot: + raise SystemExit( + "contract drift: `testId` is no longer a key of shared RERUN_SLOT " + f"(present: {sorted(rerun_slot)}). The rerun template selects by " + "pytest nodeid, and no other slot is substituted verbatim." + ) + + missing_reuse = [k for k in ("REUSE", "HOST", "PORT") if k not in reuse_env] + if missing_reuse: + raise SystemExit( + f"contract drift: REUSE_ENV key(s) {missing_reuse} no longer in " + f"shared (present: {sorted(reuse_env)}). A rerun child needs all " + "three to report into the dashboard that launched it." + ) + if REQUIRED_RUNNER_ID not in runner_ids: raise SystemExit( f"contract drift: runner id {REQUIRED_RUNNER_ID!r} is no longer in " @@ -161,6 +220,13 @@ def main() -> int: f'WORKER_QUERY_RUN_ID = "{worker_query["runId"]}"', f'ENV_RUN_ID = "{run_id_env}"', "", + f'RERUN_SLOT_TEST_ID = "{rerun_slot["testId"]}"', + f'ENV_RUNNER_CWD = "{runner_cwd_env}"', + "", + f'ENV_REUSE = "{reuse_env["REUSE"]}"', + f'ENV_REUSE_HOST = "{reuse_env["HOST"]}"', + f'ENV_REUSE_PORT = "{reuse_env["PORT"]}"', + "", ] out = shared.parent / "selenium-devtools-py" / "src" / "selenium_devtools" / "_contract.py" out.write_text("\n".join(lines)) diff --git a/packages/selenium-devtools-py/src/selenium_devtools/__init__.py b/packages/selenium-devtools-py/src/selenium_devtools/__init__.py index 84b52af8..308957f6 100644 --- a/packages/selenium-devtools-py/src/selenium_devtools/__init__.py +++ b/packages/selenium-devtools-py/src/selenium_devtools/__init__.py @@ -20,7 +20,7 @@ import sys from typing import Optional -from . import backend, instrumentation, lifecycle +from . import backend, instrumentation, lifecycle, rerun from ._contract import CONTRACT_VERSION from .capturer import SessionCapturer from .run_id import reset_run_id @@ -85,6 +85,12 @@ def enable( if _active["capturer"] is not None: return _active["capturer"] + # Before the backend is launched: the directory a rerun spawns in travels + # through the environment the backend process inherits. A framework plugin + # has already published richer commands by now and this leaves those alone. + rerun.configure_script() + rerun.log_published() + process = None try: if host is not None or port is not None: @@ -163,6 +169,9 @@ def disable() -> None: # A new enable() in this process is a NEW run, so the id must not outlive # this one — the backend would otherwise keep the previous run's data. reset_run_id() + # Same reasoning: a re-enable() re-derives its commands rather than + # inheriting the ones this run published. + rerun.reset() process = _active["process"] if process is not None: # only set when we launched it ourselves process.terminate() diff --git a/packages/selenium-devtools-py/src/selenium_devtools/_contract.py b/packages/selenium-devtools-py/src/selenium_devtools/_contract.py index 5b787491..1da2e710 100644 --- a/packages/selenium-devtools-py/src/selenium_devtools/_contract.py +++ b/packages/selenium-devtools-py/src/selenium_devtools/_contract.py @@ -20,3 +20,10 @@ WORKER_QUERY_RUN_ID = "runId" ENV_RUN_ID = "DEVTOOLS_RUN_ID" + +RERUN_SLOT_TEST_ID = "{{testId}}" +ENV_RUNNER_CWD = "DEVTOOLS_RUNNER_CWD" + +ENV_REUSE = "DEVTOOLS_APP_REUSE" +ENV_REUSE_HOST = "DEVTOOLS_APP_HOST" +ENV_REUSE_PORT = "DEVTOOLS_APP_PORT" diff --git a/packages/selenium-devtools-py/src/selenium_devtools/backend.py b/packages/selenium-devtools-py/src/selenium_devtools/backend.py index 9f1bfe17..afecc5af 100644 --- a/packages/selenium-devtools-py/src/selenium_devtools/backend.py +++ b/packages/selenium-devtools-py/src/selenium_devtools/backend.py @@ -4,6 +4,7 @@ the JS adapters do (no cross-ecosystem resolution). So the backend is obtained at runtime, and the resolution order encodes the local-vs-published split: + 0. reuse handshake set → attach to the backend that spawned us (RERUN) 1. DEVTOOLS_PORT set → attach to an already-running backend (CI, manual) 2. DEVTOOLS_BACKEND_CMD set → spawn that explicit command 3. monorepo dist present → node packages/backend/dist/server.js (LOCAL dev) @@ -15,6 +16,7 @@ from __future__ import annotations +import logging import os import re import shlex @@ -25,6 +27,7 @@ from pathlib import Path from typing import List, Optional, Tuple +from ._contract import ENV_REUSE, ENV_REUSE_HOST, ENV_REUSE_PORT from .constants import ( BACKEND_NPM_PACKAGE, BACKEND_NPM_VERSION, @@ -33,8 +36,11 @@ ENV_BACKEND_CMD, ENV_HOST, ENV_PORT, + LOGGER_NAME, ) +_log = logging.getLogger(f"{LOGGER_NAME}.backend") + # Match the ACTUAL bound port from Fastify's "Server listening at http://…:PORT" # line — NOT the earlier "Starting … on port 3000" line, which is only the # *preferred* port. When 3000 is busy the backend negotiates a different port, @@ -92,11 +98,41 @@ def _spawn_and_wait_for_port( raise TimeoutError("backend did not report a port within the timeout") +def reuse_target() -> Optional[Tuple[str, int]]: + """The backend that spawned us, when this process is a rerun child. + + The dashboard's Rerun spawns a fresh process and points it back at itself + through these three variables. Without honouring them the child launches a + SECOND backend and opens a SECOND dashboard window, reporting its run + there — so the window the user pressed Rerun in never updates, which looks + like a rerun that captured nothing. + """ + if os.environ.get(ENV_REUSE) != "1": + return None + host = os.environ.get(ENV_REUSE_HOST) + port = os.environ.get(ENV_REUSE_PORT) + if not host or not port: + return None + try: + return host, int(port) + except ValueError: + _log.warning("ignoring reuse handshake: %s is not a port (%r)", + ENV_REUSE_PORT, port) + return None + + def launch_or_attach() -> Tuple[str, int, Optional[subprocess.Popen]]: """Return ``(host, port, process)``. ``process`` is None when we attached to a backend we don't own (caller must not terminate it).""" host = os.environ.get(ENV_HOST, DEFAULT_HOST) + # Ahead of DEVTOOLS_PORT: this is the backend that asked for this run, so it + # wins over an ambient preference the parent happened to be started with. + reuse = reuse_target() + if reuse is not None: + _log.info("reusing the dashboard that requested this run at %s:%s", *reuse) + return reuse[0], reuse[1], None + if os.environ.get(ENV_PORT): return host, int(os.environ[ENV_PORT]), None diff --git a/packages/selenium-devtools-py/src/selenium_devtools/capturer.py b/packages/selenium-devtools-py/src/selenium_devtools/capturer.py index 9cc65798..5dec496d 100644 --- a/packages/selenium-devtools-py/src/selenium_devtools/capturer.py +++ b/packages/selenium-devtools-py/src/selenium_devtools/capturer.py @@ -11,7 +11,7 @@ import threading from typing import Any, List, Optional, Protocol -from . import frames +from . import frames, rerun from ._contract import ( SCOPE_COMMANDS, SCOPE_CONSOLE_LOGS, @@ -56,7 +56,12 @@ def ensure_metadata( self.session_id = session_id self._tx.send_json( SCOPE_METADATA, - frames.metadata(session_id, to_jsonable(capabilities or {}), url), + frames.metadata( + session_id, + to_jsonable(capabilities or {}), + url, + run_options=rerun.run_options(), + ), ) # ── commands ─────────────────────────────────────────────────────────────── diff --git a/packages/selenium-devtools-py/src/selenium_devtools/constants.py b/packages/selenium-devtools-py/src/selenium_devtools/constants.py index 28ac0cb6..dfb1ddb8 100644 --- a/packages/selenium-devtools-py/src/selenium_devtools/constants.py +++ b/packages/selenium-devtools-py/src/selenium_devtools/constants.py @@ -10,12 +10,12 @@ # shared's TEST_RUNNER_IDS so a rename there fails generation rather than # shipping a value the app narrows away. # -# What the dashboard's Run / Rerun / Run-all controls may offer. The adapter -# sends no rerun or launch command, and the backend's spawn path is the wdio -# binary, so every launch control is refused: absent this the app falls back to -# `DEFAULT_CAPABILITIES` (all true) and the buttons render enabled, then fail on -# click. Revisit with the Preserve-and-Rerun work. -RUN_CAPABILITIES = { +# Every run control refused — what `rerun.py` publishes until it has built a +# command, and what it keeps for a run it cannot address. Sending this matters: +# absent an explicit bag the app falls back to `DEFAULT_CAPABILITIES` (all +# true), so the buttons render enabled and fail on click, and the backend's +# fallback for a rerun it was given no command for is the wdio binary. +RUN_CAPABILITIES_NONE = { "canRunSuites": False, "canRunTests": False, "canRunAll": False, diff --git a/packages/selenium-devtools-py/src/selenium_devtools/frames.py b/packages/selenium-devtools-py/src/selenium_devtools/frames.py index 9afa2bb1..e0bfc561 100644 --- a/packages/selenium-devtools-py/src/selenium_devtools/frames.py +++ b/packages/selenium-devtools-py/src/selenium_devtools/frames.py @@ -11,7 +11,7 @@ from typing import Any, List, Optional from ._contract import RUNNER_ID -from .constants import RUN_CAPABILITIES +from .constants import RUN_CAPABILITIES_NONE from .types import ( CommandLog, ConsoleLog, @@ -28,6 +28,7 @@ def metadata( session_id: str, capabilities: Optional[dict] = None, url: Optional[str] = None, + run_options: Optional[dict] = None, ) -> Metadata: caps = capabilities or {} return { @@ -41,7 +42,11 @@ def metadata( # fact under an older name and is deliberately not sent, so this stream # carries one answer rather than two. "runner": RUNNER_ID, - "options": {"runCapabilities": dict(RUN_CAPABILITIES)}, + # What the dashboard's run controls may offer, plus the commands that + # service them. Built by `rerun.py` and passed in, so this stays pure. + "options": dict(run_options) + if run_options + else {"runCapabilities": dict(RUN_CAPABILITIES_NONE)}, } diff --git a/packages/selenium-devtools-py/src/selenium_devtools/lifecycle.py b/packages/selenium-devtools-py/src/selenium_devtools/lifecycle.py index e0799e06..bfe5402f 100644 --- a/packages/selenium-devtools-py/src/selenium_devtools/lifecycle.py +++ b/packages/selenium-devtools-py/src/selenium_devtools/lifecycle.py @@ -147,14 +147,20 @@ def auto_open_enabled() -> bool: """Whether the dashboard window should auto-open. Default ON, opt-out only. Rule: open unless ``DEVTOOLS_OPEN`` is set to a falsy value - (``0``/``false``/``no``/``off``/empty). This matches the JS adapters, whose - ``openUi`` option defaults true regardless of TTY. + (``0``/``false``/``no``/``off``/empty), or this process is a rerun child — + the window that pressed Rerun is already up and watching the very backend + this run reports to, so a second one would take the focus to show the same + stream. Mirrors the JS adapters, which gate `openUi` on reuse the same way. The previous "default off when stdout isn't a TTY" gate silently disabled auto-open for the common case — running from an IDE or ``python demo.py`` with no attached TTY — so the user opened the URL in their main Chrome instead. CI/headless runs disable it explicitly with ``DEVTOOLS_OPEN=0``. """ + from . import backend # local: keeps module import order free of a cycle + + if backend.reuse_target() is not None: + return False val = os.environ.get(ENV_OPEN) if val is None: return True diff --git a/packages/selenium-devtools-py/src/selenium_devtools/pytest_plugin.py b/packages/selenium-devtools-py/src/selenium_devtools/pytest_plugin.py index cc25f549..b98eb774 100644 --- a/packages/selenium-devtools-py/src/selenium_devtools/pytest_plugin.py +++ b/packages/selenium-devtools-py/src/selenium_devtools/pytest_plugin.py @@ -14,7 +14,7 @@ from typing import Dict, Optional import selenium_devtools as devtools -from . import assertions, frames +from . import assertions, frames, rerun from .constants import ENV_OPT_IN, ENV_PORT, LOGGER_NAME from .utils import now_ms @@ -60,10 +60,6 @@ def __init__(self) -> None: self._classes: Dict[str, dict] = {} self._tests: Dict[str, dict] = {} self._starts: Dict[str, int] = {} - # Collection index per nodeid. pytest runs in collection order and - # interleaves module-level tests with classes, which the tree's default - # (a suite's own tests, then its nested suites) cannot express. - self._order: Dict[str, int] = {} def mark_start(self, nodeid: str) -> None: self._starts.setdefault(nodeid, now_ms()) @@ -76,7 +72,6 @@ def record( line: int, state: str, abs_file: Optional[str] = None, - order: Optional[int] = None, ) -> None: # `file` is pytest's rootdir-relative path: it prefixes every nodeid, so # it stays the grouping key and the readable title. `abs_file` is the @@ -85,9 +80,16 @@ def record( # app pairs the two by exact path, so a relative one leaves the Source # tab reporting the file as never captured the moment a test is picked. path = abs_file or file - if order is not None: - self._order[nodeid] = order - position = self._order.get(nodeid) + # Sibling position is the SOURCE LINE, never this run's collection + # index. pytest runs in collection order and interleaves module-level + # tests with classes, which the tree's default (a suite's own tests, + # then its nested suites) cannot express — but an index describes only + # the collection it came from. A rerun collects one test, so its index + # is 0 and the row jumps to the top of its file, taking the class suite + # it used to sit below with it. A line is a property of the test, so it + # survives a partial collection; within a module pytest collects in + # definition order, so the two agree wherever both are meaningful. + position = line # `_starts` is only populated once the test actually starts, so a test # recorded at COLLECTION does not freeze a start time it never had. start = self._starts.get(nodeid, now_ms()) @@ -225,6 +227,25 @@ def _forget_ini_cache(config) -> None: # noqa: ANN001 cache.pop("enable_assertion_pass_hook", None) +def _configure_rerun(config, rootdir: Optional[str]) -> None: # noqa: ANN001 + """Hand pytest's own view of its invocation to the rerun builder. + + `invocation_params.args` is the raw argument list and `config.args` the + file/dir/nodeid arguments pytest resolved out of it — asking pytest which + were positional is what makes stripping them safe, since inferring it needs + a table of every option that takes a value. Both are read here, before + `enable()`, because the directory a rerun spawns in reaches the backend + through the environment its process inherits. + """ + params = getattr(config, "invocation_params", None) + args = list(getattr(params, "args", ()) or ()) + rerun.configure_pytest( + args=args, + positionals=list(getattr(config, "args", ()) or ()), + rootdir=rootdir, + ) + + def pytest_configure(config) -> None: # noqa: ANN001 if _opted_in(): _enable_assertion_pass_hook(config) @@ -232,6 +253,7 @@ def pytest_configure(config) -> None: # noqa: ANN001 # `rootpath` on pytest 7+, `rootdir` before it. root = getattr(config, "rootpath", None) or getattr(config, "rootdir", None) _rootdir = str(root) if root else None + _configure_rerun(config, _rootdir) capturer = devtools.enable() # pytest owns the suite tree — suppress the adapter's default script suite. from . import instrumentation @@ -261,11 +283,11 @@ def pytest_collection_finish(session) -> None: # noqa: ANN001 if capturer is None: return items = getattr(session, "items", []) or [] - for index, item in enumerate(items): + for item in items: file, line, name = item.location _registry.record( item.nodeid, file, name, line or 0, "pending", - abs_file=_absolute(file), order=index, + abs_file=_absolute(file), ) _publish(capturer) _report_passing_assertion_support(items) diff --git a/packages/selenium-devtools-py/src/selenium_devtools/rerun.py b/packages/selenium-devtools-py/src/selenium_devtools/rerun.py new file mode 100644 index 00000000..99d43bba --- /dev/null +++ b/packages/selenium-devtools-py/src/selenium_devtools/rerun.py @@ -0,0 +1,275 @@ +"""The commands the dashboard's Run and Rerun controls spawn. + +A rerun is not a message to this process. The dashboard POSTs to +``/api/tests/run`` and the backend execs a command string the adapter published +in its metadata, in a FRESH process — so the whole feature is decided here, +before any test runs, and shipped once: + +``launchCommand`` + Re-runs everything. Services the header's Run-all, and is what the backend + falls back to for anything it cannot target. +``rerunCommand`` + The same command narrowed to one entry, ending in the slot the backend + fills with that entry's uid. + +A pytest entry's uid is its nodeid, and a nodeid addresses a file, a class or a +single test in one syntax (``file.py``, ``file.py::Class``, +``file.py::Class::test``). That is why one slot covers every row the tree can +offer, where the name-pattern runners need a filter flag per level and a +special case for the ones whose flag matches only leaves. + +A plain script cannot address a part of itself, but it does not need to: its +tree is one synthetic suite holding one synthetic test, and both denote the +whole run, so the launch command doubles as the rerun template and every +control is serviceable. Capabilities are DERIVED from which commands were built +rather than declared: a control the adapter advertises but cannot service is +worse than one it never offered, because the backend's fallback for a command +it wasn't given is the wdio binary. + +Where the rerun REPORTS is `backend.reuse_target()`'s job, not this module's — +the backend points the child it spawns back at itself. +""" + +from __future__ import annotations + +import logging +import os +import shlex +import sys +from typing import Dict, List, Optional, Sequence + +from ._contract import ENV_RUNNER_CWD, RERUN_SLOT_TEST_ID +from .constants import LOGGER_NAME, RUN_CAPABILITIES_NONE + +_log = logging.getLogger(f"{LOGGER_NAME}.rerun") + +# Options that narrow which tests run. A targeted rerun names its entry +# outright, so any of these left in place can only narrow that further — `-k +# login` alongside an explicit nodeid for `test_logout` selects nothing at all, +# and pytest reports that as a clean exit, so the rerun looks like it worked. +# +# The xdist ones go for a second reason: a one-entry rerun has nothing to +# parallelise, and each worker it spawned would connect to the dashboard under +# its own run id (see run_id.py), so one rerun would report as several runs. +_VALUE_FILTERS = ("-k", "-m", "--deselect", "-n", "--numprocesses", "--dist") +_FLAG_FILTERS = ( + "--lf", + "--last-failed", + "--ff", + "--failed-first", + "--nf", + "--new-first", + "--sw", + "--stepwise", + "--stepwise-skip", +) +_SHORT_VALUE_FILTERS = tuple(opt for opt in _VALUE_FILTERS if len(opt) == 2) + +_options: Dict[str, object] = {"runCapabilities": dict(RUN_CAPABILITIES_NONE)} + +# The value we last wrote to ENV_RUNNER_CWD, or None if we never wrote one. +# The VALUE and not a "we wrote it once" flag, because two different things +# have to be told apart at the second run: our own leftover, which must be +# replaced, and a value the caller put there since, which must be respected. +# A flag says only that we wrote at some point and cannot distinguish them. +# Deliberately not cleared by `reset()` — what this process wrote survives a +# disable/enable, and forgetting it would read our leftover as an instruction. +_stamped_runner_cwd: Optional[str] = None + + +def run_options() -> Dict[str, object]: + """The `options` bag the metadata frame carries.""" + return dict(_options) + + +def reset() -> None: + """Forget what was configured — a re-`enable()` reconfigures from scratch.""" + global _options + _options = {"runCapabilities": dict(RUN_CAPABILITIES_NONE)} + + +def _reset_for_tests() -> None: + """Reset module state between unit tests (never used in production). + + Includes the ENV_RUNNER_CWD stamp, which a real process keeps for its + lifetime and `reset()` therefore leaves alone — a test needs the + fresh-process state instead. + """ + global _stamped_runner_cwd + reset() + _stamped_runner_cwd = None + + +def configure_script(argv: Optional[Sequence[str]] = None) -> None: + """Publish the relaunch command for a plain script. + + A no-op once anything is published: a framework plugin configures before + the adapter is enabled and knows how to address one test, which this + cannot. So this is the floor, never an override. + """ + if _options.get("launchCommand") or _options.get("rerunCommand"): + return + args = list(argv if argv is not None else sys.argv) + script = args[0] if args else "" + # No script to relaunch: `python -c ...` reports the flag itself here and an + # interactive session reports nothing. Publishing either would advertise a + # Run-all that cannot work, which is worse than leaving it refused. + if not script or not os.path.isfile(script): + return + launch = _quote([sys.executable, os.path.abspath(script), *args[1:]]) + # The same command services a row-scoped rerun, because a script's tree is + # one synthetic suite holding one synthetic test — both denote the whole + # run, so relaunching it IS running that row. Refusing those two controls + # would leave a disabled button beside the only row the tree has. The + # template carries no slot, so the backend substitutes nothing into it. + _publish(launch=launch, rerun=launch, base_dir=os.getcwd()) + + +def configure_pytest( + *, args: Sequence[str], positionals: Sequence[str], rootdir: Optional[str] +) -> None: + """Publish both commands for a pytest run. + + `args` is pytest's own argument list (no program name), `positionals` the + file/dir/nodeid arguments pytest resolved out of it, and `rootdir` the + directory its nodeids are relative to. + """ + launch = _quote( + [sys.executable, "-m", "pytest", *_absolute_positionals(args, positionals)] + ) + targeted = _drop_positionals(_strip_filters(args), positionals) + rerun = ( + f"{_quote([sys.executable, '-m', 'pytest', *targeted])} {RERUN_SLOT_TEST_ID}" + ) + _publish(launch=launch, rerun=rerun, base_dir=rootdir or os.getcwd()) + + +def _publish(*, launch: Optional[str], rerun: Optional[str], base_dir: str) -> None: + global _options + options: Dict[str, object] = { + "runCapabilities": { + "canRunSuites": bool(rerun), + "canRunTests": bool(rerun), + "canRunAll": bool(launch), + } + } + if launch: + options["launchCommand"] = launch + if rerun: + options["rerunCommand"] = rerun + _options = options + _stamp_runner_cwd(base_dir) + + +def _stamp_runner_cwd(base_dir: str) -> None: + """Name the directory the backend spawns a rerun in. + + It has to be the ROOTDIR for pytest, not the invocation directory: a nodeid + is reported relative to rootdir but a positional path is resolved relative + to the process's cwd, so the two only agree when the run was launched from + rootdir. Spawning elsewhere makes every nodeid a path that does not exist. + + The backend reads this from its own environment, so it only lands if we own + that process — an already-running dashboard keeps the cwd it was started in. + + Still holding what we wrote, it is REPLACED: a second `enable()` in one + process is a second run, and keeping the first one's directory would spawn + its reruns in the wrong project, where every nodeid names a path that does + not exist. Holding anything else, it is left alone — an explicit + `DEVTOOLS_RUNNER_CWD` is an instruction, whenever it was exported. + """ + global _stamped_runner_cwd + current = os.environ.get(ENV_RUNNER_CWD) + if current is not None and current != _stamped_runner_cwd: + return + _stamped_runner_cwd = os.path.abspath(base_dir) + os.environ[ENV_RUNNER_CWD] = _stamped_runner_cwd + + +def _strip_filters(args: Sequence[str]) -> List[str]: + """Drop the options that narrow a run to a subset of its tests. + + Attached and `=` forms are matched by prefix (`-kexpr`, `--deselect=x`). + Clustered short options (`-xk expr`) are not recognised — argparse allows + them but nothing writes them, and a partial match would strip a flag the + run needs. + """ + kept: List[str] = [] + skip_value = False + for arg in args: + if skip_value: + skip_value = False + continue + if arg in _FLAG_FILTERS: + continue + if arg in _VALUE_FILTERS: + skip_value = True + continue + if any(arg.startswith(f"{opt}=") for opt in _VALUE_FILTERS): + continue + if any( + arg.startswith(opt) and len(arg) > len(opt) for opt in _SHORT_VALUE_FILTERS + ): + continue + kept.append(arg) + return kept + + +def _drop_positionals( + args: Sequence[str], positionals: Sequence[str] +) -> List[str]: + """Drop the file/dir/nodeid arguments the run was launched with. + + pytest UNIONS its positional selectors, so one left in place is selected in + addition to the rerun's own target — the rerun would run the whole original + selection again, and a second rerun would stack another selector on top. + + Which arguments are positional is asked of pytest rather than inferred: + guessing needs a table of every option that takes a value, and dropping a + value while keeping its option makes that option swallow the id we append. + """ + drop = set(positionals) + return [arg for arg in args if arg not in drop] + + +def _absolute_positionals( + args: Sequence[str], positionals: Sequence[str] +) -> List[str]: + """Absolutise the positional paths, leaving every other argument alone. + + The launch command is spawned from rootdir (see `_stamp_runner_cwd`) while + its paths were written relative to wherever the run was launched. Only the + positionals are rewritten — an option's relative path argument is left as + written, and resolves only if those two directories agree. + """ + targets = set(positionals) + return [_absolute_target(arg) if arg in targets else arg for arg in args] + + +def _absolute_target(arg: str) -> str: + """Absolutise the path half of a positional, keeping any `::` selector.""" + path, sep, selector = arg.partition("::") + return f"{os.path.abspath(path)}{sep}{selector}" + + +def _quote(args: Sequence[str]) -> str: + return " ".join(shlex.quote(arg) for arg in args) + + +def log_published() -> None: + """Say what the dashboard's run controls will do, once, at bringup. + + Worth a line because the alternative way to find out is to click a button + and watch nothing happen. + """ + caps = _options.get("runCapabilities", {}) + targeted = isinstance(caps, dict) and caps.get("canRunTests") + if targeted: + _log.info("dashboard can rerun a single test, a suite, or the whole run") + elif isinstance(caps, dict) and caps.get("canRunAll"): + _log.info( + "dashboard can rerun the whole run; single tests are not " + "addressable outside a test framework" + ) + else: + _log.info("dashboard run controls are unavailable for this invocation") diff --git a/packages/selenium-devtools-py/tests/test_backend.py b/packages/selenium-devtools-py/tests/test_backend.py index 9ee79e1c..aafc6afa 100644 --- a/packages/selenium-devtools-py/tests/test_backend.py +++ b/packages/selenium-devtools-py/tests/test_backend.py @@ -4,6 +4,7 @@ from pathlib import Path from selenium_devtools import backend +from selenium_devtools._contract import ENV_REUSE, ENV_REUSE_HOST, ENV_REUSE_PORT class TestBackendResolution(unittest.TestCase): @@ -67,5 +68,71 @@ def test_pinned_backend_version_is_set(self): self.assertRegex(backend.BACKEND_NPM_VERSION, r"^\d+\.\d+\.\d+$") +class TestRerunChildReportsIntoTheDashboardThatAskedForIt(unittest.TestCase): + """A rerun is a fresh process the backend spawns, pointed back at itself. + + Ignoring that handshake is silent in the worst way: the child launches a + second backend and reports its run there, so the window the user pressed + Rerun in never updates. + """ + + KEYS = (ENV_REUSE, ENV_REUSE_HOST, ENV_REUSE_PORT, "DEVTOOLS_PORT", + "DEVTOOLS_HOST", "DEVTOOLS_BACKEND_CMD") + + def setUp(self): + self._saved = {k: os.environ.get(k) for k in self.KEYS} + for k in self.KEYS: + os.environ.pop(k, None) + + def tearDown(self): + for k, v in self._saved.items(): + if v is None: + os.environ.pop(k, None) + else: + os.environ[k] = v + + def _handshake(self, host="127.0.0.1", port="5599"): + os.environ[ENV_REUSE] = "1" + os.environ[ENV_REUSE_HOST] = host + os.environ[ENV_REUSE_PORT] = port + + def test_it_attaches_to_the_inherited_backend_without_spawning(self): + self._handshake() + + host, port, proc = backend.launch_or_attach() + + self.assertEqual((host, port), ("127.0.0.1", 5599)) + self.assertIsNone(proc) # attached, so teardown must not kill it + + def test_the_handshake_wins_over_an_ambient_port_preference(self): + # DEVTOOLS_PORT is inherited from the parent's environment, but the + # backend that requested this run is the one it must report to. + os.environ["DEVTOOLS_PORT"] = "4321" + self._handshake(port="5599") + + _, port, _ = backend.launch_or_attach() + + self.assertEqual(port, 5599) + + def test_no_handshake_means_no_reuse(self): + self.assertIsNone(backend.reuse_target()) + + def test_a_partial_or_malformed_handshake_is_ignored(self): + for label, env in ( + ("no host", {ENV_REUSE: "1", ENV_REUSE_PORT: "5599"}), + ("no port", {ENV_REUSE: "1", ENV_REUSE_HOST: "127.0.0.1"}), + ("flag off", {ENV_REUSE: "0", ENV_REUSE_HOST: "h", ENV_REUSE_PORT: "1"}), + ("port not a number", + {ENV_REUSE: "1", ENV_REUSE_HOST: "h", ENV_REUSE_PORT: "later"}), + ): + with self.subTest(label): + for k in (ENV_REUSE, ENV_REUSE_HOST, ENV_REUSE_PORT): + os.environ.pop(k, None) + os.environ.update(env) + + # Degrades to launching its own backend rather than raising. + self.assertIsNone(backend.reuse_target()) + + if __name__ == "__main__": unittest.main() diff --git a/packages/selenium-devtools-py/tests/test_lifecycle.py b/packages/selenium-devtools-py/tests/test_lifecycle.py index 133ed917..36b5f9d9 100644 --- a/packages/selenium-devtools-py/tests/test_lifecycle.py +++ b/packages/selenium-devtools-py/tests/test_lifecycle.py @@ -10,6 +10,8 @@ import unittest from unittest import mock +from selenium_devtools._contract import ENV_REUSE, ENV_REUSE_HOST, ENV_REUSE_PORT + from selenium_devtools import lifecycle from selenium_devtools.lifecycle import BrowserHandle @@ -120,16 +122,24 @@ def test_chrome_not_found_returns_empty_handle_without_spawning(self): handle.close() # empty handle stays safe to close +_REUSE_KEYS = (ENV_REUSE, ENV_REUSE_HOST, ENV_REUSE_PORT) + + class TestAutoOpenEnabled(unittest.TestCase): def setUp(self): - self._saved = os.environ.get(lifecycle.ENV_OPEN) - os.environ.pop(lifecycle.ENV_OPEN, None) + # The reuse handshake also decides this, so clear it — otherwise these + # cases would answer for the wrong reason inside a rerun child. + keys = (lifecycle.ENV_OPEN, *_REUSE_KEYS) + self._saved = {k: os.environ.get(k) for k in keys} + for k in keys: + os.environ.pop(k, None) def tearDown(self): - if self._saved is None: - os.environ.pop(lifecycle.ENV_OPEN, None) - else: - os.environ[lifecycle.ENV_OPEN] = self._saved + for k, v in self._saved.items(): + if v is None: + os.environ.pop(k, None) + else: + os.environ[k] = v def test_env_falsy_disables(self): for val in ("0", "false", "no", "off", ""): @@ -152,6 +162,33 @@ def test_defaults_on_when_unset(self): with mock.patch.object(sys.stdout, "isatty", return_value=True): self.assertTrue(lifecycle.auto_open_enabled()) + def test_a_rerun_child_opens_no_second_window(self): + # The window that pressed Rerun is already up and watching the backend + # this run reports to. Opening another takes the focus to show the same + # stream — and it was the visible symptom of ignoring the handshake. + os.environ[ENV_REUSE] = "1" + os.environ[ENV_REUSE_HOST] = "127.0.0.1" + os.environ[ENV_REUSE_PORT] = "5599" + + self.assertFalse(lifecycle.auto_open_enabled()) + + def test_an_explicit_open_does_not_override_the_handshake(self): + # DEVTOOLS_OPEN is inherited from the parent run, so it says nothing + # about whether THIS process should open a window. + os.environ[lifecycle.ENV_OPEN] = "1" + os.environ[ENV_REUSE] = "1" + os.environ[ENV_REUSE_HOST] = "127.0.0.1" + os.environ[ENV_REUSE_PORT] = "5599" + + self.assertFalse(lifecycle.auto_open_enabled()) + + def test_an_incomplete_handshake_still_opens(self): + # No usable target means this process launches its own backend, and + # then a window is the only way to see it. + os.environ[ENV_REUSE] = "1" + + self.assertTrue(lifecycle.auto_open_enabled()) + class TestShutdownFlow(unittest.TestCase): def setUp(self): diff --git a/packages/selenium-devtools-py/tests/test_pytest_plugin.py b/packages/selenium-devtools-py/tests/test_pytest_plugin.py index 57174ce5..4f7a351c 100644 --- a/packages/selenium-devtools-py/tests/test_pytest_plugin.py +++ b/packages/selenium-devtools-py/tests/test_pytest_plugin.py @@ -210,45 +210,61 @@ def test_a_failure_still_wins_over_pending_siblings(self): self.assertEqual(reg.snapshot()[0]["state"], "failed") -class TestCollectionOrderIsCarried(unittest.TestCase): +class TestSiblingPositionIsCarried(unittest.TestCase): """The tree renders a suite's own tests then its nested suites, which IS mocha's execution order (`Runner.runSuite` runs `suite.tests` first). pytest - runs in collection order and interleaves the two, so it stamps `order` and - the app merges the buckets by it.""" - - def _file_suite(self): + interleaves the two, so it stamps `order` and the app merges the buckets by + it. That position is the source LINE — see the comment in `record`.""" + + # A class holding two tests, and a module-level test written after it. + CLASS_TESTS = ( + (f"{FILE}::TestLogin::test_valid", "TestLogin.test_valid", 10), + (f"{FILE}::TestLogin::test_invalid", "TestLogin.test_invalid", 20), + ) + MODULE_TEST = (f"{FILE}::test_the_login_page_loads", + "test_the_login_page_loads", 40) + + def _full_collection(self): reg = _SuiteRegistry() - for index, (nid, name) in enumerate(( - (f"{FILE}::TestLogin::test_valid", "TestLogin.test_valid"), - (f"{FILE}::TestLogin::test_invalid", "TestLogin.test_invalid"), - (f"{FILE}::test_the_login_page_loads", "test_the_login_page_loads"), - )): - reg.record(nid, FILE, name, 0, "pending", order=index) - return reg.snapshot()[0] + for nid, name, line in (*self.CLASS_TESTS, self.MODULE_TEST): + reg.record(nid, FILE, name, line, "pending") + return reg def test_a_class_sits_where_its_first_test_starts(self): - file_suite = self._file_suite() + file_suite = self._full_collection().snapshot()[0] - self.assertEqual(file_suite["suites"][0]["order"], 0) - self.assertEqual(file_suite["tests"][0]["order"], 2) + self.assertEqual(file_suite["suites"][0]["order"], 10) + self.assertEqual(file_suite["tests"][0]["order"], 40) - def test_the_declared_order_survives_a_later_state_change(self): - # record() runs again at start and at completion; the collection index - # is assigned once and must not be lost. + def test_the_position_survives_a_later_state_change(self): + # record() runs again at start and at completion; the position must not + # move as the state settles. reg = _SuiteRegistry() nid = f"{FILE}::test_plain" - reg.record(nid, FILE, "test_plain", 0, "pending", order=7) - reg.record(nid, FILE, "test_plain", 0, "passed") + reg.record(nid, FILE, "test_plain", 7, "pending") + reg.record(nid, FILE, "test_plain", 7, "passed") self.assertEqual(reg.snapshot()[0]["tests"][0]["order"], 7) - def test_no_order_is_stamped_when_none_was_given(self): - # The plain-script path and any caller that does not know an order must - # leave the frame exactly as it was. - reg = _SuiteRegistry() - reg.record(f"{FILE}::test_plain", FILE, "test_plain", 0, "passed") + def test_a_rerun_of_one_test_leaves_it_where_it_was(self): + # A rerun is a fresh process collecting ONE test, so its collection + # index is 0 — stamping that would send the row to the top of its file, + # above the class it was written below. The line does not move. + full = self._full_collection().snapshot()[0] - self.assertNotIn("order", reg.snapshot()[0]["tests"][0]) + rerun = _SuiteRegistry() + nid, name, line = self.MODULE_TEST + rerun.record(nid, FILE, name, line, "passed") + + self.assertEqual( + rerun.snapshot()[0]["tests"][0]["order"], + full["tests"][0]["order"], + ) + # …and still after the class it sits below. + self.assertGreater( + rerun.snapshot()[0]["tests"][0]["order"], + full["suites"][0]["order"], + ) class _Report: diff --git a/packages/selenium-devtools-py/tests/test_rerun.py b/packages/selenium-devtools-py/tests/test_rerun.py new file mode 100644 index 00000000..ea08904f --- /dev/null +++ b/packages/selenium-devtools-py/tests/test_rerun.py @@ -0,0 +1,315 @@ +"""The commands the dashboard's run controls spawn. + +Every case here is a string the BACKEND will hand to a shell in a fresh +process, so the assertions are about the exact command text — there is no +runtime on our side to correct a template that came out wrong. +""" + +import os +import sys +import unittest + +from selenium_devtools import rerun +from selenium_devtools._contract import ENV_RUNNER_CWD, RERUN_SLOT_TEST_ID + +PY = rerun._quote([sys.executable]) +ROOT = "/repo" +# A path that really exists — `configure_script` refuses anything that is not a +# file, so a made-up path would make those cases pass for the wrong reason. +SCRIPT = os.path.abspath(__file__) + + +class RerunTestCase(unittest.TestCase): + def setUp(self) -> None: + # `_reset_for_tests`, not `reset`: ownership of ENV_RUNNER_CWD outlives + # a reset by design, so each case has to start as a fresh process would. + rerun._reset_for_tests() + self._saved_cwd = os.environ.pop(ENV_RUNNER_CWD, None) + self.addCleanup(self._restore) + + def _restore(self) -> None: + rerun._reset_for_tests() + os.environ.pop(ENV_RUNNER_CWD, None) + if self._saved_cwd is not None: + os.environ[ENV_RUNNER_CWD] = self._saved_cwd + + def configure_pytest(self, args, positionals=(), rootdir=ROOT): + rerun.configure_pytest( + args=list(args), positionals=list(positionals), rootdir=rootdir + ) + return rerun.run_options() + + +class TestPytestCommands(RerunTestCase): + def test_the_rerun_template_ends_in_the_slot_the_backend_fills(self): + options = self.configure_pytest(["tests/test_a.py"], ["tests/test_a.py"]) + + # Bare, not quoted: the backend shell-quotes the id it substitutes, and + # a pre-quoted slot would nest the quotes and select nothing. + self.assertEqual( + options["rerunCommand"], f"{PY} -m pytest {RERUN_SLOT_TEST_ID}" + ) + + def test_the_launch_command_keeps_the_whole_selection(self): + options = self.configure_pytest(["tests/"], ["tests/"]) + + self.assertEqual( + options["launchCommand"], + f"{PY} -m pytest {os.path.abspath('tests/')}", + ) + + def test_the_interpreter_is_this_one_not_whatever_python_resolves_to(self): + # The rerun must run under the venv that has selenium and this adapter + # installed. `python3` off the backend's PATH need not be that one. + options = self.configure_pytest(["tests/"], ["tests/"]) + + self.assertTrue(options["launchCommand"].startswith(PY)) + self.assertTrue(str(options["rerunCommand"]).startswith(PY)) + + def test_a_nodeid_positional_is_dropped_so_reruns_do_not_stack(self): + # The state a rerun's own child process is in: its argv carries the + # nodeid the previous rerun selected. Left in place, pytest would union + # it with the next one and each rerun would run one test more. + options = self.configure_pytest( + ["tests/test_a.py::TestLogin::test_valid"], + ["tests/test_a.py::TestLogin::test_valid"], + ) + + self.assertEqual( + options["rerunCommand"], f"{PY} -m pytest {RERUN_SLOT_TEST_ID}" + ) + + def test_options_that_do_not_select_tests_survive(self): + options = self.configure_pytest( + ["-p", "no:cacheprovider", "-x", "--maxfail=2", "tests/"], ["tests/"] + ) + + self.assertEqual( + options["rerunCommand"], + f"{PY} -m pytest -p no:cacheprovider -x --maxfail=2 " + f"{RERUN_SLOT_TEST_ID}", + ) + + def test_an_options_value_is_never_mistaken_for_a_positional(self): + # `no:cacheprovider` is not a path, and pytest does not report it as a + # positional — dropping it while keeping `-p` would make `-p` swallow + # the id appended after it. + options = self.configure_pytest(["-p", "no:cacheprovider"], []) + + self.assertIn("-p no:cacheprovider", str(options["rerunCommand"])) + + +class TestSelectorsAreStripped(RerunTestCase): + """A targeted rerun names its entry, so anything else that narrows the run + can only narrow it further — usually to nothing, which pytest reports as a + clean exit.""" + + def assert_stripped(self, args): + options = self.configure_pytest([*args, "tests/"], ["tests/"]) + self.assertEqual( + options["rerunCommand"], + f"{PY} -m pytest {RERUN_SLOT_TEST_ID}", + f"{args} survived into the rerun template", + ) + # …but the launch command re-runs what the user asked for. + self.assertIn(args[0], str(options["launchCommand"])) + + def test_keyword_and_marker_filters(self): + for args in (["-k", "login"], ["-m", "smoke"], ["-kloGin"], ["-msmoke"]): + with self.subTest(args=args): + self.assert_stripped(args) + + def test_deselect_in_both_forms(self): + for args in ( + ["--deselect", "tests/test_a.py::test_x"], + ["--deselect=tests/test_a.py::test_x"], + ): + with self.subTest(args=args): + self.assert_stripped(args) + + def test_last_failed_and_stepwise_flags(self): + for flag in ("--lf", "--last-failed", "--ff", "--sw", "--stepwise", "--nf"): + with self.subTest(flag=flag): + self.assert_stripped([flag]) + + def test_xdist_parallelism(self): + # A one-entry rerun has nothing to parallelise, and each worker would + # connect to the dashboard as its own run. + for args in (["-n", "4"], ["-nauto"], ["--numprocesses=4"], ["--dist", "load"]): + with self.subTest(args=args): + self.assert_stripped(args) + + +class TestCapabilitiesMatchWhatCanBeServiced(RerunTestCase): + def test_pytest_can_service_every_control(self): + options = self.configure_pytest(["tests/"], ["tests/"]) + + self.assertEqual( + options["runCapabilities"], + {"canRunSuites": True, "canRunTests": True, "canRunAll": True}, + ) + + def test_a_plain_script_can_rerun_its_one_entry(self): + # A script's tree is one synthetic suite holding one synthetic test, and + # both denote the whole run — so relaunching the script IS running that + # row, and refusing those controls would disable the button beside the + # only row there is. + rerun.configure_script([SCRIPT]) + options = rerun.run_options() + + self.assertEqual( + options["runCapabilities"], + {"canRunSuites": True, "canRunTests": True, "canRunAll": True}, + ) + self.assertEqual( + options["launchCommand"], f"{PY} {rerun._quote([SCRIPT])}" + ) + + def test_a_script_rerun_carries_no_slot_to_substitute(self): + # There is nothing to select, so the template is the launch command + # itself and the backend substitutes nothing into it. + rerun.configure_script([SCRIPT]) + options = rerun.run_options() + + self.assertEqual(options["rerunCommand"], options["launchCommand"]) + self.assertNotIn(RERUN_SLOT_TEST_ID, str(options["rerunCommand"])) + + def test_a_script_that_is_not_a_file_publishes_nothing(self): + # `python -c '...'` reports the flag itself as argv[0], and an + # interactive session reports nothing. Either would advertise a Run-all + # that reruns a path that does not exist. + for argv in (["-c"], [""], []): + with self.subTest(argv=argv): + rerun.reset() + rerun.configure_script(argv) + + self.assertNotIn("launchCommand", rerun.run_options()) + self.assertFalse( + any(rerun.run_options()["runCapabilities"].values()) + ) + + def test_nothing_configured_refuses_everything(self): + # The default has to be all-off: with no bag at all the app falls back + # to all-true and the buttons render enabled. + self.assertEqual( + rerun.run_options(), + { + "runCapabilities": { + "canRunSuites": False, + "canRunTests": False, + "canRunAll": False, + } + }, + ) + + def test_a_script_launch_never_overwrites_a_framework_one(self): + # `enable()` calls configure_script unconditionally, and for a pytest + # run the plugin has already published the better commands. The script + # argv here is a real file, so only the already-published check can be + # what stops it. + self.configure_pytest(["tests/"], ["tests/"]) + rerun.configure_script([SCRIPT]) + + self.assertIn("rerunCommand", rerun.run_options()) + self.assertTrue(rerun.run_options()["runCapabilities"]["canRunTests"]) + + def test_reset_refuses_everything_again(self): + self.configure_pytest(["tests/"], ["tests/"]) + rerun.reset() + + self.assertNotIn("launchCommand", rerun.run_options()) + + +class TestTheDirectoryTheRerunSpawnsIn(RerunTestCase): + def test_pytest_spawns_in_rootdir_because_nodeids_are_relative_to_it(self): + self.configure_pytest(["tests/"], ["tests/"], rootdir=ROOT) + + self.assertEqual(os.environ[ENV_RUNNER_CWD], ROOT) + + def test_a_script_spawns_where_it_was_launched(self): + rerun.configure_script([SCRIPT]) + + self.assertEqual(os.environ[ENV_RUNNER_CWD], os.getcwd()) + + def test_an_explicit_setting_is_left_alone(self): + os.environ[ENV_RUNNER_CWD] = "/somewhere/else" + + self.configure_pytest(["tests/"], ["tests/"]) + + self.assertEqual(os.environ[ENV_RUNNER_CWD], "/somewhere/else") + + def test_a_second_run_in_one_process_restamps_the_directory(self): + # disable() + enable() again, for a project somewhere else. Keeping the + # first run's directory would spawn its reruns in the wrong project, + # where every nodeid names a path that does not exist. + self.configure_pytest(["tests/"], ["tests/"], rootdir="/repo-a") + rerun.reset() # what disable() does + self.configure_pytest(["tests/"], ["tests/"], rootdir="/repo-b") + + self.assertEqual(os.environ[ENV_RUNNER_CWD], "/repo-b") + + def test_an_explicit_setting_survives_a_second_run_too(self): + # The replacement above must not turn into ownership of a value someone + # else set: an explicit DEVTOOLS_RUNNER_CWD is an instruction. + os.environ[ENV_RUNNER_CWD] = "/somewhere/else" + + self.configure_pytest(["tests/"], ["tests/"], rootdir="/repo-a") + rerun.reset() + self.configure_pytest(["tests/"], ["tests/"], rootdir="/repo-b") + + self.assertEqual(os.environ[ENV_RUNNER_CWD], "/somewhere/else") + + def test_an_override_between_two_runs_is_respected(self): + # Exported AFTER a run this adapter stamped, which is the case a plain + # "we wrote it once" flag gets wrong: the value in the environment is no + # longer the one we wrote, so it is the caller's and not our leftover. + self.configure_pytest(["tests/"], ["tests/"], rootdir="/repo-a") + rerun.reset() + os.environ[ENV_RUNNER_CWD] = "/chosen/by/the/caller" + + self.configure_pytest(["tests/"], ["tests/"], rootdir="/repo-b") + + self.assertEqual(os.environ[ENV_RUNNER_CWD], "/chosen/by/the/caller") + + def test_an_override_equal_to_our_own_stamp_is_indistinguishable(self): + # Accepted limitation, pinned here so it reads as a decision rather + # than an oversight. The environment is the only channel, and a caller + # who exports the SAME path the adapter already stamped leaves it + # byte-identical to our leftover — no comparison can separate the two, + # so the second run treats it as ours. Pinning a directory across runs + # is done by exporting it BEFORE the first `enable()`, which is never + # claimed as ours (the two tests above). + self.configure_pytest(["tests/"], ["tests/"], rootdir="/repo-a") + rerun.reset() + os.environ[ENV_RUNNER_CWD] = "/repo-a" + + self.configure_pytest(["tests/"], ["tests/"], rootdir="/repo-b") + + self.assertEqual(os.environ[ENV_RUNNER_CWD], "/repo-b") + + def test_unsetting_it_between_two_runs_stamps_again(self): + # The other side of the same comparison: nobody set it, so there is no + # instruction to respect and the second run needs its own directory. + self.configure_pytest(["tests/"], ["tests/"], rootdir="/repo-a") + rerun.reset() + del os.environ[ENV_RUNNER_CWD] + + self.configure_pytest(["tests/"], ["tests/"], rootdir="/repo-b") + + self.assertEqual(os.environ[ENV_RUNNER_CWD], "/repo-b") + + +class TestQuoting(RerunTestCase): + def test_a_path_with_a_space_survives_the_shell(self): + options = self.configure_pytest( + ["my tests/test_a.py"], ["my tests/test_a.py"] + ) + + self.assertIn( + rerun._quote([os.path.abspath("my tests/test_a.py")]), + str(options["launchCommand"]), + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/packages/shared/src/runner.ts b/packages/shared/src/runner.ts index 44507e42..372f66c4 100644 --- a/packages/shared/src/runner.ts +++ b/packages/shared/src/runner.ts @@ -45,6 +45,27 @@ export const RUNNER_ENV = { NIGHTWATCH_BIN: 'DEVTOOLS_NIGHTWATCH_BIN' } as const +/** + * Slots an adapter leaves in its `rerunCommand`, which the backend fills in + * when the dashboard reruns one entry. The two differ in what they select by, + * and the difference is not cosmetic: + * + * - `testName` is a NAME PATTERN. It is regex-escaped on substitution because + * every runner that consumes it filters by regex (mocha `--grep`, jest + * `--testNamePattern`, cucumber `--name`). + * - `testId` is an EXACT id, substituted verbatim and shell-quoted. pytest + * selects by nodeid (`file.py::Class::test`), matched literally, so the + * escaping `testName` needs would corrupt it — measured, `pytest + * 'test_thing\.py::test_a'` collects nothing. + * + * A template carries one or the other; `testId` is filled from the entry's + * `uid`, `testName` from its label. + */ +export const RERUN_SLOT = { + testName: '{{testName}}', + testId: '{{testId}}' +} as const + /** POST /api/tests/run body. */ export interface RunnerRequestBody { uid: string diff --git a/packages/shared/src/ws.ts b/packages/shared/src/ws.ts index ee5eee02..5f8a6ce1 100644 --- a/packages/shared/src/ws.ts +++ b/packages/shared/src/ws.ts @@ -24,11 +24,21 @@ export type ControlScope = export type WsMessageScope = TraceScope | ControlScope -/** Payload broadcast under the `clearExecutionData` scope. */ +/** + * Payload broadcast under the `clearExecutionData` scope. + * + * Two unrelated events arrive under this one scope, and only the sender can + * tell them apart: a RUN STARTING because someone pressed Run/Rerun, and a + * single entry resetting inside a run already in flight (Nightwatch re-emits a + * cucumber scenario suite that way). `runStart` marks the first, so a receiver + * never has to infer it from the uid — inferring it from the uid is what made + * the second rerun of a session keep the first one's console and network rows. + */ export interface ClearExecutionDataWsPayload { uid?: string entryType?: 'suite' | 'test' clearSuiteTree?: boolean + runStart?: boolean } /** Discriminated-union envelope for every message that crosses the WS. */