Skip to content

test: cut the e2e suite to what needs a browser, and serve it production builds - #1930

Merged
joshunrau merged 9 commits into
DouglasNeuroInformatics:mainfrom
joshunrau:test/faster-e2e
Nov 19, 2026
Merged

joshunrau merged 9 commits into
DouglasNeuroInformatics:mainfrom
joshunrau:test/faster-e2e

Conversation

@joshunrau

@joshunrau joshunrau commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

pnpm test:e2e ran 419 tests against four dev servers, took 7 minutes on two local workers (about 15 on CI's one), and with a second session's pnpm check alongside it pushed an 18 GB machine into swap until it crashed. It now runs 98 tests against production builds in about 1.3 minutes locally, and the behaviour the removed tests proved is pinned in unit and API tests instead.

Changes

  • The e2e suite serves production builds: the gateway's production server, and vite preview of the web and playground builds, which pnpm test:e2e builds first through turbo. The API still runs pnpm dev:test. A bare playwright test or test:dev serves whatever was last built.
  • Firefox is no longer run, and CI installs only Chromium.
  • New API variable PASSWORD_BREACH_CHECK_ENABLED (default true). When false, setting a password no longer asks the Have I Been Pwned API. The e2e API server and the api vitest project set it to false, so neither depends on an outside server. docker-compose.yaml forwards it.
  • The API's gateway sync interval in e2e is a 2 s constant the suite owns and passes to the API, in place of the 10 s from .env. apps/api's dev:test now loads .env with env-cmd --no-override, so a variable already set in its environment wins over .env.
  • pnpm test:coverage and pnpm test:e2e wait for a lock in the git common directory, shared by every worktree. A second run on the machine waits and names the run it is waiting on. A lock whose holder has exited is taken over.
  • The suite goes from 399 tests to 98:
    • Deleted: tests whose behaviour a unit or API test already pinned. Each covering test was read first, and a missing assertion was added where it fell short.
    • Moved: rules owned by one component, hook or route moved to unit tests. Row-scoping checks, which mocked models cannot show, moved to new suites in apps/api/test/suites/ (07–09) that run against the real database.
    • Merged: tests that walked one flow now share one setup.
    • No fixed sleeps: preconditions are seeded over the API instead of the UI, and the tests that slept now poll for the state they wait on.
  • pnpm lint generates apps/outreach's Astro content types (gen:types, astro sync) before the root type-check reads them. On a fresh checkout such as CI's, the type-check could run first and read every astro:content import as any.
  • Page-object members and ApiClient methods that nothing uses any more are removed. The agent docs and the testing-strategy doc describe the integration tier, the production-build suite and the lock.

Decisions

  • Production builds over dev servers. The gateway's dev server alone held about 1.25 GB, because it runs Vite in-process. The cost is that e2e no longer sees dev-only behaviour.
    • On a demo instance the production web build makes the group-manage form read-only (manage.tsx), and the e2e instance is a demo because demo setup seeds its instruments.
    • So the group-settings save path moved to 09-group-access.suite.ts, and the e2e now asserts the read-only form instead.
  • Firefox is dropped rather than kept for a smoke subset. It ran 20 tests and added a browser install to CI. A Chromium-only suite loses its only non-Chromium engine.
  • Deleting rather than keeping an e2e twin of a unit test. Tests that only duplicate a unit or API test are gone, even where the e2e version felt more realistic. What only a browser and the other apps together can prove stays in e2e.
  • The lock covers test:coverage and test:e2e, not pnpm test or pnpm check as a whole. Plain vitest is often left in watch mode, and lint is light. CI's single job takes the lock uncontended.
  • CI still runs one e2e worker. Raising it is left for a measured follow-up.

Risks

  • pnpm test:coverage in an interactive terminal starts vitest in watch mode, which holds the lock until it is closed. Another session's pnpm check waits for it.
  • A deployment that sets PASSWORD_BREACH_CHECK_ENABLED=false stops refusing breached passwords. Unset, nothing changes.

Notes

  • Memory, measured on two local workers before the test cuts: the move to production builds took the suite's typical footprint from 5.4 to 3.6 GB, its peak from 8.0 to 6.9 GB, and its run from 7.9 to 4.7 minutes.
  • A second review run covered the fix made after the first. pnpm check then showed 07-mail-delivery.suite.ts was racy: it freed a probed port that another process could start listening on, so it now uses port 1. That second run, over that one file, returned no findings.
  • The commit that fixed CI's lint after this PR opened touches only JSON and Markdown, which the review does not read. Before the fix, running the root lint alone on a fresh worktree reproduced CI's 29 astro:content errors; with it, the same run passes.
  • The review's series-summary fix was proved by mutation. Removing the ability check in summarizeRecords makes all three summary cases fail, including by-series, which passed with or without scoping before the fix.

Adversarial review

Reviewed at commit d29ce83. pnpm check: green.

Findings

  1. apps/api/test/suites/08-authorization.suite.ts:420 — the by-series summary case passes whether or not the route is scoped, because no record in the suite carries a seriesInstrumentId — write a series-keyed record in the foreign group and keep the [] assertion
  2. testing/src/specs/gateway-landing.spec.ts:5 — the comment justifies the toPass retry with an admin-settings language toggle that this change deleted — rewrite it with the reason that still holds, or drop the retry
  3. testing/src/specs/group-switcher.spec.ts:55 — "the archived group is listed first" no longer holds once the groups are created concurrently — create the archived group first, or drop the claim and its assertion
  4. apps/api/test/suites/08-authorization.suite.ts:155 — createRecord repeats createSubject's session request — share one createSession helper

Fixed

    1. The foreign group now also holds a record with a series id; checked by mutation (above).
    1. French is active by default on the gateway and no spec turns it off, so the retry and its comment are removed.
    1. The archived group is created before the other three, which are still created concurrently.
    1. One createSession helper; createSubject and createRecord build on it.

Attestation

I, Claude Opus 5.5, attest that the review block above reports the adversarial review as it ran on the commit named, with every finding it returned.

🤖 Generated with Claude Code

joshunrau and others added 8 commits November 18, 2026 23:30
Every password set or changed asks the Have I Been Pwned API about it.
PASSWORD_BREACH_CHECK_ENABLED (default true) turns that off, so the e2e
suite and the api integration suites no longer depend on an outside
server, and an air-gapped deployment need not wait on one either.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The gateway, web and playground dev servers held about 2.5 GB between
them (the gateway alone 1.25 GB, running Vite in-process) and served
every module unbundled. The suite now starts the gateway's production
server and `vite preview` of the web and playground builds, which turbo
builds before it. On two workers a run fell from 7.9 to 4.7 minutes and
its typical footprint from 5.4 to 3.6 GB.

Firefox ran only the @smoke tests and is dropped, along with its CI
install. The API's gateway sync interval is now a constant the suite
owns (2 s) and passes to the API, instead of the 10 s a deployment uses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Most end-to-end tests asserted a rule a unit or API test already pinned,
or one that a component or a single route owns. Each was deleted only
after its covering test was read, and where that test fell short the
missing assertion was added first. Row scoping, which mocked models
cannot show, moved to new suites in apps/api/test that run against the
real database; tests that walked one flow were merged to share their
setup, and preconditions are seeded over the API instead of the UI.

The suite goes from 399 tests to 98 and, on two workers, from 4.7 to
1.3 minutes. Fixed sleeps are gone: the gateway tests poll the API for
the synchronized state, the slow-import and malformed-submission tests
are covered by unit tests, and the JSPsych example runs with its timing
patched. Page-object members and ApiClient methods nothing uses any more
are removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…suite

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pnpm test:coverage and pnpm test:e2e now wait for a lock held in the
git common directory, which every worktree shares. Two sessions running
`pnpm check` at once exhausted an 18 GB machine's memory, and two e2e
runs collide on the same ports regardless. The waiting run names the one
it waits on, and a lock whose holder has exited is taken over.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…should

The series datahub summary is now checked against a foreign record that
carries a series id, without which the summary returned nothing whether
or not it was scoped. The archived group is created before the others,
so it is the first of the user's groups as the test assumes. The
gateway landing test no longer retries for a language toggle no spec
makes any more, and the authorization suite's two session helpers share
one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ed port

The suite freed a probed port and assumed it stayed free until it sent
mail. Another process started listening on it in between, and delivery
failed on authentication instead of being refused. Port 1 lies outside
the range the OS assigns to other sockets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added Status: Waiting on Review Automated: unreviewed, pushed since the last review, or review re-requested Status: CI Failing Automated: a check failed; the author fixes it and removed Status: Waiting on Review Automated: unreviewed, pushed since the last review, or review re-requested labels Nov 19, 2026
`//#lint:root` type-checks apps/outreach, whose `astro:content` types
live in the git-ignored `.astro/` that only `astro check` (outreach's
own `lint`, running alongside) wrote. On a fresh checkout such as CI's,
the root type-check could run first and read every collection as `any`.
`@opendatacapture/outreach#gen:types` runs `astro sync`, caches `.astro/`,
and `//#generated` now depends on it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added Status: Waiting on Review Automated: unreviewed, pushed since the last review, or review re-requested and removed Status: CI Failing Automated: a check failed; the author fixes it labels Nov 19, 2026
@joshunrau
joshunrau merged commit 4708eef into DouglasNeuroInformatics:main Nov 19, 2026
3 checks passed
@joshunrau
joshunrau deleted the test/faster-e2e branch November 19, 2026 15:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Status: Waiting on Review Automated: unreviewed, pushed since the last review, or review re-requested

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant