Resolve backend E2E server launches through the installed TSX package

[alexeygrigorev]

Resolve backend E2E server launches through the installed TSX package

Status: pending
Tags: bug, backend, testing, P1
Depends on: None
Blocks: None

Scope

Make every backend Playwright server launch work in the standard agent worktree layout without a hand-created node_modules link.

Replace the current mix of repository-root .bin/tsx paths and npx tsx launches with one shared resolver in backend/e2e/helpers/. The resolver shall locate tsx/cli from backend/package.json using Node's normal dependency lookup and return:

[process.execPath, [resolvedTxCliPath, backendTestServerScript]]

Update all backend E2E users of scripts/test-server.ts to use it: global setup and the 11 specs that currently launch isolated servers. Keep each spec's own environment, readiness polling, and process teardown unchanged; this issue centralizes executable resolution only, not general server lifecycle management.

Also change the ignore pattern from node_modules/ to node_modules so an accidental symlink or file with that name cannot escape .gitignore.

The selected strategy is direct Node execution of a resolved tsx/cli entrypoint, not npx. It avoids dependence on shell lookup behavior while allowing dependencies installed at the enclosing workspace root to be discovered from a worktree beneath that checkout. Missing dependencies must fail loudly with a clear TSX/module-resolution error; do not add a fallback or auto-install behavior.

CI is unaffected: it runs npm ci, which installs the workspace dependencies normally.

Acceptance Criteria

• No backend E2E code spawns npx tsx or a node_modules/.bin/tsx path to launch scripts/test-server.ts; all such launches use the shared resolver.
• backend/e2e/global-setup.js and all isolated-server specs use the same resolved TSX launcher.
• In a fresh detached Git worktree created beneath this checkout, with no worktree-local backend/node_modules, both formerly broken specs start their servers and pass.
• The full backend E2E workflow passes in that same clean worktree.
• git check-ignore -v node_modules matches the updated ignore rule.
• No tracked compatibility shim, dependency installation fallback, or worktree-specific path remains.

Test Scenarios

Scenario: Standard agent worktree has no local backend dependencies

Given: main has Node workspace dependencies installed, and a detached worktree exists under .tmp/worktrees/ from that checkout.
When: verify that the worktree has no backend/node_modules, then from the worktree's backend/ run:

npx playwright test e2e/frontend-module-characterization.spec.js --reporter=line
npx playwright test e2e/planning-surfaces-design.spec.js --reporter=line

Then: both commands pass without spawn ... ENOENT, manual symlinks, or environment-specific setup.

Scenario: Complete backend browser suite

Given: the same clean worktree.
When: run:

npm --prefix backend run test:e2e

Then: the full Playwright workflow passes, including global setup/teardown and capability evidence collection.

Scenario: Ignore safety net

Given: the repository changes are checked out.
When: run:

git check-ignore -v node_modules

Then: Git reports the node_modules ignore rule.

Out of Scope

• Consolidating readiness polling, environment construction, or stop/teardown logic into one server lifecycle abstraction.
• Changing application runtime startup, CI installation, infrastructure, or deployed Lambda packaging.
• Making arbitrary worktrees outside this checkout inherit dependencies from another checkout; those still need a normal dependency install.
• Product UI or behavioral changes.

Dependencies

• None. The enclosing checkout used to create standard worktrees must have completed its normal npm ci/dependency setup, as today.

[alexeygrigorev]

CHANGES REQUESTED

Verification was run in detached worktree /home/alexey/git/dataops/.tmp/worktrees/issue-194-tester at base f028842d493c1af3f18fdbc4c0db6f8b5ed92e8a. Its staged tree is identical to the engineer worktree (55f240922dc77bac6d6c02c75484e9eda924e7db). Neither the worktree root nor backend/node_modules existed before the browser runs. Toolchain: Node v24.13.1, npm 11.8.0, Playwright 1.61.1.

Evidence

Static audit — passing

• git diff --cached --check: exit 0.
• Changed files: .gitignore, backend/e2e/global-setup.js, the shared backend/e2e/helpers/tsx-launcher.js, and all 11 isolated-server specs.
• Resolver users: global-setup.js plus exactly 11 specs. The helper uses createRequire(<backend>/package.json).resolve('tsx/cli') and returns [process.execPath, [resolvedTxCliPath, backendTestServerScript]].
• In this layout it resolved to /home/alexey/git/dataops/node_modules/tsx/dist/cli.mjs (TSX 4.22.4) while executing the detached worktree's scripts/test-server.ts.
• No E2E executable-launch match remains for spawn('npx'), node_modules/.bin/tsx, or another direct .bin/tsx path. The only textual scripts/test-server.ts match is an explanatory comment in docs-content-root.js.
• git ls-tree -r HEAD | rg '(^|/)node_modules(/|$)|(^|/)\\.bin/' finds no tracked entries.
• Ignore probes both report .gitignore:1:node_modules for node_modules and backend/node_modules.

Focused scenarios

Command 1:

npx playwright test e2e/frontend-module-characterization.spec.js --reporter=line

Exit 1: 4 passed, 1 failed. Both global setup and the isolated server started through the new resolver—there was no spawn ... ENOENT. The failure is the Users-table assertion at frontend-module-characterization.spec.js:339: expected horizontal overflow greater than 362px, received 362px. Artifact: backend/test-results/frontend-module-characteri-daef9-time-errors-at-mobile-width/.

The same command on unchanged main also exited 1 with 4 passed and the identical overflow failure. This is existing UI/test drift, not a regression from issue 194.

Command 2:

npx playwright test e2e/planning-surfaces-design.spec.js --reporter=line

Exit 1: 2 failed. Again, servers started through the resolver without ENOENT. Failures:

• Newsletter expected computed gap: 16px, received 20px (planning-surfaces-design.spec.js:152).
• Calendar expected .calendar-grid > .calendar-day count 42, received 0 (planning-surfaces-design.spec.js:176).

The same command on unchanged main also exited 1 with the same two failures. These are also existing drift. Artifacts are under backend/test-results/planning-surfaces-design-*; stable desktop screenshots remain under .tmp/screenshots/planning-surfaces/.

Complete backend E2E workflow

npm --prefix backend run test:e2e

Exit 1: 69 total, 61 passed, 8 failed, completed in 5.6 minutes. Capability coverage collected 139 stable states. Playwright traces, videos, screenshots, and error contexts are under backend/test-results/.

Failures:

1. calendar-seams.spec.js:98 — expected .calendar-grid>section count 7, received 0.
2. calendar-seams.spec.js:216 — same selector/count mismatch.
3. frontend-module-characterization.spec.js:307 — reproduced independently on main; Users-table overflow expectation is stale versus current fitted table layout.
4. planning-surfaces-design.spec.js:127 — reproduced independently on main; newsletter gap is now 20px rather than expected 16px.
5. planning-surfaces-design.spec.js:166 — reproduced independently on main; calendar DOM no longer matches the expected 42 .calendar-day children.
6. sponsor-crm-communications-production-portal.spec.js:52 — timed out waiting for the Open booking button at line 355.
7. vite-dev-portal.spec.js:332 — development stack startup failed: node_modules/.bin/tsx ENOENT and node_modules/.bin/vite ENOENT.
8. vite-dev-portal.spec.js:490 — same development-stack startup failure in representative mode.

The Vite failures are within this issue's clean-worktree goal: scripts/dev-portal.mjs:61-72 still launches path.join(cwd, 'node_modules', '.bin', 'tsx') and path.join(cwd, 'node_modules', '.bin', 'vite').

Acceptance results

1. No isolated-server npx tsx / .bin/tsx launches; shared resolver used — PASS.
2. Global setup and all 11 isolated specs use the resolver — PASS.
3. Both formerly broken specs start and pass in the clean worktree — FAIL. Servers start, but the focused suites contain five assertion failures total; three of those were reproduced on unchanged main.
4. Full backend E2E passes — FAIL: 61/69 passed; 8 failed.
5. Updated ignore rule matches — PASS.
6. No compatibility shim, install fallback, or worktree-specific path remains — PARTIAL/FAIL. The changed E2E launcher has none, but the existing development-stack launcher retains worktree-root .bin paths exposed by the required workflow.

Requested fixes

• Extend direct dependency resolution to the development stack. Resolve the installed Vite CLI through createRequire rooted at the repository package and execute it with process.execPath; likewise resolve TSX rather than using repository-root .bin. Keep the loud resolution errors and add no installation fallback.
• Reconcile the stale frontend characterization, planning-surface, and calendar-seam expectations with the current approved DOM/CSS contracts, or restore the intended UI behavior. Since several reproduce on unchanged main, these should either be fixed as part of making issue 194's required suite green or split into explicitly sequenced blocking issues.
• Investigate the missing Open booking control using backend/test-results/sponsor-crm-communications-7621d-ney-and-safe-review-cleanup/error-context.md, then rerun the complete workflow from a fresh detached worktree.

[alexeygrigorev]

Addressed the requested fixes in /home/alexey/git/dataops-issue-194. The staged tree is 910d379c30e9365c509edd6c53e70947d9509be7; an identical staged copy remains available in fresh detached worktree /home/alexey/git/dataops/.tmp/worktrees/issue-194-fix. Neither validation worktree had a root or backend node_modules.

Changes:

• Extended direct installed-package resolution to the development stack. Vite resolves through createRequire rooted at the repository package and TSX through its installed CLI; both execute with process.execPath, fail loudly, and have no install fallback.
• Added coverage asserting development child CLIs resolve through package manifests rather than .bin.
• Reconciled stale calendar selectors and Users-table overflow expectations with the current approved DOM/CSS contracts.
• Restored the intended newsletter/calendar shared surface spacing contract so the responsive gap is 16px.
• Added visible synthetic-data waits before Open booking clicks to remove the sponsor CRM test race.

Verification in the clean detached worktree:

• npx playwright test e2e/frontend-module-characterization.spec.js --reporter=line — 5 passed.
• npx playwright test e2e/planning-surfaces-design.spec.js --reporter=line — 2 passed.
• npx playwright test e2e/calendar-seams.spec.js --reporter=line — 2 passed.
• npx playwright test e2e/sponsor-crm-communications-production-portal.spec.js --reporter=line — 1 passed.
• npx playwright test e2e/vite-dev-portal.spec.js --reporter=line — 2 passed.
• npm --prefix backend run test:e2e — 69 passed; capability evidence collected across 139 stable states.
• node --test scripts/dev-portal.test.mjs — 17 passed.
• Frontend unit tests and backend unit/integration tests also pass (959 passed, 1 skipped).
• git diff --cached --check, ignore probes for node_modules and backend/node_modules, and the .bin/npx tsx launch audit pass.

The unrelated issue-195-sidebar-tree checkout was not modified.

[alexeygrigorev]

TEST PASS

Independent verification target: engineer staged tree 910d379c30e9365c509edd6c53e70947d9509be7 from /home/alexey/git/dataops-issue-194.

I recreated that exact tree in fresh detached worktree /home/alexey/git/dataops/.tmp/worktrees/issue-194-independent-v2 at base f028842d493c1af3f18fdbc4c0db6f8b5ed92e8a. Neither worktree-local node_modules nor backend/node_modules existed before browser execution. Toolchain: Node v24.13.1, npm 11.8.0, Playwright 1.61.1. The tracked tree still hashes to 910d379c30e9365c509edd6c53e70947d9509be7 after all runs.

An earlier attempt was invalidated by an unrelated active DataOps browser run in the main checkout holding fixed port 3001. I left that process alone, waited for it to exit, and reran cleanly. The results below are from the clean rerun only.

Static audit

• git diff --check — exit 0.
• Shared TSX resolution: backend/e2e/helpers/tsx-launcher.js uses createRequire(<backend>/package.json).resolve('tsx/cli') and returns [process.execPath, [resolvedTxCliPath, backendTestServerScript]].
• Resolved command in the detached layout:

node /home/alexey/git/dataops/node_modules/tsx/dist/cli.mjs
     /home/alexey/git/dataops/.tmp/worktrees/issue-194-independent-v2/backend/scripts/test-server.ts

• Global setup plus exactly 11 isolated-server specs import/use the shared resolver.
• No match remains for spawn('npx'), node_modules/.bin/tsx, or another direct .bin/tsx server launch.
• Development-stack CLIs also resolve through installed package manifests rather than repository-root .bin; scripts/dev-portal.test.mjs covers both TSX and Vite.
• git ls-tree -r HEAD finds no tracked node_modules or .bin entries.
• Ignore probes report .gitignore:1:node_modules for both node_modules and backend/node_modules.

Required commands

Command	Result
npx playwright test e2e/frontend-module-characterization.spec.js --reporter=line	exit 0 — 5 passed
npx playwright test e2e/planning-surfaces-design.spec.js --reporter=line	exit 0 — 2 passed
npm --prefix backend run test:e2e	exit 0 — 69 passed in 8.0m
node --test scripts/dev-portal.test.mjs	exit 0 — 17 passed
git check-ignore -v node_modules and git check-ignore -v backend/node_modules	exit 0

Full-suite capability evidence collected 139 stable states. After the run, backend/node_modules remained absent; the worktree-root node_modules contained only Vite-generated temporary/cache entries, not installed dependencies.

Visual/layout evidence

The touched planner spacing has generated screenshots for Newsletter and Calendar at desktop, 820px, and 390px in both themes. I audited dimensions, nonblank pixel statistics, OCR content, computed gaps, document/body overflow, exact-palette checks, Axe coverage, and browser error observers. At 820px both surfaces measured gap: 16px; neither surface nor page overflowed.

I added an explicit authenticated mobile /users capture at .tmp/screenshots/issue-194-independent-v2/users-mobile-390x844.png. Its table wrapper measured 362x362, while document and body widths were both 390. OCR confirms the stacked Name/Email/Role/Created/Actions presentation, with no 404/error-page content.

Other retained artifacts include .tmp/screenshots/planning-surfaces/*, .tmp/screenshots/planning-surfaces-820/*, .tmp/screenshots/issue-110/*, and .tmp/screenshots/issue-190/processes.png.

Acceptance results

1. No isolated-server npx tsx / .bin/tsx launches remain; all use the shared resolver — PASS.
2. Global setup and all 11 isolated specs use the same resolved TSX launcher — PASS.
3. Both formerly broken specs start their servers and pass in a fresh detached worktree with no backend-local dependencies — PASS.
4. Full backend E2E passes in that same worktree — PASS.
5. Updated ignore rule matches both probes — PASS.
6. No compatibility shim, install fallback, or worktree-specific path remains — PASS.

Process note: this staged tree also reconciles browser/UI contracts and restores the intended Newsletter/Calendar responsive spacing. That work overlaps #198 and expands issue 194’s original “no product UI changes” boundary, but the first tester explicitly allowed fixing those required-suite blockers as part of this issue. PM acceptance should coordinate/update or subset #198 so the overlap is intentional and not duplicated.

[alexeygrigorev]

ACCEPTED

PM acceptance reviewed staged tree 910d379c30e9365c509edd6c53e70947d9509be7 against the tester evidence and retained artifacts in /home/alexey/git/dataops/.tmp/worktrees/issue-194-independent-v2/.tmp/screenshots/.

From the operator perspective:

• The clean-worktree launcher journey is complete: global setup, isolated specs, and the development stack resolve installed CLIs directly, fail loudly when dependencies are absent, and require no manual symlink or install fallback.
• The mobile Users representation is usable: the authenticated 390px capture shows stacked Name, Email, Role, Created, and Actions content with no error page or forced table scrolling.
• Newsletter and Calendar remain coherent at 820px in light and dark themes. Their captures show the expected navigation, filters, empty/error guidance, month/week hierarchy, ISO weeks, and swipe guidance; pixel statistics confirm nonblank rendered states.
• Copy and empty states remain public-safe and consistent with the existing portal vocabulary.
• The sponsor CRM waits preserve the visible synthetic-data-to-action sequence without changing product behavior.

The implementation matches the repaired target and all six acceptance criteria. Software Engineer may commit this tree with Closes #194.

Issue 198 coordination: this tree deliberately contains the minimum browser-contract and spacing repairs needed to make the required Issue 194 workflow pass. It uses permissive descendant Calendar selectors and only replaces the obsolete forced Users overflow assertion. Issue 198 remains open for its stricter canonical grouped selectors, richer stacked-row assertions, complete responsive/theme evidence, parity-harness repairs, and SAM packaging follow-up. The Issue 198 owner should rebase on this change and remove any duplicated CSS/minimal-selector delta before continuing.

[alexeygrigorev]

ON-CALL REPORT — PASS

Reviewed merge commit 3e0fb196d7304f4807067a1fd3d177c4bfb53e6a, which closes this issue.

Runs inspected

• Deploy DataOps V1 Lambda: success — https://github.com/DataTalksClub/dataops/actions/runs/32746968920. Frontend unit coverage, backend tests, canonical Task/Card transaction gate, backend typecheck/build, SAM validation, packaged artifact checks, stack deployment, runtime seed contract, and backend smoke test all passed. Runtime smoke result was root=302 login=302.
• Validate Backend Browser E2E: success — https://github.com/DataTalksClub/dataops/actions/runs/32746968969. Playwright completed with 70 passed in 4.5 minutes.
• Commit-scoped workflow listing found no other applicable path-filtered runs for this push.

Independent CloudFront health probes also returned HTTP 302 for / and /login.

Classification

None: no CI infrastructure failure, deployment failure, content-validation failure, external-account blocker, flake, or regression.

Pipeline status

origin/main remains at 3e0fb196. No on-call fix was required. Issue #194 is closed and its approved change is deployed.

The shared checkout currently has unrelated in-progress #201 edits on issue-201-hidden-home-exception-badges; they were not touched. Recommended next step is to continue the existing #201 implementation/review track without disturbing those edits.

ON-CALL DONE: PASS