Skip to content

Stop a test through recovery errors; one stall counter in Tester - #288

Merged
DavertMik merged 1 commit into
mainfrom
tester-stall-recovery
Nov 1, 2026
Merged

DavertMik merged 1 commit into
mainfrom
tester-stall-recovery

Conversation

@DavertMik

@DavertMik DavertMik commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Browser recovery goes through the loop's catch, not a check on every iteration

  • Explorer.runWithRecovery() throws BrowserRecoveryError when the page is missing and can't be restored. It used to throw a plain Error, which recover(error) treated as non-fatal, so the test kept running against a dead browser.
  • recoverOrRestart() calls watchActiveTestPage() after any successful recovery. Pages restored inside runWithRecovery now keep reporting console errors and dialogs. The duplicate calls in recover() and recoverFromExecutionError() are gone.
  • Tester no longer calls explorer.recover() at the start of every iteration.

One definition of "no progress" in Tester

  • The two stall methods are merged into shouldStopAfterStall(task, note).
  • Removed the hasNoBrowserProgress check, which was always true where it ran.
  • Removed the "AI model is not responding with actions" stop. The stall stop at 3 already covered it.
  • isStruggling now reads the stall counter, so interact, visualClick and exitIframe count as progress there too. Tester no longer reads consecutiveFailures or consecutiveEmptyResults. Captain and Driller still use them, so TaskAgent is unchanged.

Behaviour change: Pilot's "struggling" review now starts after two iterations in a row with no progress, one before the stall stop. Before, three failed clicks in a single iteration could trigger it right away.

Test plan

  • bun test tests/unit/tester*.test.ts tests/unit/explorer*.test.ts passes
  • bun test tests/integration passes (one Prima aimock test was flaky and passed on rerun)
  • Full unit suite: 8 failures in provider.test.ts and apibot-config.test.ts, in files that import neither tester nor explorer
  • Regression run, which needs a maintainer to approve it

🤖 Generated with Claude Code

- Explorer: a missing page that can't be restored throws BrowserRecoveryError,
  so the loop's catch hook stops the test; recoverOrRestart re-watches the
  test page after every recovery
- Tester: drop the per-iteration recover() check, merge the two stall tails
  into shouldStopAfterStall, remove the always-true progress check and the
  empty-results stop, base isStruggling on the stall counter

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 1, 2026

Copy link
Copy Markdown

Explorbot Self-Regression

Commit b12c935 · run

Scenario Result Attempts Time Previous median Change
basic (native) PASS 1/3 4m 53s 6m 6s -20% faster
experience: control OK — failed as expected 1/1 25s 22s +14%
experience: seeded PASS 1/3 50s 43s +16%
slow (native) PASS 1/3 2m 8s 1m 36s +33% slower
gated (native) PASS 1/3 2m 15s 2m 23s -6%

Time is the passing attempt (the last attempt if none passed). Change compares it with the median passing time of the last 2 runs; ±20% or more is flagged.

Attempt details

  • basic (native) attempt 1 — PASS (4m 53s): login evidence: PASS (post-login plan=true, post-login research=true); research: PASS (files=5, wellFormed=true, keywords=7/3); scenarios: PASS (tests=6/5, features=4/3); tests passed: PASS (5 passed, 0 failed (reporter: 5 passed, 0 failed))
  • experience: control attempt 1 — PASS (25s): control: OK — failed as expected (0 passed, 0 failed)
  • experience: seeded attempt 1 — PASS (50s): seeded: PASS (1 passed, 0 failed)
  • slow (native) attempt 1 — PASS (2m 8s): login evidence: PASS (post-login plan=true, post-login research=true); research: PASS (files=2, wellFormed=true, keywords=7/3); scenarios: PASS (tests=7/5, features=4/3); tests passed: PASS (1 passed, 0 failed (reporter: 1 passed, 0 failed))
  • gated (native) attempt 1 — PASS (2m 15s): login evidence: PASS (post-login plan=true, post-login research=true); research: PASS (files=4, wellFormed=true, keywords=6/3); scenarios: PASS (tests=8/5, features=4/3); tests passed: PASS (1 passed, 0 failed (reporter: 1 passed, 0 failed))

Session analysis — basic (native):

Session Analysis

Empty-result search on the Issues page behaves correctly. Issue creation, validation, title search, and status filtering ran, but their checks have gaps or automation failures, so they cannot be confirmed.

Coverage

  • Pages: /issues, /issues?new=1, /issues?q=...&status=
  • Features: issue creation form, required-title validation, title search, status filter, empty search results

What works

  • Issue search (no-match state): ET-4 Search issues with text that matches nothing and verify the empty result

UX issues

None confirmed.

Execution Issues

  • ET-1 Create a new issue from the New Issue button and verify it appears in the issues table - Row priority, labels, status never asserted; only title checked.
  • ET-2 Submit the new issue form without a title and verify the issue is rejected - Submit step logged FAILED; no before-submit row count baseline.
  • ET-3 Search issues by a title fragment and verify only matching issues are listed - Search box click logged FAILED; text typed without target field.
  • ET-5 Filter issues by a status option and verify only issues with that status are listed - Only negative table assertions; no check Open rows appear.

Session analysis — slow (native):

Session Analysis

Creating an issue from the Issues page works: the new issue saves with its entered title and appears in the list alongside the existing issues. Only this one flow was explored, so editing, filtering, and other issue actions are unverified.

Coverage

  • Pages: /issues, /issues?new=1
  • Features: new issue drawer (description, assignee, submit), issues list display

What works

  • Issue creation — ET-1 Create a new issue from the toolbar and verify it appears in the issues list

Session analysis — gated (native):

Session Analysis

Issue creation from the toolbar works end to end: a new issue appears in the issues table with its entered title, status, and priority, alongside the existing issues. No product defects were observed in this flow.

Coverage

  • Pages: /workspace, /issues?new=1, /issues
  • Features: Toolbar issue creation, issues table listing, issue row status and priority display

What works

  • Issue creation - ET-1 Create a new issue from the toolbar and verify it appears in the issues table

Execution Issues

  • ET-1 Create a new issue from the toolbar and verify it appears in the issues table - form fill failed once before retry; row checks are substring-only.

@DavertMik
DavertMik merged commit e310a29 into main Nov 1, 2026
3 checks passed
@DavertMik
DavertMik deleted the tester-stall-recovery branch November 1, 2026 17:29

This branch was successfully deployed

1 active deployment
regression — b71baaa7 Deployed Oct 31, 2026 by DavertMik via regression #200
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant