feat(app-a11y): scan the WDIO config-level before() window, and report it as a hook

[kamal-kaur04]

Stacked PR. Base is SDK-7422-app-a11y-scan-registration (#165), not main, so the diff here is the before-hook work alone — 507 insertions across 5 files. It retargets main automatically when #165 merges. One commit (adb8b23, the @Measure placement fix) also exists on the base as bd75537; same change, different SHA, so it shows in the commit list but not in the diff, and drops out on rebase.

What is this about?

Driver commands issued from WDIO's config-level before() were never scanned for App-A11y, and the rake counted them as expected coverage anyway. On the account that prompted SDK-7422 that window is app launch, login and account selection — the screens a customer most expects covered.

Those hooks are not test-framework hooks, so when they run neither onHookStart nor onBeforeTest has fired and the scan gate has no entry for the session at all. The gate now opens at driver creation, unconditionally when autoScanning is on — the same policy onHookStart already applies to Mocha hooks. Per-test include/exclude filters need a test to evaluate and there is none yet; the following onBeforeTest recomputes the gate for the test proper.

The scans needed a parent

Scans from that window went out with neither thTestRunUuid (no test exists yet) nor thHookRunUuid, so App-A11y's lookup of the scan's parent in BTCER found nothing — the scan reached the hub and then belonged to nothing:

PHASE >> config-level before() STARTED
   scan  method=execute  testRun=ABSENT  hookRun=ABSENT     ← belongs to nothing
PHASE >> mocha before all STARTED
   scan  method=back     testRun=8e466b38  hookRun=6caa3175

The window is now reported as a BEFORE_ALL hook run through the existing framework path (framework.trackEvent), so the framework mints the uuid, the binary emits HookRunStarted/HookRunFinished, and this module's own onHookStart observer stamps it onto the scans. No new plumbing, one source of truth for the uuid.

Two things fell out better than designed: resolveInstance already creates an instance for a BEFORE_* PRE when none exists (it has to — suite-level before all fires before any INIT_TEST), and the first test then reuses that instance, so the hook lands with a real test_run_id rather than orphaned.

It only fires when the user actually has such a hook

Opening this from the WDIO lifecycle would report a hook that never ran on every build in the fleet. And it cannot be detected from config: ConfigParser.addService folds the user's config hooks and every service's hooks into one config.before array (each hook.bind(service), so even name inspection fails) and the runner fires them together with Promise.all. A service is a sibling of the user's hook, not an observer of it.

So the hook run is opened on demand from the scan path — the exact condition that matters, since the only reason it exists is to parent a scan.

config	hook events reported
with a config-level before() — axzkajmr2odsx8dvyx2wfxw6mxtec3d9sbpmp08t	5 — 4 Mocha + 1 for the window
without it — lpbvftwaslksugogditfsvc1pspzd4fwgtneyxnh	4 — Mocha only, nothing extra

A failed hook is reported as failed

WDIO swallows a throwing config hook — executeHooksWithArgs resolves with the error rather than rejecting, deliberately ("so in case of a rejection it won't cause the hook to fail"). Run one that throws and the exit code stays 0, the spec reporter says 1 passing, Allure is green, and the TRA build reads passed. The failure exists in exactly one place: a log.error on stdout.

An early version of this branch made that worse — it reported the hook run as passed. Unreported means the dashboard omits a failure; passed means it asserts something false about it. The instrumentation wrapper is the only thing in the process that sees the rejection, so it records the message and the hook run reports the truth:

before: HookRunFinished BEFORE_ALL result=passed  uuid=1bcd33f4
after:  HookRunFinished BEFORE_ALL result=failed  uuid=d9c3ccb7

TRA build verdict moves with it — passed {passed: 2} → failed {passed: 1, failed: 1} (qkfnla45gogeddlwlfbszpft6ff6estskur5yqoq).

beforeSuite counts too, and is easy to miss: the Mocha adapter registers it as a root before-all (this._runner.suite.beforeAll(this.wrapHook('beforeSuite'))), so it runs inside the same window. Keyed only on 'before', a throwing beforeSuite reported a passed window (eq3p9ktfb1uiiq5mtxqj670cu42o6vf82z6nzcdx). Mocha's own hooks stay excluded — each has its own hook run with its own result, so folding them in would report one failure twice.

The hook is named after the hook

The name comes from the live hook stack, not a fixed label, because the window does not always open in before():

first scannable command comes from	reported name	build
before()	wdio "before" hook	mu2hqmwrk4ygyh1yp8queidnxfdeeju7dk4ubbdx
beforeSuite (before() touches no driver)	wdio "beforeSuite" hook	eestbqwlk8mvjhzqkijfl0lc4gnzqxredgofblrw

Captured once, when the window opens: the binary takes a hook's name from the record pushed at PRE, and by POST a different hook is running. So the name says where the window opened, not where it failed — the failure message carries the failing hook's name, so both facts survive.

Verification

Bench: test-samples/app_automate-wdio_mocha-android/examples/repro-triple/ — phase boundaries annotated into the session log with browserstack_executor, so every scan is attributed by position rather than inferred from its command type. Verdicts read from the App Automate session-logs API.

window	baseline 9.35.0	this branch
config-level before()	0	7 — execute ×2, executeScript ×2, click ×3
mocha before all	2	2
mocha before each #1	1	1
test 1, pre-reload	3	3
	6 total	13 total

hookInstrumentation is what makes the window observable at all — [hook-window] before#0 (bound before) finished in 37970ms — and it is logging only, no events.

Unit: 4 new files/suites worth — instrumentation lifecycle (incl. sync-stays-sync and unwinding on throw), the on-demand hook run (opens once, never while a framework hook is the parent, closes at the first test, never closes what it did not open), dynamic naming, and failure reporting. Full suite at parity — 75 pre-existing failures before and after. npm run lint clean.

Global-hook scans carry no test run uuid (178c541)

The scan is a browserstack_executor: {"action":"appAllyScan", arguments: {...}} payload, so the session log carries the exact params sent. work/SDK-7422-app-a11y-fixes/uuid-verdict.py <build> <session> prints a per-scan census — phase, thTestRunUuid, thHookRunUuid, command.

flow	build	sessions	window scans	test-body scans
mocha, pre-fix	rpgacbl0dyjcmmg34kwm3y5xui1bdirosz16ncll	5a6695f7c00b047ca90b7d367d580f5f3058bc8f	test uuid present — 22ec974c, test 1's own	present
mocha, post-fix	r89xojas5cvljamjpwrj8x7zcf3pc3umt8qyoptk	72d8f60ebfd6485d670021fb68f43c4163ee6897, e60022566902b3f72651c4784448e2d39ff73302	test uuid absent, hook uuid kept	present
cucumber, pre-fix	fyiiwkwdl8deghsyanxtq7e0shsrlkdc59xrmg01	dc355a966c7539ccb73ae42e8af69b75c1495b0d	absent	present
cucumber, post-fix	yldiivrvnkhf0xi7ttvzekksvxoafxulcrx66gac	d316f0f4dd32466cbe16b444a959abccd6242328	absent	present

Pre-fix, mocha's two before all scans ran before any test existed and still carried the uuid the test body later reported under. Post-fix they carry only their hook uuid, and all 16 post-reload scans still land, so the base branch's reload fix is unaffected.

Cucumber is identical either side because the defect was CLI-flow-only: mocha mints TEST_ANALYTICS_ID at instance creation, so the window had a uuid to leak, while the classic path sets it at Listener.testStarted and never had one. The guard is wired on both paths so the classic path cannot regress if that ever moves earlier.

Known limits, stated rather than implied

• A command in the hook's first tick can still escape. WDIO runs same-named hooks concurrently, and registration completes ~6 ms after session creation (measured: session created 08:03:57.817, wrapping done 08:03:57.823). Not closable from inside a service — no driver object exists to wrap earlier. Any prior driver call absorbs it.
• The exit code still stays 0 on a failed config hook. We report it to TRA; the customer's CI stays green. That is WDIO's design and no service can override it.
• Reporters show none of this. hook:start/hook:end come from @wdio/mocha-framework; the runner emits nothing for config hooks, so Allure/spec/junit cannot see the window. A reviewer checking the Allure report will find no trace of this change.
• before each Browserstack wdio service code #1 loses its test uuid — it falls inside the pre-test window, while chore: update package and lock files devDependencies #2/Updated README. #3 keep theirs. Those scans keep their own hook run uuid, and hook-run→test parentage is untouched here; only scan params changed.
• Scans from a hook that later failed are still counted as expected coverage by the rake — interacts with SDK-7406.

Related Jira task/s

https://browserstack.atlassian.net/browse/SDK-7422

Release (mandatory for every PR — required for the ready-for-review label)

Version bump: (required — tick exactly one)

• minor (backwards-compatible feature)
• patch (bug fix or other small change)

Release notes type: (optional)

• New Feature
• Bug Fix
• Other Improvement

Release notes (customer-facing): (optional but encouraged)

• Accessibility scans now run for driver commands issued from your WDIO config's before() hook, so screens visited during setup are covered.
• Hooks in your WDIO config that touch the driver now appear on the build, including when they fail — previously a failing before() was reported nowhere.

Release notes (internal): (required — engineer-facing; what actually changed / why)

• accessibilityModule: open the scan gate at onBeforeExecute (driver creation) so the pre-test window — WDIO config-level before/beforeSession — is scanned instead of silently uncovered while still counting as expected coverage.
• accessibilityModule: report that window as a BEFORE_ALL hook run via the existing framework.trackEvent path, opened ON DEMAND from the scan path so suites without such a hook report nothing extra. Named from the live hook stack; 'idle' → 'attempted' → 'open' → 'closed' so a failed PRE is neither retried per command nor closed by a POST pairing with nothing.
• New hookInstrumentation.ts: wrap the handlers of every session-scoped config hook to log start/finish, record failures, and expose the executing hook. Sync handlers stay sync so a sync throw is not converted into a rejection. Record resets per instrumented config.
• Failure reporting spans PRE_TEST_WINDOW_HOOKS (before, beforeSuite) — the latter runs in the same window because the Mocha adapter registers it as a root before-all.

Checklist

• Ready to review
• Has it been tested locally?

PR Validations

Run Tests: Comment RUN_TESTS to trigger sanity tests.

[kamal-kaur04]

Debug-log coverage across the patched set — verified, then pinned (74db214)

Only four of the eight session-scoped hooks had ever been seen firing on a device, because earlier benches didn't define the rest. So I built one that defines all eight (examples/repro-triple/repro-all-hooks.conf.ts) and checked what the instrumentation actually produced.

All eight ran, all eight were instrumented, all eight logged:

hook	handlers wrapped at construction	start/finish lines
before	1	2
beforeSuite	1	2
beforeHook	1	8
beforeTest	3	6
afterTest	2	4
afterHook	1	8
afterSuite	1	2
after	2	4

Two things in that table are worth reading rather than skimming:

• beforeHook / afterHook log 8 lines each — they fire once per Mocha hook, and this spec has four (before all, before each, after each, after all). That's 4 invocations × start+finish.
• The handler counts above 1 are not the user's. beforeTest: 3, afterTest: 2, after: 2 include expect-webdriverio's snapshot and soft-assert services, which share those arrays. The index in the label (beforeTest#0, #1, #2) is what separates them; "registered at construction" means the user's hooks plus any service constructed before ours, never our own.

Pinned with a test that iterates BROWSER_CONTEXT_HOOKS itself and asserts a started and a finished line for each, so adding a name to that list without the wrapping reaching it fails in CI rather than showing up as a silence in a device log.

Suite: 1,153 passing, 75 pre-existing failures, lint clean.

[kamal-kaur04]

Both flows, both frameworks — implemented and run (c27900b)

The fix was mocha/CLI-shaped in two ways, and asking for cucumber surfaced both.

1. The non-CLI flow had the same defect

AccessibilityHandler (the classic path) registers the session in beforeTest, beforeScenario and beforeHook — never before the first test. So a config-level hook's commands went unscanned there too, while still counting as expected coverage. before() now opens the gate when autoScanning is on, and each of those three still recomputes it.

This is the only path cucumber and jasmine ever take (CLISupportedFrameworks = ['mocha']), and the path mocha takes under multiremote.

2. Cucumber's hooks were not instrumented at all

beforeFeature, beforeScenario, beforeStep, afterStep, afterScenario, afterFeature were missing from BROWSER_CONTEXT_HOOKS — every one has a live driver, so a cucumber suite's hook windows were invisible. Added, and beforeFeature joins PRE_TEST_WINDOW_HOOKS since it precedes the first scenario.

What was actually run

Bench: examples/repro-cucumber/ — feature + steps + two configs, every cucumber lifecycle hook defined.

case	build	result
cucumber, prod 9.35.0	kfribnppwbyt98ob8xc9lftkjvbm3xeroffaqjy1	config-level before() window: 0 scans
cucumber, this branch	4ouu5t3ci52ouoo3pmxcczehbzxatuln4fqhxv08	window: 1 scan; all 7 cucumber hooks + before/after logged start/finish
cucumber, failing before()	tupaf5ev3rr3qclpvq2sx9628hs7ot6lvzev2phd	exit 0, 1 passed — and the only record of the failure is ours
mocha CLI, all 8 hooks defined	(earlier)	every hook instrumented, beforeHook/afterHook 4× each
mocha CLI, failing before() / beforeSuite	qkfnla45… / eq3p9ktf…	hook run result=failed, TRA verdict failed

The cucumber failing path, in full:

ERROR @wdio/utils:shim: Error: BOOM: cucumber config-level before failed
[hook-window] before#0 (bound before) rejected: BOOM: cucumber config-level before failed in 3050ms
Spec Files: 1 passed, 1 total          ← WDIO swallows it, exit code 0

The one asymmetry left, stated plainly

	CLI (mocha)	non-CLI (cucumber, jasmine, mocha-multiremote)
config-level hook window scanned	✅ 7 scans	✅ 1 scan (was 0)
window bounded in logs	✅	✅
failure recorded	✅	✅
hook run reported to TRA	✅ wdio "before" hook, passed/failed	❌ not implemented

Direct flow reports hooks through insights-handler, and its hook state is shared with the real hooks — setCurrentHook is what parents subsequent logs. Injecting a synthetic hook there could mis-parent a customer's logs, so it needs its own design and test pass rather than riding on this change. Scans in that window therefore stay unattributed on the non-CLI path, exactly as they are on main today; what changes is that they now happen.

Unit: 1,160 passing (75 pre-existing failures), lint clean. New coverage includes the classic gate opening in before(), staying closed when autoScanning is off, and being recomputed by a filtered test.

[kamal-kaur04]

Cucumber now reports the hook run too — the asymmetry is closed (0fafe79)

Base branch merged in first (56c8160, a real merge — no rewrite), constants moved to constants.ts, and then the piece that was missing: cucumber gets the hook run.

Direct flow already reports cucumber hooks through insights-handler, so that is where the pre-test window goes rather than inventing a channel. Two small adjustments made it fit:

• getHookRunDataForCucumber now prefers a stored name over deriving one from hookType, so a synthesised hook can carry its own label. TestMeta already had the field; the builder was overwriting it.
• The run is stashed under a fixed key, so its finish can find its start — the same guard the real cucumber hooks rely on.

The accessibility handler drives it, on demand from the scan path and closed at the first scenario, exactly as the CLI module does. It receives the reporter by injection rather than reaching for insights-handler itself, so the handler stays unaware of which framework it serves and the service decides who gets one.

On device

case	hook on the dashboard	build verdict	build
happy	[BEFORE_ALL] wdio "before" hook — passed	passed {passed: 2}	zzppjaz9vfv1tsexw1nkdbuguqyqhfxsyzopgyrm
before() throws	[BEFORE_ALL] wdio "before" hook — failed	failed {passed: 1, failed: 1}	czrlfmfadwt32xjgdtsmzuup46wmutqqftujmje7

The scenario itself reports passed in both — WDIO swallows the hook error regardless — so that hook row is the only place the failure is visible. Which is the whole point of reporting it. The window's scan still lands (1x back) either way.

Parity, updated

	CLI (mocha)	cucumber	jasmine / mocha-multiremote
window scanned	✅	✅	✅
window bounded in logs	✅	✅	✅
failure recorded	✅	✅	✅
hook run reported	✅	✅ new	❌ by decision

Jasmine and mocha-multiremote are left out deliberately: neither reports hooks on a path this can reuse, so each needs its own design rather than a guess. The scanning fix covers them regardless, since it lives in the classic handler's before().

Unit: 1,164 passing (75 pre-existing failures), lint clean. Four new tests for the cucumber lifecycle — opens once, closes at the first scenario, never closes what it did not open, and reports nothing when no reporter is installed.

[kamal-kaur04]

Jasmine and multiremote scoped out — and the premise checked against the code (e3b0d1f)

Both the gate and the hook instrumentation were framework-agnostic, so a jasmine or multiremote run would have picked them up. Now gated on PRE_TEST_SCAN_FRAMEWORKS (mocha, cucumber) and not-multiremote, at the two places that act: the classic handler's before() and the service's call to instrumentBrowserContextHooks.

Verified on device, both directions

Same jasmine bench, with a config-level before() that touches the driver:

	scans, whole session	[hook-window] lines	hook runs	test
prod 9.35.0 — jhiej0xnkfxdigpk29psmzykmldy96mvysjzxirs	0	0	0	passes
this branch — ca39y63f7h2qmuck5tylhcaajlxj2fxhtlkv1vya	0	0	0	passes

Byte-for-byte unchanged, which is the requirement. Bench kept at examples/repro-jasmine/ so it stays a standing control.

But the code does not say what the premise says

Worth recording, because it changes how the exclusion should be read. I checked rather than trusting it, and it contradicts the premise in two places:

// accessibility-handler.ts
/**
 * WDIO's jasmine adapter emits the same service hooks as mocha (SDK-7190);
 * cucumber goes through beforeScenario/afterScenario instead.
 */
private static readonly TEST_HOOK_FRAMEWORKS = ['mocha', 'jasmine']

// util.ts
* handles the three edge cases, and clears the app on conflict [...] There is NO Tier-2 branch:
* every WebdriverIO test framework (mocha/jasmine/cucumber) supports App Automate.

There is no framework gate on App-A11y anywhere in the service, and no multiremote gate outside the launcher's CLI check. beforeTest admits jasmine and sets the scan gate; commandWrapper has no framework check at all.

So "App-A11y is unsupported on jasmine" is a product position, not something the code enforces. The reason jasmine shows 0 scans on prod today is not a guard — it is that nothing provisions the wrapped-command set on that path. That is a meaningfully different situation from being switched off, and if the unsupported status is meant to hold, an explicit guard is the thing that would make it true.

I have not added such a guard: it would change jasmine's behaviour, which is the opposite of the instruction here. Flagging it as its own decision.

Unit: 1,167 passing (75 pre-existing failures), lint clean. Three new tests — jasmine gets no gate, multiremote gets no gate, and jasmine opens no hook run even with a reporter installed.

[kamal-kaur04]

Correction to my earlier comment: multiremote was never scanning, and not by policy

I wrote above that there is "no multiremote gate outside the launcher's CLI check". That is true of explicit guards and misleading as a conclusion — there is a structural one, and it is stronger than a policy flag.

A multiremote browser has no sessionId:

// webdriverio/src/multiremote.ts — MultiRemoteDriver
export class MultiRemoteDriver {
    instances: string[]
    isMultiremote = true as const
    __propertiesObject__: Record<string, PropertyDescriptor>
    // ...no sessionId

// same file, element path
// @ts-expect-error ToDo(Christian): remove eventually
delete client.sessionId

Which is exactly why wdio-runner reads ids per instance for multiremote rather than off the browser:

prev[multiRemoteBrowser.getInstance(browserName).sessionId] = ...
caps[browserName].sessionId = multiRemoteBrowser.getInstance(browserName).sessionId

The service passes this._browser.sessionId straight into AccessibilityHandler.before(), so under multiremote the handler receives undefined. And the entire classic scan gate is keyed on session id:

this._sessionId && AccessibilityHandler._a11yScanSessionMap[this._sessionId] && ...

undefined && is falsy, so commandWrapper can never scan, and beforeTest's if (this._sessionId) never writes. Multiremote a11y is dead by construction — one session id cannot address N sessions.

Two consequences worth stating:

1. Mocha + multiremote never had App-A11y, so nothing is being taken away by scoping it out. The launcher's !isMultiremote on CLI bootstrap only decides which path it takes; the path it lands on cannot scan.
2. Without my explicit guard, the pre-test gate would have written an 'undefined' key into the shared static map. Still no scans — _sessionId is falsy either way — but junk in a process-wide map shared across instances. The guard prevents that, which is a better reason for it than the one I first gave.

Pinned as a test (c79e2ba), because the reason lives three files away from the code that depends on it.

Contrast with jasmine, which is worth keeping distinct: jasmine has a session id and is in TEST_HOOK_FRAMEWORKS, so it is unscanned for a much softer reason (nothing provisions the wrapped-command set on that path). Multiremote cannot scan; jasmine merely does not.

[kamal-kaur04]

Device-verified: global-hook scans no longer carry a test run uuid (178c541)

178c541 went up without a device run behind it, so here it is — pre-fix and post-fix, both flows.

Verification reads the session log, not the SDK log: the scan is a browserstack_executor: {"action":"appAllyScan", arguments: {...}} payload, so the log carries the exact params we sent. work/SDK-7422-app-a11y-fixes/uuid-verdict.py <build> <session> prints a per-scan census — phase (from the bench's PHASE_MARKER annotations), thTestRunUuid, thHookRunUuid, command.

Pre-fix — the defect, mocha/CLI flow

Build rpgacbl0dyjcmmg34kwm3y5xui1bdirosz16ncll · session 5a6695f7c00b047ca90b7d367d580f5f3058bc8f

 #  phase at scan                    test uuid  hook uuid  command
 1  mocha before all STARTED         22ec974c   4ffc67e8   back
 2  mocha before all STARTED         22ec974c   4ffc67e8   back
 3  mocha before each #1 STARTED     22ec974c   157abbac   back
 4  test 1 body STARTED (pre-reload) 22ec974c   -          back
 5  test 1 body STARTED (pre-reload) 22ec974c   -          back
 6  test 1 body STARTED (pre-reload) 22ec974c   -          back

Scans 1–2 run before any test exists and still carry 22ec974c — the very uuid the test body later reports under. That is the mis-attribution: a hook scan claiming to be test 1's.

Post-fix — same conf, same device

Build r89xojas5cvljamjpwrj8x7zcf3pc3umt8qyoptk · sessions 72d8f60ebfd6485d670021fb68f43c4163ee6897 (original) and e60022566902b3f72651c4784448e2d39ff73302 (reloaded)

session 72d8f60e... (pre-reload)
 #  phase at scan                    test uuid  hook uuid  command
 1  mocha before all STARTED         -          d8c4a1e2   back
 2  mocha before all STARTED         -          d8c4a1e2   back
 3  mocha before each #1 STARTED     -          0fc93e9f   back
 4  test 1 body STARTED (pre-reload) 8ce5cfc4   -          back
 5  test 1 body STARTED (pre-reload) 8ce5cfc4   -          back
 6  test 1 body STARTED (pre-reload) 8ce5cfc4   -          back

session e6002256... (post-reload)
 #  phase at scan                    test uuid  hook uuid  command
 1  test 1 body RESUMED              8ce5cfc4   -          click
 …
 8  mocha before each #2 STARTED     847ed76c   0cc91e13   back
 9  test 2 body STARTED              847ed76c   -          click
13  mocha before each #3 STARTED     739f7e4e   4691ebb0   back
14  test 3 body STARTED              739f7e4e   -          back

Window scans lose the test uuid and keep their hook uuid; every test-body scan keeps its own. All 16 post-reload scans still land, so the D2 fix on the base branch is unaffected.

The nuance worth naming

before each #1 also falls inside the window, so it loses its test uuid while #2 and #3 — which run after the window closes — keep theirs (scans 8 and 13 carry both uuids). That asymmetry is real. It costs nothing: those scans keep their own hook run uuid, and hook-run→test parentage is untouched by this commit, which only changed scan params. Narrowing the window to exclude framework beforeEach would need the window to close on the framework hook rather than on the first test, and that would reopen the gap this window exists to close.

Cucumber — no-op, and that is the expected result

	build	window scans	test-body scans
pre-fix	fyiiwkwdl8deghsyanxtq7e0shsrlkdc59xrmg01	no test uuid	stamped
post-fix	yldiivrvnkhf0xi7ttvzekksvxoafxulcrx66gac	no test uuid	stamped

Identical, because the defect was CLI-flow-only: mocha mints TEST_ANALYTICS_ID at instance creation, so the window had a uuid to leak, while the classic path sets it at Listener.testStarted and never had one. The guard is wired on both paths anyway so the classic path cannot regress if that ever moves earlier.