Skip to content

fix: unblock scheduled Android E2E and surface its hidden failures - #7

Merged
mahmoud-elmorabea merged 7 commits into
mainfrom
scheduled-e2e-flake-fixes
Oct 23, 2026
Merged

mahmoud-elmorabea merged 7 commits into
mainfrom
scheduled-e2e-flake-fixes

Conversation

@mahmoud-elmorabea

@mahmoud-elmorabea mahmoud-elmorabea commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Fixes four harness problems hitting the scheduled Android "Maestro SDK E2E" workflow in customerio-android, which checks out this repo's main every weekday.

Why

  • Inbox bell selector. message-inbox failed on all 18 scheduled runs the API lists (2026-09-29 through 2026-10-22). Each fails at "Notifications inbox" is visible. customerio-android #834 (merged 2026-09-09, SDK 4.25.0) made the bell's accessibility label host-supplied. The sample now sets Notifications, or Notifications, N unread while the badge shows. Both forms appear on the same bell in one failing run.
  • Logcat filter. The Android SDK logs every module, geofence included, under the logcat tag [CIO]. The filter only kept CustomerIO, so device.log and device-live.log contained zero SDK lines. Maestro's own logcat from the same run has about 1,600 [CIO] lines.
  • Hidden device drops. When ADB drops the device at the first launchApp, the one-time retry usually turns the job green. The drop was then only visible inside the uploaded evidence.

What changed

  1. flows/message_inbox.yaml: the Android bell selector is now Notifications(, [0-9]+ unread| inbox)?. Text selectors match the whole string, so this covers the current label, the unread-badge label and the pre-4.25 label. It matched exactly the bell node in every captured hierarchy. The iOS branch is unchanged.
  2. e2e, run.sh: added '[CIO]:V' to both logcat captures and kept the existing tags.
  3. scripts/android_device.sh: when the device-offline recovery path triggers, the harness prints a ::warning:: naming the suite and appends a line to $GITHUB_STEP_SUMMARY. The summary step is skipped when that variable is unset. The retry logic is unchanged. tests/test_android_device.sh covers four cases: warning plus summary on trigger, the local fallback, no warning on the non-recovery paths, and the "never reconnected" case.

Verification

  • bash tests/run.sh passes. maestro check-syntax reports OK for the changed flow. Shellcheck shows no new findings compared with main.
  • Logcat filterspec: checked on an API 35 emulator with probe lines. '[CIO]:V' keeps [CIO] lines in both -d and streaming mode. The old filter drops them, and other tags stay silenced.
  • Dispatched customerio-android maestro-e2e.yml runs against this branch. Both used harness commit afb74d2.
    • message-inbox: run 39812858255 passed. The bell matched and was tapped. All later steps passed for the first time since 2026-09-09: overlay row and CTA, dismiss, queue removal, and screenshots inbox_05/inbox_06. device.log has 1,663 [CIO] lines; before this change it had 0.
    • geofence: run 39816438577 passed. device.log has 1,533 [CIO] lines, 260 of them [Geofence].
    • The device stayed online in both runs, so the new warning was not exercised on CI. Unit tests cover it.

Geofence assertion

Why. The final geofence check compared timestamps from two different clocks. mark_time.js recorded the runner's time just before the inside setLocation, and the assert only accepted an activity stamped at or after it. The SDK stamps the transition with the emulator's clock. In 11 passing runs the emulator trailed the runner by 0.11–1.07 s and the ENTER arrived 2–4 s after the move, so the margin was only 1–4 s. On 2026-10-08 (run 37739722962) the backend stored fence 83's ENTER at 1791442727, but the mark was about 1791442729, so every poll rejected it until the 90 s cap. The 2026-08-13 failure (run 31675299152) fits the same timing. This cause is inferred from timing evidence.

What changed.

  1. flows/geofence_basic.yaml, scripts/assert_customer_activity.js: before the move, one read records the customer's existing fence-83 activity ids. None are expected: the customer is new and the start point is about 1.85 km from the fence. The final assert now requires an activity whose id is not in that list, and the runner-clock timestamp is gone. If the baseline read fails, the flow fails with a named reason. Poll bounds are unchanged. mark_time.js had no other user and is removed. The MIN_TIMESTAMP_SECONDS checks in the other flows are unchanged.
  2. An unreadable response now counts as a failed read. That covers a body that doesn't parse, or one whose results or activities isn't an array, in both the customer lookup and the activities read. The baseline fails with baseline_activities_unreadable, and a poll counts it as that attempt's miss and records it.
  3. The job log now keeps the evidence, since it outlives the 14-day artifacts. run.sh prints one >> backend … line per activity baseline or assertion (result, bounds, activities seen) and, on Android, a >> android clock line with the emulator-minus-runner offset.
  4. The App API's geofence activity has no enter/exit field. Its fields are customer_id, customer_identifiers, data, delivery_id, delivery_type (empty), id, timestamp and type. The check doesn't need a direction field: the SDK registers business fences with an initial ENTER trigger only, and between the baseline and the assert the device only moves into the fence.

Verification.

  • bash tests/run.sh passes. The new tests/test_assert_customer_activity.js replays the flow's inside-crossing steps with the 2026-10-08 timestamps. That test fails on the old flow and script (no_matching_activity_after_120) and passes now. A malformed-baseline test fails on the previous commit and passes now. maestro check-syntax reports OK, and shellcheck shows no new findings.
  • Geofence runs against the feature branch passed: run 39830344550 and run 39836829952. Their job logs show the >> backend lines and clock offsets of +0.128 s and −0.269 s. A third run, 39838499983, stalled in "Install Maestro and evidence dependencies" before the flow started and was cancelled.
  • Geofence validation on this PR branch at fd62d5e passed: run 39847715560. The job log shows a baseline with no fence-83 ids and a new fence-83 activity after 9 polls (6,534 ms within the unchanged 90 s bound). The emulator trailed the runner by 0.353 s. device.log includes 1,544 [CIO] lines, including 261 geofence lines.
  • Message-inbox validation on this PR branch at fd62d5e passed: run 39853308733. The job log confirms the harness commit and reports 1/1 Flow Passed in 4m 21s. Evidence sanitization and upload also succeeded.

🤖 Generated with Claude Code

iOS validation

An iOS geofence dispatch against the exact PR head fd62d5e8e798a309e1d1231dff778e8c182678d5 passed: run 39934962637. The flow reports 1/1 Flow Passed in 4m 37s. The backend evidence confirms zero matching activity IDs in the baseline and a new matching geofence activity afterward (one poll, 411 ms, within the unchanged 90-second bound). Evidence sanitization and upload also passed.

mahmoud-elmorabea and others added 7 commits October 22, 2026 20:15
customerio-android #834 (SDK 4.25.0) made the visual inbox bell's
accessibility label host-supplied. The Java sample sets "Notifications",
or "Notifications, N unread" while the badge shows, so the exact
"Notifications inbox" selector has failed every scheduled Android
message-inbox run since 2026-09-10. Match the three label forms as a
whole-string pattern; the iOS branch is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Android SDK logs every module, geofence included, under the logcat
tag "[CIO]", so the CustomerIO-only filter left device.log and
device-live.log without a single SDK line. Keep the existing tags and
add "[CIO]:V" to both captures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When ADB drops the device at the first launchApp, the one-time
reconnect-and-retry usually turns the job green, so the drop was only
visible inside the uploaded evidence. Emit a GitHub Actions warning
naming the suite and add a step-summary line whenever that path
triggers. The retry itself is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The geofence assertion compared the activity timestamp, stamped by the
emulator's clock when the transition arrives, with a mark taken on the
runner's clock just before the inside setLocation. The emulator trails
the runner by up to about a second and the ENTER arrives 2-4 s after the
move, leaving a 1-4 s margin. On 2026-10-08 the backend stored fence
83's ENTER at 1791442727 while the mark was 1791442729, so every poll
rejected it until the 90 s cap.

Take a baseline from the backend instead: before the move, record the
customer's existing fence activity ids (expected none), then require an
activity whose id is not in that list. The baseline is a single read and
fails the flow by name if it cannot be taken. Polling bounds are
unchanged; mark_time.js had no other user and is removed. Other flows'
MIN_TIMESTAMP_SECONDS checks keep their meaning.

run.sh now prints into the job log, which outlives the 14-day
artifacts, one line per activity baseline/assertion (result, bounds,
activities seen) and, on Android, the emulator-minus-runner clock
offset measured over adb.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n ENTER

The Ext API geofence activity has no enter/exit field: the documented
activity fields carry no direction, `name` is absent, and `data` holds
only the fence, geoset, transition id and attributes. Record the matched
activity's field names in the sink and the job log so that stays
checked, and explain in the flow why a new fence activity after the
baseline can only be the ENTER.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first CI run showed the geofence activity also carries delivery_id
and delivery_type, which the App API documents as the recipient device.
Print its value so the job log shows whether it says anything about the
transition direction.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
parse() returned {} for a body that did not parse, and a 200 without an
activities array became an empty list. A baseline taken from such a
response recorded no ids, so a later valid poll could count an activity
that already existed as new.

Require the body to parse and the field to be an array of objects, for
both the customer lookup and the activities read. The baseline fails as
baseline_activities_unreadable; a poll counts it as that attempt's miss
and records it as last_error. Poll bounds are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@Shahroz16 Shahroz16 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, the baseline gets rid of the clock comparison that broke on 2026-10-08. One ask before merging inline.

Comment thread flows/geofence_basic.yaml
return activity.type === TYPE && nameMatches && propertyMatches && property2Matches
}

function isNew(activity) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would a loose timestamp floor next to the baseline make sense, say the mark minus 60 s? As is, any fence-83 activity missing from the baseline passes, including an older one that shows up late or an EXIT, since the activity has no direction. Unlikely with a fresh customer 1.85 km out, so not blocking.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The optional timestamp floor has not been added. The baseline check remains as implemented: a fresh customer, a start outside the target fence, and a move into it. Those constraints reduce the late-activity risk, but the API still does not expose direction, and the iOS validation does not rule out the edge case you describe. Leaving this thread open as a nonblocking follow-up rather than marking the suggestion implemented.

@mahmoud-elmorabea
mahmoud-elmorabea merged commit 7779e85 into main Oct 23, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants