Repository navigation
fix: unblock scheduled Android E2E and surface its hidden failures - #7
Conversation
customerio-android #834 (SDK 4.25.0) made the visual inbox bell's accessibility label host-supplied. The Java sample sets "Notifications", or "Notifications, N unread" while the badge shows, so the exact "Notifications inbox" selector has failed every scheduled Android message-inbox run since 2026-09-10. Match the three label forms as a whole-string pattern; the iOS branch is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Android SDK logs every module, geofence included, under the logcat tag "[CIO]", so the CustomerIO-only filter left device.log and device-live.log without a single SDK line. Keep the existing tags and add "[CIO]:V" to both captures. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When ADB drops the device at the first launchApp, the one-time reconnect-and-retry usually turns the job green, so the drop was only visible inside the uploaded evidence. Emit a GitHub Actions warning naming the suite and add a step-summary line whenever that path triggers. The retry itself is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The geofence assertion compared the activity timestamp, stamped by the emulator's clock when the transition arrives, with a mark taken on the runner's clock just before the inside setLocation. The emulator trails the runner by up to about a second and the ENTER arrives 2-4 s after the move, leaving a 1-4 s margin. On 2026-10-08 the backend stored fence 83's ENTER at 1791442727 while the mark was 1791442729, so every poll rejected it until the 90 s cap. Take a baseline from the backend instead: before the move, record the customer's existing fence activity ids (expected none), then require an activity whose id is not in that list. The baseline is a single read and fails the flow by name if it cannot be taken. Polling bounds are unchanged; mark_time.js had no other user and is removed. Other flows' MIN_TIMESTAMP_SECONDS checks keep their meaning. run.sh now prints into the job log, which outlives the 14-day artifacts, one line per activity baseline/assertion (result, bounds, activities seen) and, on Android, the emulator-minus-runner clock offset measured over adb. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n ENTER The Ext API geofence activity has no enter/exit field: the documented activity fields carry no direction, `name` is absent, and `data` holds only the fence, geoset, transition id and attributes. Record the matched activity's field names in the sink and the job log so that stays checked, and explain in the flow why a new fence activity after the baseline can only be the ENTER. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first CI run showed the geofence activity also carries delivery_id and delivery_type, which the App API documents as the recipient device. Print its value so the job log shows whether it says anything about the transition direction. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
parse() returned {} for a body that did not parse, and a 200 without an
activities array became an empty list. A baseline taken from such a
response recorded no ids, so a later valid poll could count an activity
that already existed as new.
Require the body to parse and the field to be an array of objects, for
both the customer lookup and the activities read. The baseline fails as
baseline_activities_unreadable; a poll counts it as that attempt's miss
and records it as last_error. Poll bounds are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Shahroz16
left a comment
There was a problem hiding this comment.
Looks good, the baseline gets rid of the clock comparison that broke on 2026-10-08. One ask before merging inline.
| return activity.type === TYPE && nameMatches && propertyMatches && property2Matches | ||
| } | ||
|
|
||
| function isNew(activity) { |
There was a problem hiding this comment.
Would a loose timestamp floor next to the baseline make sense, say the mark minus 60 s? As is, any fence-83 activity missing from the baseline passes, including an older one that shows up late or an EXIT, since the activity has no direction. Unlikely with a fresh customer 1.85 km out, so not blocking.
There was a problem hiding this comment.
The optional timestamp floor has not been added. The baseline check remains as implemented: a fresh customer, a start outside the target fence, and a move into it. Those constraints reduce the late-activity risk, but the API still does not expose direction, and the iOS validation does not rule out the edge case you describe. Leaving this thread open as a nonblocking follow-up rather than marking the suggestion implemented.
Fixes four harness problems hitting the scheduled Android "Maestro SDK E2E" workflow in customerio-android, which checks out this repo's
mainevery weekday.Why
message-inboxfailed on all 18 scheduled runs the API lists (2026-09-29 through 2026-10-22). Each fails at"Notifications inbox" is visible. customerio-android #834 (merged 2026-09-09, SDK 4.25.0) made the bell's accessibility label host-supplied. The sample now setsNotifications, orNotifications, N unreadwhile the badge shows. Both forms appear on the same bell in one failing run.[CIO]. The filter only keptCustomerIO, sodevice.loganddevice-live.logcontained zero SDK lines. Maestro's own logcat from the same run has about 1,600[CIO]lines.launchApp, the one-time retry usually turns the job green. The drop was then only visible inside the uploaded evidence.What changed
flows/message_inbox.yaml: the Android bell selector is nowNotifications(, [0-9]+ unread| inbox)?. Text selectors match the whole string, so this covers the current label, the unread-badge label and the pre-4.25 label. It matched exactly the bell node in every captured hierarchy. The iOS branch is unchanged.e2e,run.sh: added'[CIO]:V'to both logcat captures and kept the existing tags.scripts/android_device.sh: when the device-offline recovery path triggers, the harness prints a::warning::naming the suite and appends a line to$GITHUB_STEP_SUMMARY. The summary step is skipped when that variable is unset. The retry logic is unchanged.tests/test_android_device.shcovers four cases: warning plus summary on trigger, the local fallback, no warning on the non-recovery paths, and the "never reconnected" case.Verification
bash tests/run.shpasses.maestro check-syntaxreports OK for the changed flow. Shellcheck shows no new findings compared withmain.'[CIO]:V'keeps[CIO]lines in both-dand streaming mode. The old filter drops them, and other tags stay silenced.maestro-e2e.ymlruns against this branch. Both used harness commitafb74d2.message-inbox: run 39812858255 passed. The bell matched and was tapped. All later steps passed for the first time since 2026-09-09: overlay row and CTA, dismiss, queue removal, and screenshotsinbox_05/inbox_06.device.loghas 1,663[CIO]lines; before this change it had 0.geofence: run 39816438577 passed.device.loghas 1,533[CIO]lines, 260 of them[Geofence].Geofence assertion
Why. The final geofence check compared timestamps from two different clocks.
mark_time.jsrecorded the runner's time just before the insidesetLocation, and the assert only accepted an activity stamped at or after it. The SDK stamps the transition with the emulator's clock. In 11 passing runs the emulator trailed the runner by 0.11–1.07 s and the ENTER arrived 2–4 s after the move, so the margin was only 1–4 s. On 2026-10-08 (run 37739722962) the backend stored fence 83's ENTER at1791442727, but the mark was about1791442729, so every poll rejected it until the 90 s cap. The 2026-08-13 failure (run 31675299152) fits the same timing. This cause is inferred from timing evidence.What changed.
flows/geofence_basic.yaml,scripts/assert_customer_activity.js: before the move, one read records the customer's existing fence-83 activity ids. None are expected: the customer is new and the start point is about 1.85 km from the fence. The final assert now requires an activity whose id is not in that list, and the runner-clock timestamp is gone. If the baseline read fails, the flow fails with a named reason. Poll bounds are unchanged.mark_time.jshad no other user and is removed. TheMIN_TIMESTAMP_SECONDSchecks in the other flows are unchanged.resultsoractivitiesisn't an array, in both the customer lookup and the activities read. The baseline fails withbaseline_activities_unreadable, and a poll counts it as that attempt's miss and records it.run.shprints one>> backend …line per activity baseline or assertion (result, bounds, activities seen) and, on Android, a>> android clockline with the emulator-minus-runner offset.customer_id,customer_identifiers,data,delivery_id,delivery_type(empty),id,timestampandtype. The check doesn't need a direction field: the SDK registers business fences with an initial ENTER trigger only, and between the baseline and the assert the device only moves into the fence.Verification.
bash tests/run.shpasses. The newtests/test_assert_customer_activity.jsreplays the flow's inside-crossing steps with the 2026-10-08 timestamps. That test fails on the old flow and script (no_matching_activity_after_120) and passes now. A malformed-baseline test fails on the previous commit and passes now.maestro check-syntaxreports OK, and shellcheck shows no new findings.>> backendlines and clock offsets of +0.128 s and −0.269 s. A third run, 39838499983, stalled in "Install Maestro and evidence dependencies" before the flow started and was cancelled.fd62d5epassed: run 39847715560. The job log shows a baseline with no fence-83 ids and a new fence-83 activity after 9 polls (6,534 ms within the unchanged 90 s bound). The emulator trailed the runner by 0.353 s.device.logincludes 1,544[CIO]lines, including 261 geofence lines.fd62d5epassed: run 39853308733. The job log confirms the harness commit and reports1/1 Flow Passed in 4m 21s. Evidence sanitization and upload also succeeded.🤖 Generated with Claude Code
iOS validation
An iOS geofence dispatch against the exact PR head
fd62d5e8e798a309e1d1231dff778e8c182678d5passed: run 39934962637. The flow reports1/1 Flow Passed in 4m 37s. The backend evidence confirms zero matching activity IDs in the baseline and a new matching geofence activity afterward (one poll, 411 ms, within the unchanged 90-second bound). Evidence sanitization and upload also passed.