Skip to content

Retry only failed Android E2E flows - #57998

Open
cipolleschi wants to merge 5 commits into
mainfrom
cipolleschi/ci/android-e2e-selective-retries
Open

Retry only failed Android E2E flows#57998
cipolleschi wants to merge 5 commits into
mainfrom
cipolleschi/ci/android-e2e-selective-retries

Conversation

@cipolleschi

@cipolleschi cipolleschi commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Summary:

Android Maestro E2E jobs currently stop after the first flow that exhausts its in-process retries. The workflow-level retry then starts the whole suite again, including flows that already passed.

This changes the retry model so that:

  • each CI attempt runs every selected flow once and continues after individual failures;
  • per-flow passed, failed, and pending state is saved atomically after every flow;
  • cumulative results and attempt counts are shown in the GitHub job summary and uploaded as an artifact;
  • retry workflows download that state and skip flows that already passed, running only failed or unfinished flows;
  • a flavor whose downloaded state is already fully passed skips emulator startup and APK installation entirely;
  • a missing state artifact falls back to running the full suite, so infrastructure failures remain retryable.

The Android E2E timeout is increased from 60 to 90 minutes to give the initial all-flows attempt enough time to finish. The existing three workflow attempts now provide up to three executions per failing flow instead of multiplying workflow retries by in-process retries.

CI follow-ups

  • RNTester Debug completed all 40 flows with 38 passes and 2 failures. Retry 1 downloaded its state, skipped exactly those 38 passing flows, and executed only the two failures, validating the selective retry behavior.
  • RNTester Release became unhealthy after an early flow failure and timed out while later flows were still pending. The state file was being written incrementally, but its artifact upload was inside the timed composite action and was killed by the same timeout. Moved state upload into the outer reusable workflow so if: always() preserves passed, failed, and pending results after an E2E timeout.
  • The next Release retry downloaded the preserved state, skipped its 7 prior passes, reran the remaining flows, and recovered to 40/40. A later matrix retry could still start that completed flavor and fail during an unnecessary APK installation, so retry jobs now skip the entire E2E action when downloaded state is fully passed.
  • Both test_js variants failed because the repository Jest preset throws whenever console.error is called, and the new failure-path unit test intentionally exercised a production error log. The test now mocks that expected log while continuing to assert the aggregate flow failure.
  • RNTester Debug consistently failed image-wide-gamut.yml with exactly 83.974% screenshot similarity and legacy-native-module.yml before the APIs search became visible, while Release passed both immediately. Reproducing Debug with an offline Android emulator showed the actual cause: importing ImageExample.js eagerly starts a Facebook image prefetch, and its promise could reject before the Image Loading Events example attached its rejection callback. That produced an Uncaught (in promise): UnknownHostException LogBox notification, which covered the screenshot and intercepted the APIs tab. The prefetch now attaches a rejection handler immediately while preserving the existing example-level success/failure reporting. No LogBox dismissal or E2E-specific workaround remains.

Changelog:

[INTERNAL] [CHANGED] - Retry only failed or unfinished Android Maestro E2E flows.

Test Plan:

  • git diff --check (passed)
  • node --check .github/workflow-scripts/maestro-android.js (passed)
  • yarn jest .github/workflow-scripts/__tests__/maestro-android-test.js --runInBand --config '{"testEnvironment":"node","transform":{}}' (passed)
  • ruby -ryaml -e "ARGV.each { |file| YAML.parse_file(file) }" .github/actions/maestro-android/action.yml .github/workflows/e2e-android-rntester.yml .github/workflows/e2e-android-templateapp.yml .github/workflows/test-all.yml (passed)
  • yarn prettier --check .github/workflow-scripts/maestro-android.js .github/workflow-scripts/__tests__/maestro-android-test.js .github/actions/maestro-android/action.yml .github/workflows/e2e-android-rntester.yml .github/workflows/e2e-android-templateapp.yml .github/workflows/test-all.yml (passed)
  • Verified the retry-state jq predicate returns false only when every recorded flow has passed.
  • ./node_modules/.bin/prettier --check packages/rn-tester/js/examples/Image/ImageExample.js (passed)
  • Reproduced on a local Android Debug emulator with external DNS unavailable. Before the fix, logcat reported Uncaught (in promise): UnknownHostException and LogBox covered the bottom navigation. After the fix, no unhandled rejection or LogBox appeared, the APIs tab displayed explorer_search, and the wide-gamut example was unobstructed.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 18, 2026
@meta-codesync

meta-codesync Bot commented Aug 18, 2026

Copy link
Copy Markdown

@cipolleschi has imported this pull request. If you are a Meta employee, you can view this in D116463690.

@cipolleschi
cipolleschi force-pushed the cipolleschi/ci/android-e2e-selective-retries branch from 29f35f4 to eab1a5b Compare August 18, 2026 19:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. p: Facebook Partner: Facebook Partner

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant