Repository navigation
Conversation
Firebase Test Lab rejects `Nexus5X`, so no test has run since at least September 2025. Jeremy Massel listed moving to `mac-metal` as one option in AINFRA-3575. The artifact paths change because the old ones pointed at `:aztec` unit test results. --- Generated with the help of Claude Code (Opus 5.5), https://code.claude.com
Connected tests were the only consumer of `firebase.secrets.json`, and they now run on an emulator. That leaves nothing to migrate to `a8c-secrets`, which is the question AINFRA-3575 asked. --- Generated with the help of Claude Code (Opus 5.5), https://code.claude.com
The app declares `WRITE_EXTERNAL_STORAGE` up to API 29 and `READ_EXTERNAL_STORAGE` up to API 32. On API 34 `GrantPermissionRule` therefore fails with `READ_EXTERNAL_STORAGE cannot be granted!`, which failed nearly the whole connected suite before any test body ran. --- Generated with the help of Claude Code (Opus 5.5), https://code.claude.com
This was referenced Nov 3, 2026
`mac-metal` is a small pool of physical machines, so it should not spend time compiling. Day One and Tumblr split the work the same way, and run the prebuilt APKs with Marathon, which `mac-metal` agents already have. Retries stay off so that flaky tests remain visible. --- Generated with the help of Claude Code (Opus 5.5), https://code.claude.com
With Marathon's defaults, `Connected Tests` took 17m 34s, compared to 13m 20s when Gradle also compiled on the same machine. The log shows a video recorded for every test, and the default batching runs tests one at a time. --- Generated with the help of Claude Code (Opus 5.5), https://code.claude.com
Building on the `android` queue and running with Marathon saved no `mac-metal` time, and made the whole build about 3 minutes longer. Compiling there took 18s in build 878, and Marathon's test run took about a minute longer than Gradle's. --- Generated with the help of Claude Code (Opus 5.5), https://code.claude.com
mokagio
changed the title
mokagio
marked this pull request as ready for review
November 3, 2026 05:24
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues block the migration of connected Android tests from Firebase Test Lab to a Buildkite emulator.
No open issues 🎉
What changed in this PR
Restores connected Android tests by replacing Firebase Test Lab with an API 34 ARM64 emulator on Buildkite’s mac-metal queue, removing the Firebase secrets dependency.
Changes:
- Runs connected tests through Gradle and uploads their reports and results.
- Requests storage permission only on API 29 and below.
- Removes the configure plugin and obsolete Firebase secrets configuration.
| File | Description |
|---|---|
build.gradle |
Removes the configure plugin and its dependency. |
app/src/androidTest/kotlin/org/wordpress/aztec/demo/BaseTest.kt |
Skips storage permission requests above API 29. |
.gitignore |
Removes configure-file and Firebase secret ignore rules. |
.configure |
Deletes the Firebase secrets configuration. |
.buildkite/pipeline.yml |
Selects mac-metal and collects connected-test artifacts. |
.buildkite/commands/connected-tests.sh |
Starts an API 34 ARM64 emulator and runs Gradle connected tests. |
Review effort: Balanced
Devs can still have a decrypted copy locally after the Firebase setup goes away. The rule stays until they've had time to clean up, tracked in the secrets allowances cleanup on Linear. --- Generated with the help of Claude Code (Opus 5.5), https://code.claude.com
--- Generated with the help of Claude Code (Opus 5.5), https://code.claude.com
Twinsen81
reviewed
Nov 3, 2026
Twinsen81
left a comment
Twinsen81
left a comment
Contributor
There was a problem hiding this comment.
Two small suggestions. Neither one is blocking.
Comment on lines
+6
to
+8
| start-android-emulator \ | ||
| --system-image "system-images;android-34;google_apis;arm64-v8a" \ | ||
| --device "pixel_5" |
Contributor
There was a problem hiding this comment.
Suggestion, not blocking: keep the emulator logs.
Without --logs-dir, start-android-emulator writes emulator-stdout.txt and emulator-stderr.txt into the AVD directory. That directory is outside the checkout and the pre-exit hook deletes it, so the job cannot upload these files. The test logcat is already in androidTest-results/connected, but only the emulator output shows an emulator crash or adb going offline during the run. These are the likely causes if the step becomes flaky again.
Suggested change
| start-android-emulator \ | |
| --system-image "system-images;android-34;google_apis;arm64-v8a" \ | |
| --device "pixel_5" | |
| start-android-emulator \ | |
| --system-image "system-images;android-34;google_apis;arm64-v8a" \ | |
| --device "pixel_5" \ | |
| --logs-dir "build/emulator-logs" |
Then add - "build/emulator-logs/**/*" to artifact_paths in pipeline.yml.
Twinsen81
approved these changes
Nov 3, 2026
Twinsen81
left a comment
Twinsen81
left a comment
Contributor
There was a problem hiding this comment.
Approving. The two suggestions above are not blocking.
Co-authored-by: Evgenii Plokhov <plokhov@gmail.com>
By default the emulator's stdout and stderr land in the AVD directory, which is outside the checkout and removed by the `pre-exit` hook, so the job cannot upload them. They are what shows an emulator crash or adb going offline if the step becomes flaky. Requested in review: #1121 (comment) --- Generated with the help of Claude Code (Opus 5.5), https://code.claude.com
mokagio
enabled auto-merge
November 3, 2026 23:16
mokagio
deleted the
ainfra-3575-does-aztec-android-need-firebase-secrets-mac-metal
branch
November 3, 2026 23:19
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes AINFRA-3575:
arm64-v8aemulator onmac-metalfirebase.secrets.json, so we no longer track encrypted secrets in the repoDetails from AI agent regarding the investigation that resulted in this solution
BaseTestalso stops requesting storage permission above API 29, which failed nearly the whole suite on API 34.The alternatives, each run with the same
BaseTestfix:mac-metal: 3 of 3 concurrent jobs passed in build 886, about 13 minutes each, on two different agents. Single runs also passed in builds 878 and 884.ERRORbefore running tests and one had a test failure. The cause is not established, and concurrency may have played a part. It also keeps the secret.androidqueue (Explore AINFRA-3575 option 3: emulator on the android queue #1122): passed once in build 881, but took 17m 13s and only works by installing the SDK and a system library in the job. It needs agent image changes to be mergeable.Building the APKs on the
androidqueue and running them with Marathon took longer than plain Gradle (build 883, 16m 48s against 14m 26s), so this stays on Gradle.One tradeoff to be aware of: capacity.
mac-metalis a small pool of physical machines, so running there could clog the queue. But, this repo sees little activity, so it seem like a good way to settle the issue.Test
Connected Testsstep on Buildkite and its uploaded reports.