Skip to content

test: [SDK-5481] add a fault-injection harness for request retries - #1776

Merged
fadi-george merged 10 commits into
mainfrom
fadi/sdk-5481-tests
Oct 15, 2026
Merged

fadi-george merged 10 commits into
mainfrom
fadi/sdk-5481-tests

Conversation

@fadi-george

@fadi-george fadi-george commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

Description

One Line Summary

Add a fault-injection harness to the demo app for watching and checking SDK request retries.

Details

Motivation

Retry and backoff behavior is hard to check by hand: you need specific requests to fail with specific statuses, at the right time, and then measure the gaps between attempts. This harness does that from the demo app, and is what the manual testing in #1775 used. Split out so that PR stays focused on the SDK change.

Scope

Test tooling only, under examples/demo/scripts/retry, plus a .gitignore entry for its state. No SDK or demo app code changes.

  • fault_proxy.py: a mitmproxy addon, listening on loopback only, that fails only requests carrying the SDK's SDK-Version header, by path and status (including Retry-After and dropped connections). The demo's own REST calls pass through.
  • retry-test.sh: starts and stops the proxy, installs and resets the demo on its own simulator, sets fault rules, drives the UI through Maestro flows in flows/, and follows SDK logs.
  • report.py: lists each request's attempts and checks the gaps against the expected backoff window.
  • make_jwt.py: signs an ES256 Identity Verification JWT with openssl, claiming the aliases and email/SMS the SDK will add. With RETRY_TEST_IV_APP_ID and RETRY_TEST_IV_KEY set, tap login <id> and tap update-jwt <id> sign one, and jwt prints one. Ported from the Android harness (test: [SDK-5535] add a fault-injection harness for operation retries OneSignal-Android-SDK#2791).
  • pause-iams flow: an app's in-app message can otherwise take a tap.
  • README.md: setup and scenarios, including an Identity Verification run of the backoff, offline and order scenarios.

report judges retries by the behavior from #1775, now on main.

Testing

Unit testing

None, test tooling only.

Manual testing

Ran the scenarios in the README on an iOS 26.5 simulator against the #1775 build. Results are in that PR's manual testing section.

Identity Verification: ran the backoff, repo executor, offline and order scenarios, plus a 401 in the middle of retries followed by update-jwt, against an app that requires it. Results are in #1775.

Affected code checklist

  • Notifications
    • Display
    • Open
    • Push Processing
    • Confirm Deliveries
  • Outcomes
  • Sessions
  • In-App Messaging
  • REST API requests
  • Public API changes

Checklist

Overview

  • I have filled out all REQUIRED sections above
  • PR does one thing
  • Any Public API changes are explained in the PR details and conform to existing APIs

Testing

  • I have included test coverage for these changes, or explained why they are not needed
  • All automated tests pass, or I explained why that is not possible
  • I have personally tested this on my device, or explained why that is not possible

Final pass

  • Code is as readable as possible.
  • I have reviewed this PR myself, ensuring it meets each checklist item

🤖 Generated with Claude Code

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Multi-model review

Reviewed by Claude Opus 5.5, GPT 5.6 Sol, and Grok 4.7. Test-only harness (762 lines). It models the #1775 executor backoff; against main it is expected that most failed requests wait for the next launch.

Act on

  • Unknown fail route still writes a catch-all rule (Opus, reproduced): read … <<< "$(route_rule …)" swallows die because the failure is in a subshell. Empty path plus re.search("", path) matches every SDK request, so a typo like fail update-usr 503 prints an error and then fails everything.
  • tap always succeeds (3/3): maestro | grep … || true hides Maestro failures, so a missed control can look like an SDK retry bug.
  • No cleanup traps (3/3): start can leave the Mac HTTP(S) proxy pointed at a dead port, and a second start overwrites saved-proxy with those redirected settings. network-flap can leave Wi-Fi off on Ctrl-C.

Consider

  • --add / any rules.json mtime rewrite resets in-memory --times counters (3/3).
  • report.py groups by stripped method+path, so parallel or successive requests (login b/c, session + add-tag) share one backoff counter (3/3).
  • identity-by-sub is listed as a User-executor route, but EXECUTOR_ROUTE requires subscriptions/[^/]+$, so it never appears in report (Opus+Grok).
  • On #1775 the User executor is woken from the 5s poll; report.py only adds poll slack for repo routes, so correct User-executor gaps get LATE (Opus).

Noted

  • Bind mitmdump to 127.0.0.1 except in manual mode.
  • Status 0 + --retry-after records a floor the SDK never sees.
  • Flows hardcode com.onesignal.example while RETRY_TEST_BUNDLE_ID is documented.

Dismissed

  • Missing in-session executor retry on this branch (Grok): expected. This PR is tooling for #1775 and documents current main behavior.
Open in Web View Automation 

Sent by Cursor Automation: PR Reviews

Comment thread examples/demo/scripts/retry/retry-test.sh
Comment thread examples/demo/scripts/retry/retry-test.sh
Comment thread examples/demo/scripts/retry/retry-test.sh
Comment thread examples/demo/scripts/retry/report.py Outdated
@fadi-george
fadi-george changed the base branch from main to fadi/sdk-5481 October 14, 2026 22:06
@fadi-george
fadi-george added this pull request to stack #1777 October 14, 2026 22:22
Base automatically changed from fadi/sdk-5481 to main October 14, 2026 23:04
fadi-george and others added 5 commits October 14, 2026 16:07
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ies by content

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… actions

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reject unknown routes before writing a rule, fail tap when the Maestro flow fails, keep the saved
proxy settings across a failed start, restore Wi-Fi when network-flap is interrupted, keep --times
counts across --add, report identity-by-sub, and use RETRY_TEST_BUNDLE_ID in the flows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@fadi-george

Copy link
Copy Markdown
Collaborator Author

Thanks. Other items from the review:

  • --add resetting --times counts: fixed, rules that didn't change keep their counts.
  • Poll slack for the User executor: already applied to every executor, and the report.py docstring now says so.
  • Loopback bind: already done.
  • Status 0 with --retry-after: now rejected.
  • Hardcoded bundle ID in flows: they use RETRY_TEST_BUNDLE_ID now.

Also rebased on main since #1775 merged.

@abdulraqeeb33 abdulraqeeb33 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes.

  1. Re-issuing the same fail rule keeps an exhausted count, so the command injects nothing while the log still shows the rule as armed.
  2. The attempt after a 400, 401, 403, 404, 409, or 410 is labeled "new attempt after success", and the client retry count is not reset.

Do not subtract slack from the minimum gap. Treat only 200, 201, and 202 as success. A negative or non-finite Retry-After is 60 seconds in the SDK.

Comment thread examples/demo/scripts/retry/fault_proxy.py Outdated
Comment thread examples/demo/scripts/retry/report.py Outdated
…Retry-After rules

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@abdulraqeeb33 abdulraqeeb33 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The three posted items are fixed. Resetting both counters on 401 and 403 is not. A held request keeps its backoff count.

Comment thread examples/demo/scripts/retry/report.py
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@abdulraqeeb33 abdulraqeeb33 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The bearer split is right for routes that drop an unsigned 401. Two leftovers.

Comment thread examples/demo/scripts/retry/report.py
Comment thread examples/demo/scripts/retry/README.md Outdated
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@fadi-george
fadi-george merged commit 13fb912 into main Oct 15, 2026
4 checks passed
@fadi-george
fadi-george deleted the fadi/sdk-5481-tests branch October 15, 2026 23:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants