Skip to content

feat: Retry event delivery and flush on close - #158

Merged
Zaimwa9 merged 6 commits into
mainfrom
openhands/sdk-spec-v0.9.0
Nov 13, 2026
Merged

Zaimwa9 merged 6 commits into
mainfrom
openhands/sdk-spec-v0.9.0

Conversation

@flagsmithsmith

Copy link
Copy Markdown
Contributor

Contributes to Flagsmith/sdk-spec#23

renovate Bot and others added 2 commits November 12, 2026 09:38
Its entry in Flagsmith/sdk-spec's compatibility matrix at v0.9.0: `core events local-evaluation`.
@flagsmithsmith flagsmithsmith Bot added the sdk-spec Bumps of the Flagsmith SDK behavioural spec label Nov 12, 2026
@coderabbitai

coderabbitai Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: e3129dae-42c8-40b7-88d8-6f906bf0d9c1

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@codspeed

codspeed Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 15 untouched benchmarks


Comparing openhands/sdk-spec-v0.9.0 (aff1621) with main (019082b)

Open in CodSpeed

Implement Flagsmith SDK spec v0.9.0.

- Retry transient failures of the events API (connection errors,
  timeouts, 408, 429, 5xx) up to 3 attempts, backing off from 1 second
  with full jitter, and time out each request after 3 seconds.
- Keep a batch that keeps failing for the next flush, ahead of newer
  events, unless its last attempt got a 500. Drop a rejected batch, and
  stop tracking events on 401 or 403.
- Drop the oldest event when the buffer is full, instead of flushing.
- Add `Flagsmith::close()`, which flushes within a timeout and stops
  tracking events.
- Evaluate experiments locally: carry the experiment in the environment
  document's feature state metadata, and the variant key, into identity
  flags, so `getExperimentFlag` records exposures under local evaluation.
- Wire the new spec steps, and the client with both local evaluation and
  events.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@flagsmithsmith

Copy link
Copy Markdown
Contributor Author

Implemented spec v0.9.0 in 7a3c366. CI failed because the SDK lacked the new steps (flushes events, closes the client with a timeout, tracks events "event_{first}" to "event_{last}", and Create the SDK client with local evaluation and events enabled), and lacked the behaviour behind them.

Event delivery (EventProcessor)

  • A flush now retries transient failures (connection errors, timeouts, 408, 429, 5xx). It makes up to 3 attempts. The backoff starts at 1 s, doubles, and uses full jitter. Each request times out after 3 s. PSR-18 has no per-request timeout, so the SDK passes timeout through Guzzle's send() when the client is Guzzle, which is what discovery finds by default. With any other client, the timeout is up to how that client is configured, and the class docblock says so.
  • If every attempt fails transiently, the batch stays buffered and goes out ahead of newer events on the next flush. The exception is a batch whose last attempt got a 500: it is dropped. Any other rejection (e.g. 400) drops the batch without retrying. A 401 or 403 stops event tracking.
  • A full buffer now drops its oldest event. It used to flush when it filled, but the spec's "full buffer drops its oldest event" scenario rules that out.
  • New Flagsmith::close(float $timeoutSeconds = 5) flushes within the deadline: the backoff and request timeouts are cut to fit it. Afterwards the client stops tracking events.
  • Fixed a bug: the default sleeper closure captured $this, and that cycle stopped the destructor from flushing on unset. The closure is now static.

Local evaluation of experiments

  • getExperimentFlag never recorded exposures under local evaluation: the mapper dropped the feature state's metadata.experiment and the variant key from the environment document. Both now reach identity flags. Environment flags still carry no experiment.

Tests

  • Unit tests cover retries, backoff bounds, kept and dropped batches, 401/403, drop-oldest, the Guzzle timeout, and close (including returning within its timeout). They also cover local experiment flags: variant served, control served, and an identity the experiment doesn't enroll. The fixture is shaped like a real Flagsmith 2.287.0 environment document.
  • Locally: 353 PHPUnit tests pass, php-cs-fixer is clean, and the spec passes 24/24 against the pinned Flagsmith stack.

A limitation of the PHP steps: the single-event track steps still flush right after tracking, as before. The harness's process never ends a PHP request, so without that flush the older scenarios that track an event and then check Flagsmith would never see it. This means that in the close scenarios the events are already sent before close() runs. As an extra check, I ran the delivery and shutdown features locally with that flush removed, so that only flushEvents()/close() sent events, and all 11 passed. That variant was a local check only; it is not committed.

Building an `Experiment` from every identity flag's metadata slowed down
`getIdentityFlags` under local evaluation by about 10%.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@flagsmithsmith

Copy link
Copy Markdown
Contributor Author

The spec check passed on 7a3c366, but CodSpeed flagged a 10–14% regression in four getIdentityFlags* benchmarks. The cause was that commit building an Experiment from every identity flag's metadata. Measured by CPU time on base, 7a3c366, and the fix: it cost +9–13%. edadcbd builds it only for flags that carry an experiment, which brings the overhead down to about 2–3%. That remainder is the per-flag variant copy and experiment check that getExperimentFlag needs.

@flagsmithsmith
flagsmithsmith Bot marked this pull request as ready for review November 12, 2026 10:57
@flagsmithsmith
flagsmithsmith Bot requested a review from a team as a code owner November 12, 2026 10:57
@flagsmithsmith
flagsmithsmith Bot requested review from a team and khvn26 and removed request for a team November 12, 2026 10:57

@Zaimwa9 Zaimwa9 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes. Spec passes, but the full retry loop now runs in __destruct, so during an events API outage every request that tracked an event is delayed by seconds. The 3 s timeout also only applies with Guzzle.

Comment thread src/Utils/EventProcessor.php Resolved
Comment thread src/Utils/EventProcessor.php Resolved
Comment thread src/Utils/EventProcessor.php Resolved
- Make a single attempt when the processor is destroyed, the flush
  interval has elapsed, or the buffer is full, so that an events API
  outage holds up an application's request by at most one request
  timeout. `flush()` and `close()` still retry.
- Flush a full buffer when an event is tracked into it, as before,
  unless it holds a batch that failed to send: only then drop its
  oldest event.
- Time out requests through Symfony's `Psr18Client` (6.2 and later)
  too, not only through Guzzle.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@flagsmithsmith

Copy link
Copy Markdown
Contributor Author

Requesting changes. Spec passes, but the full retry loop now runs in __destruct, so during an events API outage every request that tracked an event is delayed by seconds. The 3 s timeout also only applies with Guzzle.

Thanks. All three points are addressed in d61010c, with details on each inline thread:

  • Automatic flushes (destructor, flush interval, full buffer) now make a single attempt, bounded by the 3 s request timeout. Only flush() and close() retry.
  • The request timeout and the close() deadline now also apply to Symfony's Psr18Client (6.2+), checked against a real socket that never answers.
  • A full buffer is flushed, as on main, unless it holds a failed batch. Only in that case is its oldest event dropped.

Lint is clean and the 359 unit tests pass on PHP 8.4 and 8.1. CI runs the spec on this push.

@flagsmithsmith
flagsmithsmith Bot requested a review from Zaimwa9 November 12, 2026 17:18
@Zaimwa9

Zaimwa9 commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

@themis-blindfold review

Comment thread src/Utils/EventProcessor.php Resolved
Comment thread src/Utils/EventProcessor.php Resolved
@themis-blindfold

Copy link
Copy Markdown

⚖️ Themis review: 🟠 Fix before merge

Event delivery gains retries and an explicit shutdown path, but the retry policy handles server responses outside the specified set and the shutdown deadline is not enforced for all supported HTTP clients. The captured CI checks passed, including the SDK specification and PHP-version test jobs.

Area Score
🎯 Correctness 2/5
🧪 Test coverage 3/5
📐 Code quality 3/5
🚀 Product impact 4/5

🟠 Majors

  • src/Utils/EventProcessor.php: Non-retryable server errors are retried and can remain buffered.
  • src/Utils/EventProcessor.php: The close deadline is not enforced for other PSR-18 clients.
📝 Walkthrough
  • Event delivery - adds retry, buffering, authentication failure handling, and explicit close.
  • Local evaluation - maps experiment metadata and variant keys into identity flags.
  • SDK specification - updates the shared workflow and adds event lifecycle steps.
🧪 How to verify
  • Return 500 and 501 from the events API and check that each batch is dropped after one request.
  • Close with a short timeout using a non-Guzzle, non-Symfony PSR-18 client connected to an unresponsive endpoint.
  • Exercise local experiment flags for enrolled and unenrolled identities.
  • Automate: add regression tests for non-retryable 5xx responses and the close deadline across supported clients.

Product take: Reliable event delivery and bounded shutdown matter for experiment data and application latency; this is a substantial SDK improvement once those guarantees hold across clients.

🧭 Assumptions & unverified claims

No unverified assumptions or claims.

PHP was unavailable in the review environment, so the tests were not rerun locally.

May the events arrive before closing time · reviewed at d61010c

Retry 408, 429, 500, 502, 503 and 504, and drop a batch on any other
status at once, such as 501 or 505, instead of retrying every 5xx and
keeping the batch. Document that `close()` cannot cut short a request
already sent through an HTTP client it cannot give a timeout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@flagsmithsmith

Copy link
Copy Markdown
Contributor Author

@themis-blindfold review

Themis raised two majors. Both are answered on their threads, and aff1621 has the changes:

Non-retryable server errors are retried and can remain buffered.

Fixed for 501, 505 and other unlisted statuses: only 408, 429, 500, 502, 503 and 504 are retried now, and any other status drops the batch on the first attempt. 500 is still retried, because spec v0.9.0 changed sdk-spec#23's original "drop on 500" rule (Flagsmith/sdk-spec#24), and its 500, 500, 503 scenario depends on retrying it.

The close deadline is not enforced for other PSR-18 clients.

PSR-18 cannot pass a timeout, so a request already sent through a client other than Guzzle or Symfony 6.2+ can outlast the deadline. No retry or backoff starts after it, and a new test checks that with a plain PSR-18 client. The close() docblocks now state this limitation. Enforcing a hard bound would mean sending events through a curl transport of our own instead of the configured client. I left that choice to @Zaimwa9 on the thread and recommend against it.

Lint is clean and the 364 unit tests pass on PHP 8.4 and 8.1.

@Zaimwa9
Zaimwa9 merged commit 989290a into main Nov 13, 2026
9 checks passed
@Zaimwa9
Zaimwa9 deleted the openhands/sdk-spec-v0.9.0 branch November 13, 2026 13:53
Zaimwa9 pushed a commit that referenced this pull request Nov 16, 2026
🤖 I have created a release *beep* *boop*
---


##
[5.4.0](v5.3.0...v5.4.0)
(2026-11-13)


### Features

* Retry event delivery and flush on close
([#158](#158))
([989290a](989290a))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Releases v5.4.0 of the `flagsmith/flagsmith-php-client` package.

Includes retry logic for event delivery and flushing events on close.

<sup>Written for commit b4d0c10.
Summary will update on new commits.</sup>

<a
href="https://cubic.dev/pr/Flagsmith/flagsmith-php-client/pull/159?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="View guided diff"
src="https://www.cubic.dev/buttons/review-in-cubic-light.svg"></picture></a>
<a
href="https://www.cubic.dev/action/auto-fix/pr/Flagsmith/flagsmith-php-client/159?returnTo=https%3A%2F%2Fgithub.com%2FFlagsmith%2Fflagsmith-php-client%2Fpull%2F159&source=description"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/turn-on-auto-fix-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/turn-on-auto-fix-light.svg"><img
alt="Turn on auto-fix"
src="https://www.cubic.dev/buttons/turn-on-auto-fix-light.svg"></picture></a>

<!-- End of auto-generated description by cubic. -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

sdk-spec Bumps of the Flagsmith SDK behavioural spec

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant