Skip to content

feat: Mark override-affected evaluations in analytics events - #528

Merged
kinyoklion merged 4 commits into
feat/overridesfrom
rlamb/overrides-python-events
Nov 10, 2026
Merged

kinyoklion merged 4 commits into
feat/overridesfrom
rlamb/overrides-python-events

Conversation

@kinyoklion

@kinyoklion kinyoklion commented Sep 25, 2026 •

Copy link
Copy Markdown
Member

Summary

This is the fourth step of the flag overrides port described by the OVERRIDE specification. It is based on the override store branch because the phases are stacked; retarget to feat/overrides once that branch merges.

EventInputEvaluation gains override_affected, set by the client for the top-level evaluation and by the evaluator for each prerequisite record from that record's own marking. The event processor keys on this scalar alone and never reads the evaluation reason: an override-affected evaluation produces no individual feature event and no debug event, even when the flag requests them, while its index event and its summary counting are unchanged. An unaffected prerequisite record inside a marked evaluation is still recorded as usual.

The summary counter key becomes (variation, version, override_affected), so override-affected and ordinary evaluations of the same flag, variation, and version accumulate into separate counters instead of collapsing together. A counter that aggregates marked evaluations carries overrideAffected: true in the summary event; the property is present only when true, like the unknown marker. Both the sync and async event processors share this dispatch and output code.

In the all-flags state, an override-affected flag keeps its value, version, and marked reason but is presented with trackEvents and trackReason off and no debugEventsUntilDate, so a consumer bootstrapped from the state sends no individual events for it. When a migration evaluation of an overridden flag replaces the reason with a wrong-type error, the new reason keeps the marking. The async client mirrors these changes.

The OVERRIDE specification vector runner now also asserts the summaryOverrideAffected field of each vector against the marking the client hands to the event processor.

Tests cover the summarizer key split, the marker in the summary output, suppression of feature and debug events through both real event processors (including a marked prerequisite record and a mixed set of evaluations of one flag), the client-side marking of top-level and prerequisite records through a prerequisite tree, the all-flags tracking fields, the wrong-type reason, and an end-to-end payload check that an override-affected flag appears only in the summary.

SDK-3250

An evaluation that fails with an unexpected exception keeps the marking when the flag came from the override layer, so it produces no individual event and the all-flags state presents it with tracking off.


Note

Overview
Adds an override_affected marker on evaluation events so flag-override evaluations are handled differently in analytics.

Event pipeline: Override-affected evaluations still update summary counters (with overrideAffected: true when marked) and index events, but no individual feature or debug events are emitted even when tracking is on. Summary counter keys are now (variation, version, override_affected) so marked and unmarked evaluations don't share the same bucket.

Clients (sync + async): Evaluation paths pass override_affected into EventFactory; WRONG_TYPE and exception paths preserve the override marking when applicable. all_flags_state clears trackEvents, trackReason, and debugEventsUntilDate for override-affected flags so bootstrapped consumers don't fire individual events. Prerequisite evaluation records carry their own override marking.

Async FDv2: Late override reloads after client close no longer error when the event loop is closed—notifications are dropped with a debug log.

Specification vector runners and new tests cover summarizer/output behavior, event suppression, prerequisite propagation, and end-to-end payloads.

Reviewed by Cursor Bugbot for commit 1588f31. Bugbot is set up for automated code reviews on this repo. Configure here.

@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-override-store branch from 3fcdc80 to 19f88eb Compare September 28, 2026 20:29
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-events branch from 042fce2 to 0e55650 Compare September 28, 2026 20:29
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-override-store branch from 19f88eb to 7e7b3f4 Compare September 30, 2026 20:31
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-events branch from 0e55650 to c79f244 Compare September 30, 2026 20:31
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-override-store branch from 7e7b3f4 to aa28264 Compare October 1, 2026 23:43
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-events branch from c79f244 to 352d360 Compare October 1, 2026 23:43
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-override-store branch from aa28264 to 1c1bd5c Compare October 3, 2026 01:18
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-events branch from 352d360 to 9f4108c Compare October 3, 2026 01:18
@kinyoklion

Copy link
Copy Markdown
Member Author

bugbot review

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-override-store branch from 1c1bd5c to 3cd92be Compare October 13, 2026 21:02
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-events branch from 9f4108c to 051cf7d Compare October 13, 2026 21:02
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-override-store branch from 3cd92be to cbbe302 Compare October 13, 2026 22:40
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-events branch 2 times, most recently from 0124739 to 224677f Compare October 16, 2026 21:34
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-events branch from 224677f to 2b471ae Compare October 28, 2026 22:17
@kinyoklion

Copy link
Copy Markdown
Member Author

bugbot review

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-events branch 3 times, most recently from 70ada89 to cad8f0b Compare October 29, 2026 17:16
@kinyoklion

Copy link
Copy Markdown
Member Author

bugbot review

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit cad8f0b. Configure here.

@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-override-store branch from aa6274b to 8fdc8b5 Compare November 3, 2026 23:56
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-events branch from cad8f0b to b6c14aa Compare November 3, 2026 23:56
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-override-store branch from 8fdc8b5 to 0656fef Compare November 4, 2026 19:20
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-events branch from b6c14aa to 1722a4c Compare November 4, 2026 19:20
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-override-store branch from 0656fef to b8f3c91 Compare November 4, 2026 19:49
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-events branch from 1722a4c to 28c2e59 Compare November 4, 2026 19:49
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-events branch from 28c2e59 to a174ad2 Compare November 4, 2026 22:55
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-override-store branch from b8f3c91 to 03595d1 Compare November 4, 2026 22:55
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-events branch from a174ad2 to 723eed3 Compare November 4, 2026 23:09
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-override-store branch from 03595d1 to c571029 Compare November 4, 2026 23:09
@kinyoklion
kinyoklion marked this pull request as ready for review November 6, 2026 17:43
@kinyoklion
kinyoklion requested a review from a team as a code owner November 6, 2026 17:43
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-override-store branch from c571029 to 2fc0ad0 Compare November 6, 2026 17:53
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-events branch from 723eed3 to 26d9877 Compare November 6, 2026 17:53
kinyoklion added a commit that referenced this pull request Nov 10, 2026
<!-- TERMAG_PR_BODY_BEGIN -->
## Summary

This is the third step of the flag overrides port described by the
OVERRIDE specification. It is based on the model marker branch because
the phases are stacked; retarget to `feat/overrides` once that branch
merges.

**The override layer** (`ldclient/impl/overrides`). `OverrideLayer` is
the override store: it holds marked shallow copies of the flag and
segment definitions a source supplies, keyed by key, and is replaced
wholesale on each update, so the layer's contents are exactly one
snapshot at any instant and an empty snapshot clears it. Dictionaries
are decoded with the model constructors, and an invalid definition
raises rather than being defaulted. A definition whose key differs from
its key in the snapshot raises too, and the layer is left unchanged:
evaluation reads an entry by its snapshot key, while the all-flags state
and events use the definition's key. `OverrideStoreView` is the overlay
at the store read boundary: a read of a flag or segment returns the
override entry when one exists and the LaunchDarkly entry otherwise,
whatever the state of the base store, and an enumeration is the union of
both with the override entry winning, including over a deleted item. A
failure of the base enumeration propagates, as it does without the
overlay. When that read fails, `all_flags_state` logs the failure as
before and, if the override layer holds flags, returns a state of those
flags marked invalid rather than an empty one: overrides only add to the
state, and a client-side SDK bootstrapped from it uses its flags
whatever `$valid` says, so the overridden values still reach it. With no
override flags the state is the same invalid, empty state as without a
source. The sink stops accepting snapshots when the client is closed,
and when the source fails to start it also clears what the source
supplied. `OverrideSinkImpl` applies each snapshot as one serialized
operation and notifies flag change listeners of every flag whose
merged-view evaluation may have changed: entries added, removed, or
changed, plus every flag that depends on them through prerequisites or
segment references. An identical snapshot notifies nothing.

**LaunchDarkly changes reach override dependents.** The layer keeps a
dependency graph of its entries, rebuilt on each replacement, and the
store (sync and async) expands every changed LaunchDarkly item through
both its own graph and the layer's, to a fixpoint. A change to a
prerequisite flag or a segment that an override entry references
therefore notifies that entry's flag, the same way dependent flags are
notified for LaunchDarkly data, and a chain may cross between
LaunchDarkly data and overrides any number of times. Without an override
source the store's behavior is unchanged.

**Configuration and wiring.** `OverrideSource` and `OverrideSink` are
public protocols in `ldclient.interfaces`, so a custom source can be
supplied. `OverrideSourceBuilder` and `DataSystemConfig.override_source`
(also on `AsyncDataSystemConfig`) plus `ConfigBuilder.overrides(...)`
make the source an option of the FDv2 data system. The source is built
with the data system, so invalid configuration raises from the client
constructor like any other invalid component configuration. It starts
before the data system's own threads and its initial load completes
before the constructor returns; it is closed when the client is closed,
after the sink stops accepting snapshots, and an exception from its
`close` is logged. An exception from its `start` does not fail the
constructor or `start`: the data system logs it, closes the source,
clears any snapshot the source supplied before failing, and the client
runs without overrides. The SDK's own file-based source does not raise
from `start`; a custom source could. The source is not built or started
when the client is offline, and the SDK logs a warning that it is not
used. The data system's `store` becomes the overlay when a source is
configured, so evaluation, prerequisite and segment resolution, and the
all-flags read all see override precedence. The override source has no
effect on initialization status, data availability, or data source
status. The legacy FDv1 data system reports no override source. When the
data system fails to start, the client constructor stops each component
in turn and logs any failure to stop, so the event processor still stops
and the caller receives the start failure.

**The client facade.** Both `LDClient` and `AsyncLDClient` consult the
override layer before the not-initialized short-circuit: an overridden
flag is served before the client has LaunchDarkly data, and a flag the
layer does not hold takes exactly the path it took before this feature,
so a configured but empty source changes nothing with the same warning
and event as before. The all-flags state reads through the overlay.
While the client has no LaunchDarkly data it contains only the flags the
layer holds, read from the layer rather than the overlay so that
LaunchDarkly data in a store without an init marker stays out (with a
once-per-client warning), and it is invalid when the layer holds no
flags. Without a source this state is empty, because the SDK reports no
flags before the store is initialized, so the override flags only add to
what it reports; overrides never remove a flag from the state. An
evaluation that fails with an unexpected exception reports the
`EXCEPTION` reason with `overrideAffected` when the flag came from the
override layer, in a single evaluation and in the all-flags state
(OVERRIDE 1.5.4); the marker is read through `is_override_definition`,
so the error path cannot fail on an unexpected store item. On the async
client, override change notifications are marshalled onto the event
loop, like every other change notification.

**The async client with a persistent store.** The async client works out
which flags an override change affects from the in-memory store, the
only store it can read synchronously (`async_fdv2.py`). While a
persistent store is the active store, a change to an override notifies
the overridden flags, but not LaunchDarkly flags that depend on them
through a prerequisite or a segment. The documentation of
`AsyncDataSystemConfig.override_source` says so.

**Left to later PRs in the stack.**
- `all_flags_state` still keeps `trackEvents`, `trackReason` and
`debugEventsUntilDate` for override-affected flags (OVERRIDE 1.4.1.1).
The events step, #528, clears them.
- An override entry marked as deleted is rejected by #574, which is
stacked on top of the stack (launchdarkly/sdk-specs#288).

**Tests.** Unit tests for the layer, overlay (sync and async), and sink;
client tests for the gate, source lifecycle, construction errors,
offline, precedence, prerequisite and segment overrides, all-flags
state, and flag change and flag value change listeners, for both
clients; and a runner for the OVERRIDE specification test vectors
(`ldclient/testing/testdata/override-vectors/vectors.json`) through the
full client stack. The vector runner's per-evaluation summary marker
assertion is added with the events step.

[SDK-3250](https://launchdarkly.atlassian.net/browse/SDK-3250)


[SDK-3250]:
https://launchdarkly.atlassian.net/browse/SDK-3250?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Overview**
> Adds **experimental flag/segment overrides** that can take precedence
over LaunchDarkly data at evaluation time, wired through FDv2 (sync and
async).
> 
> **Configuration:** Optional `override_source` on the data system
(`ConfigBuilder.overrides(...)`, `OverrideSourceBuilder`, public
`OverrideSource` / `OverrideSink` protocols). Offline clients skip
building/starting the source.
> 
> **Runtime:** An `OverrideLayer` holds snapshot replacements;
`OverrideStoreView` overlays the feature store so per-key reads prefer
overrides; `OverrideSinkImpl` applies updates and fans out **flag
change** notifications via merged-view dependency diffing. The store can
register the layer as **external dependencies** so LaunchDarkly
prerequisite/segment updates also notify affected override flags.
> 
> **Clients:** `LDClient` / `AsyncLDClient` read the override layer
**before** the not-initialized default path and adjust
**`all_flags_state`** to return override-only bootstrapping when LD data
is unavailable (with once-per-client warnings). Evaluation errors set
**`overrideAffected`** when the definition came from overrides. Sync
client **rolls back components** if data system start fails.
> 
> **Tests:** Layer/overlay/sink units, broad sync+async client
scenarios, store cross-graph notification, and OVERRIDE **spec vectors**
(`override-vectors/vectors.json`).
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
0489514. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

<!-- TERMAG_PR_BODY_END -->

<!-- termag: box=i-0575d1c54853058d7 project=Overrides -->
<sub>Opened from termag</sub>
Base automatically changed from rlamb/overrides-python-override-store to feat/overrides November 10, 2026 22:18
Carries the override-affected marking on the evaluation event input, so the
event processor keys on the marking alone and never reads the evaluation
reason. An evaluation marked as override-affected produces no individual
feature event and no debug event, even when the flag requests them, and it is
counted in summary events like any other evaluation. The marking is part of
the summary counter key, so override-affected and ordinary evaluations of the
same flag, variation, and version accumulate into separate counters, and a
counter that aggregates marked evaluations carries the overrideAffected
marker, present only when true, like the unknown marker.

The evaluator passes each prerequisite record's own marking to its event, so
a marked prerequisite record produces no individual event while an unaffected
prerequisite inside a marked evaluation is recorded as usual. The client
passes the top-level marking to the evaluation event, keeps the marking on the
reason when a migration evaluation replaces it with a wrong-type error, and
presents an override-affected flag in the all-flags state with trackEvents
and trackReason false and no debugEventsUntilDate, so a consumer bootstrapped
from the state sends no individual events for it. The async client mirrors
these changes.

The OVERRIDE specification vector runner now also checks the marking each
evaluation contributes to its summary counter.
…ons after the loop closes

When an evaluation raises and the flag definition came from the override layer, the EXCEPTION reason carries the override indicator, the default event record is marked so no individual event is produced, and the all-flags state presents the flag with tracking off. The async data system drops a flag change notification that arrives after the event loop closed instead of raising on the reload thread.
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-events branch from 26d9877 to 1588f31 Compare November 10, 2026 22:24
@kinyoklion
kinyoklion merged commit 89dcfad into feat/overrides Nov 10, 2026
16 checks passed
@kinyoklion
kinyoklion deleted the rlamb/overrides-python-events branch November 10, 2026 23:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants