Skip to content

feat: Add the file-based override source - #529

Merged
kinyoklion merged 12 commits into
feat/overridesfrom
rlamb/overrides-python-file-source
Nov 10, 2026
Merged

kinyoklion merged 12 commits into
feat/overridesfrom
rlamb/overrides-python-file-source

Conversation

@kinyoklion

@kinyoklion kinyoklion commented Sep 25, 2026 •

Copy link
Copy Markdown
Member

Summary

This is the fifth step of the flag overrides port described by the OVERRIDE specification. It is based on the events branch because the phases are stacked; retarget to feat/overrides once that branch merges.

ldclient.integrations.overrides.FileOverrideSourceBuilder is the public entry point. It is passed to datasystem.ConfigBuilder.overrides(...) and builds an override source that reads one or more local files in the file data source document format (optional flags, flagValues, and segments members, JSON or YAML) and supplies each successful load to the SDK's override sink as a full snapshot. flagValues entries expand into full flag definitions, so the layer holds only full entities; as the specification describes for a value-only override, the expansion is a flag that is on and serves its single variation as the fallthrough, so the evaluation reason kind is FALLTHROUGH. The existing file data sources keep their current behavior, including their own flagValues expansion.

Behavior:

  • The file name chooses each file's format: a name that ends in .yaml or .yml is YAML, which needs the pyyaml package, and any other file is JSON. Files must be UTF-8. A YAML file configured without pyyaml is a construction error, since it could never load. An empty JSON file is not a valid document, nor is JSON that is not an object, such as null; an empty YAML file has no entries.
  • Files are combined in the configured order. duplicate_keys_handling is fail by default, which rejects the reload and keeps the previously loaded overrides, or ignore, which keeps the first configured file's entry.
  • A configured file that does not exist contributes no overrides and is not an error: it can be created later, deleting a file removes its overrides, and deleting the last file clears the layer. Writing {} also removes a file's overrides.
  • Change detection is in place before the initial load, so a change made between the two is not missed.
  • A file that exists but cannot be read or parsed fails that whole reload. The last good overrides stay in effect, the failure is logged, and the load is retried after a bounded delay and on the next detected change, so a file observed mid-write recovers on its own.
  • Files can be rewritten in place. While the source finds a file empty during a rewrite, it keeps the overrides in effect until the new content is written. The hold that does this is part of the reloader in feat: Add file data loading code with reload, retry, and polling #525: a file whose content is in effect and that a load finds with no entries is held back until it has stayed so, unwritten, for the retry delay (one second), each file for at most five retry delays. After that, an emptied YAML file removes its overrides and an empty JSON file fails to load and keeps them. A partly written YAML file can still parse, so the docs recommend replacing a YAML file in one operation.
  • change_detection is one of two alternatives. polling (the default) compares modification time and size on an interval, one second by default with a one second minimum; a lower interval is raised to the minimum with a warning, and a value that is not a finite number, or longer than the runtime can wait, is a construction error. watching uses the watchdog package and is a construction error when the package is not installed. Notifications are debounced so a burst from one edit produces one reload.
  • The initial load runs synchronously inside start, which the data system calls during client construction, so an override present at startup takes effect from the first evaluation.
  • Every applied change is logged at Info level, for example Flag overrides in effect: 2 flags, 1 segment (/etc/ld/a.json: 2 flags, 1 segment; /etc/ld/b.json: absent), or Flag overrides: none in effect (...).
  • A builder with no paths is a construction error. Unknown change_detection or duplicate_keys_handling values raise ValueError when set. This is stricter than the Go builder, which treats an unrecognized duplicate keys handling as fail. A path listed more than once is logged at warning level and the repeats are ignored; the first occurrence keeps its place in the order. Paths are compared the way the platform compares file names (os.path.normcase): without regard to case on Windows, exactly elsewhere.

The module is added to the API reference (docs/api-integrations.rst) and every public docstring carries the experimental note. The public docs say what an application needs (the format rule, UTF-8, in-place rewrites, how to remove overrides, replacing YAML in one operation); the hold's mechanics are described in the internal docstrings and here.

Tests cover the builder validation and defaults (including a YAML file without pyyaml), synchronous initial load, YAML, multi-file order and duplicate handling, absent files appearing and disappearing, the Info log lines, both change detection modes, last-good retention across a malformed edit in both modes, the automatic retry with no change signal, close semantics, and an end-to-end run through LDClient where a file is added, changed, and cleared with {} while the client never receives LaunchDarkly data. In-place rewrite tests, for JSON and YAML in watching and polling modes, rewrite the file slowly (truncate, pause, write) and check that the source never supplies a snapshot without the override and that a client evaluating the flag throughout serves the old value and then the new one, never the default; they also cover successive rewrites at the source's own timing, an emptied JSON file keeping the overrides, and an emptied YAML file removing them after the retry delay.

The in-place rewrite handling was reviewed in eight rounds as #560 (now closed, its changes moved here and into #525). The final mutation check ran 59 mutants against test_filedata.py and test_file_override_source.py: 54 fail a test, and the 5 survivors are equivalent in this code. Both test files ran 10 times in a row with no failure, and the harness's flag overrides suite passes against the sync and async test services at the top of the stack.

SDK-3250

Opened from termag


Note

Overview
Adds an experimental file-based flag override source so operators can force flag/segment values from local JSON or YAML files (same format as the file data source) via FileOverrideSourceBuilder, wired through datasystem.ConfigBuilder.overrides(...).

The source loads an initial snapshot synchronously at client startup, merges multiple files with configurable duplicate-key handling, and reloads on change using polling (default) or optional watchdog watching. It tolerates missing files, parse failures (keeps last good overrides with retry), and in-place rewrites without briefly clearing overrides. Applied changes are logged at Info level. API docs for ldclient.integrations.overrides are added; behavior is covered by extensive integration tests including end-to-end LDClient scenarios.

Reviewed by Cursor Bugbot for commit 9a582aa. Bugbot is set up for automated code reviews on this repo. Configure here.

@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-events branch from 042fce2 to 0e55650 Compare September 28, 2026 20:29
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-file-source branch from 021778c to 2074b74 Compare September 28, 2026 20:29
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-events branch from 0e55650 to c79f244 Compare September 30, 2026 20:31
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-file-source branch from 2074b74 to 1950d45 Compare September 30, 2026 20:31
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-events branch from c79f244 to 352d360 Compare October 1, 2026 23:43
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-file-source branch from 1950d45 to 3e47045 Compare October 1, 2026 23:43
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-events branch from 352d360 to 9f4108c Compare October 3, 2026 01:18
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-file-source branch from 3e47045 to cfb781a Compare October 3, 2026 01:18
@kinyoklion

Copy link
Copy Markdown
Member Author

bugbot review

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-events branch from 9f4108c to 051cf7d Compare October 13, 2026 21:02
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-file-source branch from cfb781a to ea4dc19 Compare October 13, 2026 21:02
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-events branch from 051cf7d to 0124739 Compare October 13, 2026 22:40
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-file-source branch from ea4dc19 to 3de11ab Compare October 13, 2026 22:40
@kinyoklion

Copy link
Copy Markdown
Member Author

bugbot review

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-events branch from 0124739 to 224677f Compare October 16, 2026 21:34
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-file-source branch from bb5a108 to 5e42ebc Compare October 16, 2026 21:34
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-events branch from 224677f to 2b471ae Compare October 28, 2026 22:17
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-file-source branch from 5e42ebc to 314dea2 Compare October 28, 2026 22:17
@kinyoklion

Copy link
Copy Markdown
Member Author

bugbot review

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-events branch from 2b471ae to c5a69c9 Compare October 28, 2026 22:40
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-file-source branch from 314dea2 to 4bace41 Compare October 28, 2026 22:40
@kinyoklion

Copy link
Copy Markdown
Member Author

bugbot review

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-events branch from c5a69c9 to 70ada89 Compare October 29, 2026 16:48
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-file-source branch from 99b851f to fac9ec8 Compare October 29, 2026 16:48
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-events branch from 70ada89 to cad8f0b Compare October 29, 2026 17:16
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-file-source branch from fac9ec8 to 0415ec1 Compare October 29, 2026 17:16
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-file-source branch from 1cba56a to d6b022b Compare November 4, 2026 22:55
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-events branch from 28c2e59 to a174ad2 Compare November 4, 2026 22:55
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-file-source branch from d6b022b to a52ba14 Compare November 4, 2026 23:09
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-events branch from a174ad2 to 723eed3 Compare November 4, 2026 23:09
kinyoklion added a commit that referenced this pull request Nov 6, 2026
<!-- TERMAG_PR_BODY_BEGIN -->
## Summary

Adds `ldclient.impl.integrations.files.filedata`, the file reading,
parsing, and merging logic that the file-based override source described
by the OVERRIDE specification is built on. The specification requires
ordered multi-file merging with duplicate keys handling, a reload that
keeps the last good data across a malformed edit and retries, debouncing
of change notifications, a polling change detector as well as a watching
one, and correct handling of a configured file that does not exist yet.
This module provides those pieces; the override source itself is added
in a later PR of the stack.

**The existing file data sources are not changed.** The FDv1 file update
processor (`Files.new_data_source`) and the FDv2 file initializer and
synchronizer (`Files.new_data_source_v2`, `datasystem.file_ds_builder`)
keep their current implementation and behavior in every respect: the
`flagValues` expansion (an on flag with fallthrough variation 0 and
version 1, evaluated with a `FALLTHROUGH` reason), the version fallback
that stamps version 1 on entries without one, the failure messages and
their levels, the FDv2 synchronizer's `OFF` state on a failed initial
load and its `INVALID_DATA` error kind, and the polling and watching
rules. A new test module, `test_file_data_sources_pinned_behavior.py`,
pins each of those behaviors so the override feature stays purely
additive. The baseline test files for the existing sources are
unmodified.

What the new module provides for the override source:

- Document parsing. The file name chooses the parser: a name that ends
in `.yaml` or `.yml`, in any case (including a file named just `.yaml`),
is YAML, which needs `pyyaml`, and any other file is JSON, parsed by the
standard JSON parser even when `pyyaml` is installed. The content is
never inspected to choose a parser. Files must be UTF-8; a leading byte
order mark is ignored, and UTF-16, UTF-32 or invalid UTF-8 fails to
parse. The format decides whether an empty file is valid: an empty or
white-space JSON file is not a document and fails to parse (`the file
has no content, which is not a JSON document`), as does JSON that is not
an object, such as `null`; YAML that is empty, only comments, `---`, or
`null` has no entries. A parse error names the format the name chose.
Flag and segment definitions are decoded into the model classes while
the file is read, so an invalid definition fails that load. The map key
is the key of a flag or segment definition for every purpose: the
definition's own key is set to it, and duplicates are detected by it, so
evaluations report the flag by that key. A definition may omit its
version, which is filled in as 1.
- Merging. Files are combined in the configured order. The duplicate
keys handling is `fail` (the load fails) or `ignore` (the first file's
entry is kept). The result records how many entries each file supplied.
- `Reloader`. Serializes reloads, debounces change signals with a settle
window that each signal extends, keeps the last good data by not
applying a failed load, retries a failed load after a bounded delay so a
file observed mid-write recovers without another notification, reports
an identical failure once, skips an application whose files are
byte-identical to the last applied ones, and applies a success after a
failure even when the contents did not change. A configured file that
does not exist can contribute no data instead of failing the load. A
load reads, parses and classifies every configured file, even after one
fails, then fails with the first failure (logging the others at Debug
level), then holds files back (below), then merges. The skip-unchanged
digest covers each path, whether its file exists, and its bytes, since
the same bytes in a file of another name can be in another format.
Closing does not wait for an in-flight reload. An exception raised by
`apply` or `on_error` is reported like any other load failure and
retried, and nothing from that load is remembered.
- In-place rewrites. Applications and editors rewrite files in place:
they truncate the file, then write the new content, and a load in
between finds it empty. A file whose content is in effect and that a
load finds empty, white space only, or parsed to nothing (YAML with only
a comment or `---`) is held back: the load keeps the current data, logs
at Debug level and reports nothing, and the other files' changes wait
too, since a load applies all files or none. `{}` is a document and is
applied at once. The hold works as follows:
- The file is held until it has stayed so, unwritten, for the retry
delay. Its state (modification time and size, read through symbolic
links) is compared only with itself, never with the clock, and a change
starts the delay over.
- The clock is read after the file's own read, so a slow read of another
file does not change its delay.
- The hold asks for a retry when the first held file is due, plus a
tenth of the retry delay; the write's change signal starts a load
sooner.
- Each file is held for at most five retry delays after its first hold,
so a file kept written while empty, or files emptied in turn, are
applied in the end. A file's bound starts over only at a load that holds
nothing back, whatever that load's outcome.
- Content, even content that cannot be parsed, or a missing file ends a
file's empty state; a failed read keeps it.
- A file whose content is not in effect (a placeholder that was always
empty, a file whose deletion was applied and that is then created empty,
a file whose emptiness was applied) is never held, and with retries off
nothing is held. A file that is deleted and created empty again before a
load applies the deletion, in the same debounce window or while another
file fails or is held, still has its content in effect, so it is held
like a file rewritten in place.
- After the delay, empty YAML is applied with no entries, and empty JSON
fails to parse and keeps the data.
- `Poller`. Compares modification time and size on an interval, so a
same-size rewrite and a same-time size change are both detected, as are
files that appear or disappear.
- `Watcher`. Uses the `watchdog` package. It watches the directory of
each file so an absent file is picked up when it appears, matches the
destination of a move event so a file written by rename is detected,
retries a directory that does not exist yet, and reacts only to
notifications that can change a file's content or presence (the watchdog
inotify mask also reports a file being opened or read). A failed
observer start is retried every second, and a watched directory that is
deleted is watched again once it exists, with one change signaled. A
configured file that is a symbolic link is followed: the directory of
the link's target is watched for the target's path, and the links are
resolved again after every reported change, after a retry sets up
watches, and when a directory that holds a target is deleted, as a
Kubernetes ConfigMap update does. Watching does not follow a link
further along the chain that is pointed elsewhere while its old target
stays, a configured directory that is itself a link and is pointed
elsewhere, or a configured directory that is renamed and replaced; the
documentation points such deployments, like a `current` link to a
release directory, to polling, which is the default. A link that cannot
be resolved, for example because its target's directory does not exist
yet or was deleted, is resolved again on the retry interval until its
target is watched, and one change is then signaled. A relative link is
joined to the real path of the directory that holds it, so a `..` in it
is right when a directory on the way is itself a link.

Not changed: a read that sees a YAML file cut off part way can still
parse, with fewer entries or a different value, and is applied until the
next reload; the override source's docs recommend replacing a YAML file
in one operation.

Tests cover parsing (format by name, empty and non-object JSON, empty
and null YAML, byte order marks and other encodings), merging, loading,
the reloader (initial load, failure retention, debounce coalescing and
window extension, retry with and without further signals,
identical-failure and first-failure reporting, skip-unchanged and
recovery, serialization of concurrent reloads, close semantics, worker
thread lifecycle), the hold (on an injected clock, with a one-hour retry
delay so no real retry fires: the exact delay, restarts on a state
change and through a symbolic link, quick successive rewrites, files
emptied together and in turn, the per-file bound and when it starts
over, failures before and after a held file, slow reads, the retry each
hold asks for; and on the real clock, that the retries fire), the
poller, and the watcher (including symbolic links to files in the same
and in another directory, a link that is replaced, a ConfigMap-style
update, a relative link in a directory reached through another link, and
a target directory that appears late or is recreated), plus the pinned
behavior of the existing sources. The reloader tests rewrite their files
in place with a 30 ms pause, through `ldclient/testing/file_util.py`.

The in-place rewrite handling was reviewed in eight rounds as #560 (now
closed, its changes moved here and into #529): multi-agent and
adversarial reviews of each round, including a randomized end-to-end
stress of about 384 rewrites with no empty snapshot. The final mutation
check ran 59 mutants against `test_filedata.py` and
`test_file_override_source.py`: 54 fail a test, and the 5 survivors are
equivalent in this code (listed in #560). Both test files ran 10 times
in a row with no failure.

Python CI runs for this PR. The PRs above it in the stack get no CI;
they were checked locally.

[SDK-3250](https://launchdarkly.atlassian.net/browse/SDK-3250)


[SDK-3250]:
https://launchdarkly.atlassian.net/browse/SDK-3250?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
<!-- TERMAG_PR_BODY_END -->

<!-- termag: box=i-0575d1c54853058d7 project=Overrides -->
<sub>Opened from termag</sub>

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Overview**
> Introduces **`ldclient.impl.integrations.files.filedata`**, shared
logic for reading JSON/YAML flag/segment files, merging them in order
(`fail` vs `ignore` on duplicate keys), and keeping data fresh via
**`Reloader`** (debounced reloads, failure retention, retries,
skip-unchanged digests, and **hold-back** when an in-effect file is seen
empty mid in-place rewrite), plus **`Poller`** and **`Watcher`**
(watchdog, symlinks, ConfigMap-style dirs) change detectors.
> 
> Adds **`ldclient/testing/file_util.py`** for simulated in-place
rewrites and large test suites: **`test_filedata.py`** for the new
module and **`test_file_data_sources_pinned_behavior.py`** to lock
FDv1/FDv2 behavior without changing those implementations in this diff.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
4c11d73. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
@kinyoklion
kinyoklion marked this pull request as ready for review November 6, 2026 17:43
@kinyoklion
kinyoklion requested a review from a team as a code owner November 6, 2026 17:43
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-file-source branch from a52ba14 to 36ceb3f Compare November 6, 2026 17:53
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-events branch from 723eed3 to 26d9877 Compare November 6, 2026 17:53

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 36ceb3f. Configure here.

Comment thread ldclient/integrations/overrides.py Resolved
Carries the override-affected marking on the evaluation event input, so the
event processor keys on the marking alone and never reads the evaluation
reason. An evaluation marked as override-affected produces no individual
feature event and no debug event, even when the flag requests them, and it is
counted in summary events like any other evaluation. The marking is part of
the summary counter key, so override-affected and ordinary evaluations of the
same flag, variation, and version accumulate into separate counters, and a
counter that aggregates marked evaluations carries the overrideAffected
marker, present only when true, like the unknown marker.

The evaluator passes each prerequisite record's own marking to its event, so
a marked prerequisite record produces no individual event while an unaffected
prerequisite inside a marked evaluation is recorded as usual. The client
passes the top-level marking to the evaluation event, keeps the marking on the
reason when a migration evaluation replaces it with a wrong-type error, and
presents an override-affected flag in the all-flags state with trackEvents
and trackReason false and no debugEventsUntilDate, so a consumer bootstrapped
from the state sends no individual events for it. The async client mirrors
these changes.

The OVERRIDE specification vector runner now also checks the marking each
evaluation contributes to its summary counter.
…ons after the loop closes

When an evaluation raises and the flag definition came from the override layer, the EXCEPTION reason carries the override indicator, the default event record is marked so no individual event is produced, and the all-flags state presents the flag with tracking off. The async data system drops a flag change notification that arrives after the event loop closed instead of raising on the reload thread.
Adds ldclient.integrations.overrides.FileOverrideSourceBuilder, the
file-based override source described by the OVERRIDE specification. The
source reads one or more JSON or YAML files in the file data source document
format, with optional flags, flagValues, and segments members, and supplies
each successful load to the SDK's override sink as a full snapshot.

Files are combined in the configured order. The duplicate keys handling is
fail by default, which rejects the reload and keeps the previously loaded
overrides, or ignore, which keeps the first file's entry. A configured file
that does not exist contributes no overrides, so a file can be created later
and deleting a file removes its overrides. A file that exists but cannot be
read or parsed fails that reload, the last good overrides stay in effect, the
failure is logged, and the load is retried after a bounded delay and on the
next detected change. Change detection is one of two modes: polling, the
default, examines the files once per second by default with a one second
minimum, and watching reacts to file system notifications through the
watchdog package. Watching without the watchdog package and a builder with no
paths are construction errors. The initial load completes during client
construction. Every applied change is logged at Info level with the overrides
in effect and what each configured file supplied.
…l of the file override source

Change detection is set up before the initial load, so an edit made while the files are first read is picked up. Close stops the reloader even when the change detector fails to close. The builder rejects a poll interval that is not a number or not finite.
…ntries as duplicate keys

A path listed more than once would make every entry of that file a duplicate of itself: under the
default fail handling the load failed, and under ignore the first listing won by accident. The
builder now drops the repeats, keeps the first occurrence in its place in the order, and logs a
warning once at build time. This is the behavior the Node SDK has.
…t on Windows

The builder dropped repeated paths by exact string match, so on Windows
a path listed twice in different case survived as two paths, and every
key of that file failed as a duplicate of itself under the default fail
handling. Paths are now compared with os.path.normcase, which ignores
case on Windows and compares exactly elsewhere. The first spelling keeps
its place in the order.
… SDKs

A flagValues entry becomes a flag without client-side availability, so
an all-flags state for client-side SDKs leaves it out even when the flag
it overrides is client-side. A full definition with clientSide is the
way to override such a flag. A test pins both.
The override source keeps the overrides in effect while it finds a file empty
during an in-place rewrite. A YAML file without the pyyaml package is now a
configuration error when the source is built, since the file name chooses the
format and such a file can never load.

The builder docs say that the file name decides the format, that files must be
UTF-8, that files can be rewritten in place, how to remove a file's overrides,
that an empty JSON file is not a valid document, and that a YAML file is best
replaced in one operation because a partly written one can still parse.

End-to-end tests rewrite the files in place, slowly, in watching and polling
modes, for JSON and YAML, and check that neither the source nor a client
evaluating the flag ever goes without the override.
Watching follows edits to the configured files and to the targets of configured links, including a ConfigMap update, but not a link further along the path or a configured directory that is pointed elsewhere or replaced. The change detection documentation now says so and points such deployments to polling.
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-file-source branch from 36ceb3f to 9a582aa Compare November 10, 2026 22:24
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-events branch from 26d9877 to 1588f31 Compare November 10, 2026 22:24
Base automatically changed from rlamb/overrides-python-events to feat/overrides November 10, 2026 23:08
@kinyoklion
kinyoklion merged commit f8b7108 into feat/overrides Nov 10, 2026
7 checks passed
@kinyoklion
kinyoklion deleted the rlamb/overrides-python-file-source branch November 10, 2026 23:09
kinyoklion added a commit that referenced this pull request Nov 10, 2026
<!-- TERMAG_PR_BODY_BEGIN -->
**Requirements**

- [x] I have added test coverage for new or changed functionality
- [x] I have followed the repository's [pull request submission
guidelines](../blob/main/CONTRIBUTING.md#submitting-pull-requests)
- [ ] I have validated my changes against all supported platform
versions
- [x] Sync/async parity: matching changes made to `async_*` siblings (or
N/A). N/A: `LDClient` and `AsyncLDClient` use the same
`FileOverrideSourceBuilder`.

**Related issues**

- Bugbot finding on #529:
#529 (comment)

**Describe the solution you've provided**

`FileOverrideSourceBuilder` accepted one path as a `str`, or a list of
paths. A single `pathlib.Path`, or any other `os.PathLike`, was passed
to `list()`. A path object is not iterable, so the constructor raised
`TypeError: 'PosixPath' object is not iterable`. The Bugbot finding
describes a different failure, a path split into its components that
fails every reload, but that cannot happen. The constructor raises
before a client exists.

The constructor now treats a `str` or an `os.PathLike` as a single path,
and converts every path with `os.fspath`. A path that does not name a
string path raises `TypeError` from the constructor, with a message that
names the value. That includes `bytes` (given alone, in a list, or named
by a path object), an `int`, and `None`. `bytes` and `bytearray` count
as single values, so a bare `bytes` path is named whole instead of being
read as a sequence of integers. The type hint is `Sequence` rather than
`List`, so that type checkers accept a list of path objects but not a
set, whose order is not fixed. The docstring says that paths may be path
objects.

Tests cover:
- a single path object;
- a list of path objects, which already worked;
- a path object whose string form differs from its path, as with
`os.DirEntry`;
- a source that loads a file given as a path object;
- the rejection of each of the values above.

Each of six broken variants of the constructor fails at least one of
these tests, including the original code and a variant that converts
with `str()` instead of `os.fspath`.

**Describe alternatives you've considered**

Leaving the builder to accept only strings, as its type hint said.
`pathlib` is the usual way to build paths, a list of path objects
already worked, and the error did not say what was wrong.

**Additional context**

This PR is stacked on #530, the top of the overrides stack, so that the
approved PRs below it do not change. Retarget it as the PRs below it
merge.

The review of this change found three existing gaps that this PR leaves
alone:
- An empty path resolves to the working directory, and its reads keep
failing.
- In polling mode, a path that contains a NUL character makes the source
fail to start, so the client runs without overrides. In watching mode,
the reload fails and keeps retrying.
- The file data sources treat a tuple of paths differently from a list.

Testing: `mypy ldclient`, `isort --check` and `pycodestyle` pass
locally. The test suite passes (2,111 tests). It was run without the
Redis, DynamoDB and Consul store integration tests, which need servers
that were not running. CI runs only for PRs based on `main` or
`feat/**`, so it will not run on this PR until it is retargeted.
<!-- TERMAG_PR_BODY_END -->

<!-- termag: box=i-0575d1c54853058d7 project=Overrides -->
<sub>Opened from termag</sub>

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Overview**
> Fixes a constructor bug where a **single** `pathlib.Path` (or other
`os.PathLike`) was passed to `list()`, causing `TypeError: 'PosixPath'
object is not iterable`.
> 
> `FileOverrideSourceBuilder` now wraps a lone `str`,
`bytes`/`bytearray`, or `PathLike` as one path, normalizes every entry
with `os.fspath()` (so objects like `os.DirEntry` use their filesystem
path, not `str()`), and stores string paths. Invalid inputs (`bytes`
paths, ints, `None`, path objects that do not resolve to a `str`) fail
fast with an explicit `TypeError`. Type hints use `Sequence` plus
`PathLike` so callers can pass path objects while unordered collections
like `set` are discouraged.
> 
> New tests cover single/list path objects, `__fspath__` behavior,
invalid types, and loading overrides from a path object.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
154b5b8. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants