Skip to content

feat(experimentation): decide the rollout when completing an experiment - #9018

Open
Zaimwa9 wants to merge 6 commits into
mainfrom
feat/experiment-complete-rollout-decision
Open

Zaimwa9 wants to merge 6 commits into
mainfrom
feat/experiment-complete-rollout-decision

Conversation

@Zaimwa9

@Zaimwa9 Zaimwa9 commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

Thanks for submitting a PR! Please check the boxes below:

  • I have read the Contributing Guide.
  • I have added information to docs/ if required so people know about the feature.
  • I have filled in the "Changes" section below.
  • I have filled in the "How did you test this code" section below.

Changes

Contributes to https://linear.app/flagsmith/issue/ENG-663/what-happens-to-rollout-when-an-experiment-ends

  • complete/ now requires a rollout_decision (control, variant with variant_id, or disable), written to the environment default.
  • Completing or deleting an experiment removes its rollout override; the rollout segment is kept.
  • The decision is stored on the experiment (rollout_decision, rollout_variant) and returned by the API.

How did you test this code?

Unit tests (100% coverage of changed lines), plus a local end-to-end run of each decision and of deleting a paused experiment.

@vercel

vercel Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

3 Skipped Deployments
Project Deployment Actions Updated
docs Ignored Ignored Preview Nov 19, 2026 3:16pm UTC
flagsmith-frontend-preview Ignored Ignored Preview Nov 19, 2026 3:16pm UTC
flagsmith-frontend-staging Ignored Ignored Preview Nov 19, 2026 3:16pm UTC

Request Review

@linear-code

linear-code Bot commented Nov 19, 2026

Copy link
Copy Markdown

ENG-663

@coderabbitai

coderabbitai Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Explain Change →

Walkthrough

Experiments now store a rollout decision and an optional variant. The completion endpoint validates the decision, applies it to the environment default, removes the live rollout override, and logs completion. Experiment deletion also removes the live rollout override. The API schemas and observability catalogue document the new fields and completion event.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Merge Risk: Moderate · up to f098b

Completing an experiment from the UI will likely fail, because the frontend does not send the now-required rollout decision. Control completions may also change the control value silently. A rare case with no multivariate options can block completion. Update the frontend and resolve these before merge.

- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@github-actions github-actions Bot added api Issue related to the REST API docs Documentation updates feature New feature or request and removed docs Documentation updates labels Nov 19, 2026
@github-actions github-actions Bot added the docs Documentation updates label Nov 19, 2026
@codecov

codecov Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.93%. Comparing base (d9c857a) to head (f098bfc).
⚠️ Report is 63 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #9018      +/-   ##
==========================================
+ Coverage   98.91%   98.93%   +0.01%     
==========================================
  Files        1707     1714       +7     
  Lines       71794    72682     +888     
==========================================
+ Hits        71014    71905     +891     
+ Misses        780      777       -3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@Zaimwa9
Zaimwa9 marked this pull request as ready for review November 19, 2026 14:41
@Zaimwa9
Zaimwa9 requested review from a team as code owners November 19, 2026 14:41
@Zaimwa9
Zaimwa9 requested review from a team, emyller and gagantrivedi and removed request for a team and emyller November 19, 2026 14:41
@github-actions github-actions Bot added feature New feature or request and removed feature New feature or request docs Documentation updates labels Nov 19, 2026
@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Docker builds report

Image Build Status Security report
ghcr.io/flagsmith/flagsmith-e2e:pr-9018 Finished ✅ Skipped
ghcr.io/flagsmith/flagsmith-api-test:pr-9018 Finished ✅ Skipped
ghcr.io/flagsmith/flagsmith-frontend:pr-9018 Finished ✅ Results ✅
ghcr.io/flagsmith/flagsmith-api:pr-9018 Finished ✅ Results ✅
ghcr.io/flagsmith/flagsmith:pr-9018 Finished ✅ Results ✅
ghcr.io/flagsmith/flagsmith-private-cloud:pr-9018 Finished ✅ Results ✅

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 issues found across 11 files

Confidence score: 3/5

  • completeExperiment in frontend/common/services/useExperiment.ts sends no body, but complete/ now requires rollout_decision, so the web app’s Complete action may fail. Send the decision with the request or keep it optional.
  • In api/experimentation/models.py, deleting a rollout variant clears rollout_variant but leaves rollout_decision as variant, so completed experiments can report a decision without its variant. Keep those fields consistent when the variant is deleted.
Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="api/experimentation/models.py">

<violation number="1" location="api/experimentation/models.py:192">
P2: `on_delete=SET_NULL` clears `rollout_variant` when the variant is deleted, but `rollout_decision` stays `variant`. A completed experiment then reports a `variant` decision with no variant, so the record no longer shows which variant won. Decide whether this is intended; if not, clear `rollout_decision` when the variant is removed, or block variant deletion while a completed experiment references it.</violation>
</file>

<file name="api/experimentation/views.py">

<violation number="1" location="api/experimentation/views.py:521">
P1: This makes `rollout_decision` mandatory for `complete/`, but the web app still posts without a body. `completeExperiment` in `frontend/common/services/useExperiment.ts` never sends a decision, so the Complete action in `ExperimentActionDropdown` and `ExperimentDetailHeader` will get a 400 and show 'Failed to complete experiment'. Add the decision to these calls in this PR, or ship the frontend change together with this API change.</violation>
</file>

Tip: Get a Slack DM when cubic finishes reviewing your PRs. Connect Slack

Turn on auto-fix | Re-trigger cubic

@action(detail=True, methods=["post"])
def complete(self, request: Request, **kwargs: object) -> Response:
return self._transition_status(ExperimentStatus.COMPLETED)
serializer = ExperimentRolloutDecisionSerializer(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: This makes rollout_decision mandatory for complete/, but the web app still posts without a body. completeExperiment in frontend/common/services/useExperiment.ts never sends a decision, so the Complete action in ExperimentActionDropdown and ExperimentDetailHeader will get a 400 and show 'Failed to complete experiment'. Add the decision to these calls in this PR, or ship the frontend change together with this API change.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At api/experimentation/views.py, line 521:

<comment>This makes `rollout_decision` mandatory for `complete/`, but the web app still posts without a body. `completeExperiment` in `frontend/common/services/useExperiment.ts` never sends a decision, so the Complete action in `ExperimentActionDropdown` and `ExperimentDetailHeader` will get a 400 and show 'Failed to complete experiment'. Add the decision to these calls in this PR, or ship the frontend change together with this API change.</comment>

<file context>
@@ -500,9 +507,27 @@ def start(self, request: Request, **kwargs: object) -> Response:
     @action(detail=True, methods=["post"])
     def complete(self, request: Request, **kwargs: object) -> Response:
-        return self._transition_status(ExperimentStatus.COMPLETED)
+        serializer = ExperimentRolloutDecisionSerializer(
+            data=request.data, context={"experiment": self.get_object()}
+        )
</file context>

)
rollout_variant = models.ForeignKey(
"multivariate.MultivariateFeatureOption",
on_delete=models.SET_NULL,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: on_delete=SET_NULL clears rollout_variant when the variant is deleted, but rollout_decision stays variant. A completed experiment then reports a variant decision with no variant, so the record no longer shows which variant won. Decide whether this is intended; if not, clear rollout_decision when the variant is removed, or block variant deletion while a completed experiment references it.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At api/experimentation/models.py, line 192:

<comment>`on_delete=SET_NULL` clears `rollout_variant` when the variant is deleted, but `rollout_decision` stays `variant`. A completed experiment then reports a `variant` decision with no variant, so the record no longer shows which variant won. Decide whether this is intended; if not, clear `rollout_decision` when the variant is removed, or block variant deletion while a completed experiment references it.</comment>

<file context>
@@ -175,6 +181,19 @@ class Experiment(LifecycleModelMixin, SoftDeleteExportableModel):  # type: ignor
+    )
+    rollout_variant = models.ForeignKey(
+        "multivariate.MultivariateFeatureOption",
+        on_delete=models.SET_NULL,
+        related_name="+",
+        null=True,
</file context>

Comment thread api/experimentation/dataclasses.py Resolved
@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor
✅ private-cloud · depot-ubuntu-latest-16 — run #23463 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  1 minute, 4 seconds
commit  f098bfc
info  🔄 Run: #23463 (attempt 1)

🗂️ Previous results
✅ private-cloud · depot-ubuntu-latest-arm-16 — run #23463 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-arm-16)

passed  4 passed

Details

stats  4 tests across 4 suites
duration  36 seconds
commit  f098bfc
info  🔄 Run: #23463 (attempt 1)

✅ oss · depot-ubuntu-latest-16 — run #23463 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  33.2 seconds
commit  f098bfc
info  🔄 Run: #23463 (attempt 1)

✅ oss · depot-ubuntu-latest-arm-16 — run #23463 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-arm-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  35.7 seconds
commit  f098bfc
info  🔄 Run: #23463 (attempt 1)

✅ private-cloud · depot-ubuntu-latest-16 — run #23459 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-16)

passed  3 passed

Details

stats  3 tests across 2 suites
duration  9 seconds
commit  9769c92
info  🔄 Run: #23459 (attempt 1)

✅ private-cloud · depot-ubuntu-latest-arm-16 — run #23459 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-arm-16)

passed  6 passed

Details

stats  6 tests across 5 suites
duration  3.9 seconds
commit  9769c92
info  🔄 Run: #23459 (attempt 1)

✅ oss · depot-ubuntu-latest-16 — run #23459 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  34 seconds
commit  9769c92
info  🔄 Run: #23459 (attempt 1)

✅ oss · depot-ubuntu-latest-16 — run #23460 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  32.9 seconds
commit  0a76979
info  🔄 Run: #23460 (attempt 1)

✅ oss · depot-ubuntu-latest-arm-16 — run #23459 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-arm-16)

passed  2 passed

Details

stats  2 tests across 2 suites
duration  4.4 seconds
commit  9769c92
info  🔄 Run: #23459 (attempt 1)

✅ oss · depot-ubuntu-latest-arm-16 — run #23460 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-arm-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  35.4 seconds
commit  0a76979
info  🔄 Run: #23460 (attempt 1)

✅ private-cloud · depot-ubuntu-latest-arm-16 — run #23456 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-arm-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  1 minute, 5 seconds
commit  8593aab
info  🔄 Run: #23456 (attempt 1)

✅ private-cloud · depot-ubuntu-latest-16 — run #23456 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-16)

passed  7 passed

Details

stats  7 tests across 6 suites
duration  19.4 seconds
commit  8593aab
info  🔄 Run: #23456 (attempt 1)

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Visual Regression

26 screenshots compared. See report for details.
View full report

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to GitHub limitations.

⚠️ Outside diff range comments (1)

Major · Send the required rollout decision when completing an experiment. · useExperiment.ts:21-24

frontend/common/services/useExperiment.ts:21-24
Major · Data Integrity & Integration · Heavy lift

Send the required rollout decision when completing an experiment.

The completeExperiment mutation sends a POST with no body. The backend complete action validates ExperimentRolloutDecisionSerializer before it changes the experiment status, and that serializer requires rollout_decision. The request fails validation, and the experiment stays running.

The completion control is reachable for running experiments. The "Mark as Completed" item in ExperimentActionDropdown.tsx and the completion control in ExperimentDetailHeader.tsx both call this mutation. ENABLE_EXPERIMENT_LIFECYCLE hides only the pause action, so it does not block this path.

Two changes are required:

  • In frontend/common/services/useExperiment.ts, the query must forward a request body. It currently drops everything except environmentId and experimentId. The experimentAction request type must also accept rollout_decision, and variant_id when the decision is variant.
  • Both completion flows must collect the selected decision and pass it to the mutation. Include variant_id only when the decision is variant.

Keep the backend validation. Sending valid bodies in backend tests does not fix these callers.

Review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: f2048dd9-0a6c-44b7-b0fe-b1b0041eb318
Commits

Reviewing files that changed from the base of the PR and between fe9b4cd and 8593aab.

Files selected for processing: 11

  • api/experimentation/dataclasses.py
  • api/experimentation/migrations/0020_add_experiment_rollout_decision.py
  • api/experimentation/models.py
  • api/experimentation/serializers.py
  • api/experimentation/services.py
  • api/experimentation/views.py
  • api/tests/unit/experimentation/test_experiment_views.py
  • api/tests/unit/experimentation/test_services.py
  • docs/docs/deployment-self-hosting/observability/_events-catalogue.md
  • mcp/src/flagsmith_mcp/openapi.json
  • openapi.yaml

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.

Comment thread api/experimentation/views.py Resolved
@github-actions github-actions Bot added the docs Documentation updates label Nov 19, 2026
@github-actions github-actions Bot added feature New feature or request and removed feature New feature or request docs Documentation updates labels Nov 19, 2026

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 5 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

Turn on auto-fix | Re-trigger cubic

Comment thread api/experimentation/views.py Resolved

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: 32de9f48-b41b-4cf2-b550-3d3483459bb7
Commits

Reviewing files that changed from the base of the PR and between 8593aab and 0a76979.

Files selected for processing: 5

  • api/experimentation/dataclasses.py
  • api/experimentation/services.py
  • api/experimentation/types.py
  • api/experimentation/views.py
  • docs/docs/deployment-self-hosting/observability/_events-catalogue.md

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 6 remain after this review.

{"enabled": False}
if rollout_decision.decision == "disable"
else {
"enabled": True,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Major · Functional Correctness · Quick win

Carry the rollout's control value into the environment default on control completion.

The rollout override stores its own control value from the request (feature_state_value). The serializer does not compare that value with the environment default. A control completion sends an environment default payload with no value, so the default keeps its existing value. After the rollout override is deleted, control identities receive that default value. A rollout value of "B" with a default of "A" therefore changes the control outcome to "A" without any recorded change.

Before deleting the override, read its control value and write that value to the environment default. Alternatively, reject completion when the two values differ. Either way, identities in the control arm must keep the value they received during the experiment.

@github-actions github-actions Bot added feature New feature or request and removed feature New feature or request docs Documentation updates labels Nov 19, 2026

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 3 files (changes from recent commits).

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Requires human review: Auto-approval blocked by 2 unresolved issues from previous reviews.

Turn on auto-fix | Re-trigger cubic

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to GitHub limitations.

⚠️ Outside diff range comments (1)

Minor · Omit variants from the environment default when the feature has no… · services.py:666

api/experimentation/services.py:666
Minor · Functional Correctness · Quick win

Omit variants from the environment default when the feature has no multivariate options.

Deleting the last multivariate option changes the feature type to STANDARD. Completing the experiment then builds variants: [] for a control or variant decision, and update_flag passes that list to check_variants. check_variants rejects it with Feature is not multivariate. before any write. The error rolls back the completion, so the experiment cannot be completed.

The empty list is not the failing check. The feature-type check is.

When the feature has no multivariate options, the environment default must omit the variants key. Features that have options must keep their current weighting.

Review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: 673afb5c-dac2-42ce-8502-aaef382cf605
Commits

Reviewing files that changed from the base of the PR and between 0a76979 and f098bfc.

Files selected for processing: 3

  • api/experimentation/views.py
  • api/tests/unit/experimentation/test_experiment_views.py
  • api/tests/unit/experimentation/test_services.py

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 5 remain after this review.

This branch was successfully deployed

1 active (outdated) deployment
Preview – docs — 0a769797 Deployed Nov 19, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api Issue related to the REST API feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant