You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(experimentation): decide the rollout when completing an experiment - #9018
Experiments now store a rollout decision and an optional variant. The completion endpoint validates the decision, applies it to the environment default, removes the live rollout override, and logs completion. Experiment deletion also removes the live rollout override. The API schemas and observability catalogue document the new fields and completion event.
Completing an experiment from the UI will likely fail, because the frontend does not send the now-required rollout decision. Control completions may also change the control value silently. A rare case with no multivariate options can block completion. Update the frontend and resolve these before merge.
- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.93%. Comparing base (d9c857a) to head (f098bfc). ⚠️ Report is 63 commits behind head on main.
The reason will be displayed to describe this comment to others. Learn more.
2 issues found across 11 files
Confidence score: 3/5
completeExperiment in frontend/common/services/useExperiment.ts sends no body, but complete/ now requires rollout_decision, so the web app’s Complete action may fail. Send the decision with the request or keep it optional.
In api/experimentation/models.py, deleting a rollout variant clears rollout_variant but leaves rollout_decision as variant, so completed experiments can report a decision without its variant. Keep those fields consistent when the variant is deleted.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="api/experimentation/models.py">
<violation number="1" location="api/experimentation/models.py:192">
P2: `on_delete=SET_NULL` clears `rollout_variant` when the variant is deleted, but `rollout_decision` stays `variant`. A completed experiment then reports a `variant` decision with no variant, so the record no longer shows which variant won. Decide whether this is intended; if not, clear `rollout_decision` when the variant is removed, or block variant deletion while a completed experiment references it.</violation>
</file>
<file name="api/experimentation/views.py">
<violation number="1" location="api/experimentation/views.py:521">
P1: This makes `rollout_decision` mandatory for `complete/`, but the web app still posts without a body. `completeExperiment` in `frontend/common/services/useExperiment.ts` never sends a decision, so the Complete action in `ExperimentActionDropdown` and `ExperimentDetailHeader` will get a 400 and show 'Failed to complete experiment'. Add the decision to these calls in this PR, or ship the frontend change together with this API change.</violation>
</file>
Tip: Get a Slack DM when cubic finishes reviewing your PRs. Connect Slack
The reason will be displayed to describe this comment to others. Learn more.
P1: This makes rollout_decision mandatory for complete/, but the web app still posts without a body. completeExperiment in frontend/common/services/useExperiment.ts never sends a decision, so the Complete action in ExperimentActionDropdown and ExperimentDetailHeader will get a 400 and show 'Failed to complete experiment'. Add the decision to these calls in this PR, or ship the frontend change together with this API change.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At api/experimentation/views.py, line 521:
<comment>This makes `rollout_decision` mandatory for `complete/`, but the web app still posts without a body. `completeExperiment` in `frontend/common/services/useExperiment.ts` never sends a decision, so the Complete action in `ExperimentActionDropdown` and `ExperimentDetailHeader` will get a 400 and show 'Failed to complete experiment'. Add the decision to these calls in this PR, or ship the frontend change together with this API change.</comment>
<file context>
@@ -500,9 +507,27 @@ def start(self, request: Request, **kwargs: object) -> Response:
@action(detail=True, methods=["post"])
def complete(self, request: Request, **kwargs: object) -> Response:
- return self._transition_status(ExperimentStatus.COMPLETED)
+ serializer = ExperimentRolloutDecisionSerializer(
+ data=request.data, context={"experiment": self.get_object()}
+ )
</file context>
The reason will be displayed to describe this comment to others. Learn more.
P2: on_delete=SET_NULL clears rollout_variant when the variant is deleted, but rollout_decision stays variant. A completed experiment then reports a variant decision with no variant, so the record no longer shows which variant won. Decide whether this is intended; if not, clear rollout_decision when the variant is removed, or block variant deletion while a completed experiment references it.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At api/experimentation/models.py, line 192:
<comment>`on_delete=SET_NULL` clears `rollout_variant` when the variant is deleted, but `rollout_decision` stays `variant`. A completed experiment then reports a `variant` decision with no variant, so the record no longer shows which variant won. Decide whether this is intended; if not, clear `rollout_decision` when the variant is removed, or block variant deletion while a completed experiment references it.</comment>
<file context>
@@ -175,6 +181,19 @@ class Experiment(LifecycleModelMixin, SoftDeleteExportableModel): # type: ignor
+ )
+ rollout_variant = models.ForeignKey(
+ "multivariate.MultivariateFeatureOption",
+ on_delete=models.SET_NULL,
+ related_name="+",
+ null=True,
</file context>
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to GitHub limitations.
⚠️ Outside diff range comments (1)
Major · Send the required rollout decision when completing an experiment. · useExperiment.ts:21-24
frontend/common/services/useExperiment.ts:21-24 Major · Data Integrity & Integration · Heavy lift
Send the required rollout decision when completing an experiment.
The completeExperiment mutation sends a POST with no body. The backend complete action validates ExperimentRolloutDecisionSerializer before it changes the experiment status, and that serializer requires rollout_decision. The request fails validation, and the experiment stays running.
The completion control is reachable for running experiments. The "Mark as Completed" item in ExperimentActionDropdown.tsx and the completion control in ExperimentDetailHeader.tsx both call this mutation. ENABLE_EXPERIMENT_LIFECYCLE hides only the pause action, so it does not block this path.
Two changes are required:
In frontend/common/services/useExperiment.ts, the query must forward a request body. It currently drops everything except environmentId and experimentId. The experimentAction request type must also accept rollout_decision, and variant_id when the decision is variant.
Both completion flows must collect the selected decision and pass it to the mutation. Include variant_id only when the decision is variant.
Keep the backend validation. Sending valid bodies in backend tests does not fix these callers.
Review info
Run configuration
Source mode: Auto
Repository: Unavailable
Branch: Unavailable
File: No configuration file from this repository applied
Commit: Unavailable
Layers: Organization UI
Review profile: ASSERTIVE
Plan: Advanced
Run ID: f2048dd9-0a6c-44b7-b0fe-b1b0041eb318
Commits
Reviewing files that changed from the base of the PR and between fe9b4cd and 8593aab.
Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.
Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 6 remain after this review.
The reason will be displayed to describe this comment to others. Learn more.
Major · Functional Correctness · Quick win
Carry the rollout's control value into the environment default on control completion.
The rollout override stores its own control value from the request (feature_state_value). The serializer does not compare that value with the environment default. A control completion sends an environment default payload with no value, so the default keeps its existing value. After the rollout override is deleted, control identities receive that default value. A rollout value of "B" with a default of "A" therefore changes the control outcome to "A" without any recorded change.
Before deleting the override, read its control value and write that value to the environment default. Alternatively, reject completion when the two values differ. Either way, identities in the control arm must keep the value they received during the experiment.
The reason will be displayed to describe this comment to others. Learn more.
Caution
Some comments are outside the diff and can’t be posted inline due to GitHub limitations.
⚠️ Outside diff range comments (1)
Minor · Omit variants from the environment default when the feature has no… · services.py:666
api/experimentation/services.py:666 Minor · Functional Correctness · Quick win
Omit variants from the environment default when the feature has no multivariate options.
Deleting the last multivariate option changes the feature type to STANDARD. Completing the experiment then builds variants: [] for a control or variant decision, and update_flag passes that list to check_variants. check_variants rejects it with Feature is not multivariate. before any write. The error rolls back the completion, so the experiment cannot be completed.
The empty list is not the failing check. The feature-type check is.
When the feature has no multivariate options, the environment default must omit the variants key. Features that have options must keep their current weighting.
Review info
Run configuration
Source mode: Auto
Repository: Unavailable
Branch: Unavailable
File: No configuration file from this repository applied
Commit: Unavailable
Layers: Organization UI
Review profile: ASSERTIVE
Plan: Advanced
Run ID: 673afb5c-dac2-42ce-8502-aaef382cf605
Commits
Reviewing files that changed from the base of the PR and between 0a76979 and f098bfc.
Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 5 remain after this review.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
apiIssue related to the REST APIfeatureNew feature or request
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thanks for submitting a PR! Please check the boxes below:
docs/if required so people know about the feature.Changes
Contributes to https://linear.app/flagsmith/issue/ENG-663/what-happens-to-rollout-when-an-experiment-ends
complete/now requires arollout_decision(control,variantwithvariant_id, ordisable), written to the environment default.rollout_decision,rollout_variant) and returned by the API.How did you test this code?
Unit tests (100% coverage of changed lines), plus a local end-to-end run of each decision and of deleting a paused experiment.