Skip to content

fix(api): block multivariate option changes during live experiments - #8917

Open
TejasLamba2006 wants to merge 6 commits into
Flagsmith:mainfrom
TejasLamba2006:fix/restrict-mv-options-in-live-experiment
Open

TejasLamba2006 wants to merge 6 commits into
Flagsmith:mainfrom
TejasLamba2006:fix/restrict-mv-options-in-live-experiment

Conversation

@TejasLamba2006

Copy link
Copy Markdown

Thanks for submitting a PR! Please check the boxes below:

  • I have read the Contributing Guide.
  • I have added information to docs/ if required so people know about the feature.
  • I have filled in the "Changes" section below.
  • I have filled in the "How did you test this code" section below.

Changes

Contributes to #8766

The API now rejects multivariate option changes on a feature that is in a running experiment:

  • MultivariateFeatureOptionViewSet create, update and destroy return 400 while the feature has a running experiment in any environment. Multivariate options belong to the feature, so one experiment locks them everywhere.
  • Feature create/update: the nested multivariate_options are rejected only when the submitted options differ from the stored ones. The frontend resubmits the whole feature on save, so other fields (description, tags, etc.) still save on a frozen feature.
  • New helper has_live_experiment in experimentation/feature_state_metadata.py.

Frontend: the create-feature modal already disables the add, edit and remove variant controls when the feature is frozen (useFeatureExperimentFreeze). That check only looks at experiments running in the current environment. Blocking variant edits from the UI for a feature running in another environment needs a project-level experiments query, which is not part of this PR. The API rejects those edits regardless.

How did you test this code?

Backend unit tests:

cd api
pytest tests/unit/features/multivariate/test_unit_multivariate_live_experiment.py
pytest tests/unit/features/ -k "multivariate or mv or feature_serializer or create_feature or update_feature"

The new test module covers create, update and delete of MV options during a live experiment (400), the same create still working without one, and feature PATCH with changed options (400) versus unchanged options plus a metadata change (200). The existing multivariate and feature suites pass.

Multivariate options were only restricted in the frontend. The API
accepted create, update and delete on the MV options endpoint, and
changed nested multivariate_options on feature update, even when the
feature was in a running experiment.

Reject these writes with a 400 while any environment has a running
experiment for the feature. On feature update, only reject when the
submitted options differ from the stored ones, so other feature fields
can still be saved.

Closes Flagsmith#8766
@TejasLamba2006
TejasLamba2006 requested a review from a team as a code owner November 9, 2026 16:45
@TejasLamba2006
TejasLamba2006 requested review from emyller and removed request for a team November 9, 2026 16:45
@vercel

vercel Bot commented Nov 9, 2026

Copy link
Copy Markdown

@TejasLamba2006 is attempting to deploy a commit to the Flagsmith Team on Vercel.

A member of the Team first needs to authorize it.

@coderabbitai

coderabbitai Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

Explain Change →

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • Resume reviews
  • Trigger review

No actionable comments were generated in the recent review.

Walkthrough

The change adds checks that prevent multivariate variation changes while an experiment is running or paused. The checks cover direct option operations, feature updates, and environment imports. Transactions and feature-row locks are added to relevant write paths, and starting an experiment locks its feature row. Tests cover rejected and permitted operations, imports, and cross-environment cases. Documentation updates the experiment prerequisite and six event source line references.

Estimated code review effort: 4 (Complex) | ~35 minutes

Merge Risk: Low · previous review 8682e

The change blocks multivariate option edits during running or paused experiments, and its main paths are covered by tests. Some narrow race windows and a pre-existing cross-project authorization gap remain. These are bounded concerns and are worth owner awareness rather than blocking the merge.

- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: 1fcd5ea6-ea04-4540-b8d3-ff5767eb5c94
Commits

Reviewing files that changed from the base of the PR and between a91b390 and 8682e20.

Files selected for processing: 3

  • api/experimentation/feature_state_metadata.py
  • api/features/import_export/tasks.py
  • api/tests/unit/features/import_export/test_unit_features_import_export_tasks.py

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

Review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: 8a697a32-dc99-42e6-a773-d976a4f84f00
Commits

Reviewing files that changed from the base of the PR and between 13539af and ffd2b0b.

Files selected for processing: 5

  • api/experimentation/feature_state_metadata.py
  • api/features/constants.py
  • api/features/multivariate/views.py
  • api/features/serializers.py
  • api/tests/unit/features/multivariate/test_unit_multivariate_live_experiment.py

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 4 remain after this review.

Comment thread api/features/multivariate/views.py Resolved
Comment thread api/features/multivariate/views.py Resolved
Comment thread api/features/serializers.py Resolved
Comment thread api/features/serializers.py Resolved

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

Review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: 8a0c27d7-d8b4-4329-8549-44983957b743
Commits

Reviewing files that changed from the base of the PR and between ffd2b0b and 5b4a677.

Files selected for processing: 6

  • api/experimentation/feature_state_metadata.py
  • api/features/import_export/services.py
  • api/features/multivariate/views.py
  • api/features/serializers.py
  • api/tests/unit/features/import_export/test_unit_features_import_export_tasks.py
  • api/tests/unit/features/multivariate/test_unit_multivariate_live_experiment.py

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 1 remain after this review.

Comment thread api/features/import_export/services.py Resolved
Comment thread api/features/import_export/services.py Resolved
Comment thread api/features/multivariate/views.py Resolved
Comment thread api/features/serializers.py Resolved

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 5

Review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: bff86148-a732-4af1-85f1-7eebf38db057
Commits

Reviewing files that changed from the base of the PR and between 5b4a677 and d564c8d.

Files selected for processing: 7

  • api/experimentation/services.py
  • api/features/import_export/services.py
  • api/features/import_export/tasks.py
  • api/features/multivariate/views.py
  • api/features/serializers.py
  • api/tests/unit/features/import_export/test_unit_features_import_export_tasks.py
  • api/tests/unit/features/multivariate/test_unit_multivariate_live_experiment.py

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 5 remain after this review.

Comment on lines +620 to +622
Feature.objects.select_for_update().filter(
pk=experiment.feature_id
).exists()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor · Functional Correctness · Quick win

Code analysis · TLA+ model checking

Script executed:

rg -n 'VALID_STATUS_TRANSITIONS|started_at|ExperimentStatus.PAUSED|transition_experiment_status' api/experimentation; sed -n '605,632p' api/experimentation/services.py

Repository: Flagsmith/flagsmith

Length of output: 3106


tla executed:

---- MODULE ReviewModel ----
EXTENDS Naturals
(* Assumption to verify: resuming a paused experiment retains started_at,
   so the new feature lock is skipped. Writer phases represent its feature
   lock, live check, and save. A violation records a save while running.
TLC
INIT Init
NEXT Next
INVARIANT Safe
*)
VARIABLES live, owner, writer, resumed, violation
vars == <<live, owner, writer, resumed, violation>>
Init == /\ live = FALSE /\ owner = "none" /\ writer = "waiting"
        /\ resumed = FALSE /\ violation = FALSE
Check == /\ writer = "waiting" /\ owner = "none"
         /\ owner' = "writer"
         /\ writer' = (IF live THEN "rejected" ELSE "ready")
         /\ UNCHANGED <<live, resumed, violation>>
Resume == /\ ~resumed /\ live' = TRUE /\ resumed' = TRUE
          /\ UNCHANGED <<owner, writer, violation>>
Save == /\ writer = "ready" /\ writer' = "done"
        /\ owner' = "none" /\ violation' = live
        /\ UNCHANGED <<live, resumed>>
Reject == /\ writer = "rejected" /\ writer' = "done"
          /\ owner' = "none" /\ UNCHANGED <<live, resumed, violation>>
Idle == /\ writer = "done" /\ resumed /\ UNCHANGED vars
Next == Check \/ Resume \/ Save \/ Reject \/ Idle
Safe == ~violation
====

Length of output: 3972


Script executed:

git diff 13539afcb99c20b4351e26f215d01b49f2a9be03 d564c8d053d14408261767f8c48432435b3adc51 -- api/experimentation/services.py api/features/multivariate/views.py api/features/serializers.py
nl -ba api/features/multivariate/views.py | sed -n '1,180p'
nl -ba api/experimentation/models.py | sed -n '125,220p'
nl -ba api/experimentation/views.py | sed -n '445,515p;580,655p'
nl -ba api/experimentation/services.py | sed -n '570,645p'
rg -n -F -e 'has_live_experiment' -e 'live_experiment' -e 'transition_experiment_status(' -- api/experimentation api/features api/tests/unit/features/multivariate

Repository: Flagsmith/flagsmith

Length of output: 33851


Script executed:

nl -ba api/experimentation/feature_state_metadata.py | sed -n '1,130p'
rg -n -F -e 'def enable_experiment_rollout' -e 'def _enable_experiment_rollout' -e 'LIVE_EXPERIMENT' -- api/experimentation api/features
rg -n -F -e 'paused' -e 'PAUSED' -- api/tests/unit/features/multivariate/test_unit_multivariate_live_experiment.py api/tests/unit/experimentation

Repository: Flagsmith/flagsmith

Length of output: 4823


Script executed:

nl -ba api/experimentation/services.py | sed -n '1110,1235p'
nl -ba api/features/multivariate/models.py | sed -n '1,240p'
nl -ba api/features/multivariate/serializers.py | sed -n '1,220p'
rg -n -F -e 'multivariate_feature_option' -e 'multivariate_options' -- api/experimentation/services.py api/experimentation/serializers.py
nl -ba api/tests/unit/experimentation/test_experiment_views.py | sed -n '550,620p'
rg -n -F -e 'EXPERIMENTATION_ENABLED' -- api/app/settings/common.py api/experimentation/permissions.py

Repository: Flagsmith/flagsmith

Length of output: 24983


Script executed:

nl -ba api/experimentation/services.py | sed -n '400,465p;900,950p;990,1050p'
nl -ba api/experimentation/permissions.py | sed -n '1,80p'
nl -ba api/tests/unit/experimentation/test_experiment_views.py | sed -n '619,700p'
rg -n -F -e 'multivariate_feature_option.get' -e 'multivariate_feature_option.' -- api/features api/evaluation
rg -n -F -e 'EXPERIMENTATION_ENABLED' -- api/app/settings docker-compose.yml docker-compose.yaml helm charts 2>/dev/null || test "$?" -eq 1

Repository: Flagsmith/flagsmith

Length of output: 13459


Script executed:

rg -n -F -e 'def create_experiment_audit_log' -e 'class AuditLog(' -e 'def apply_experiment_rollout' -- api/experimentation api/audit
nl -ba api/experimentation/services.py | sed -n '515,575p;1030,1110p'
nl -ba api/features/versioning/schemas.py | sed -n '1,85p'
nl -ba docker-compose.yml | sed -n '20,45p'
rg -n -F -e 'get_experiment_rollout' -e '_expected_variant_shares' -- api/experimentation/services.py

Repository: Flagsmith/flagsmith

Length of output: 10425


Web query:

PostgreSQL foreign key update unchanged key skip referential integrity trigger RI_FKey_fk_upd_check_required select for key share

Result:

Sources consulted:
- postgres/src/backend/utils/adt/ri_triggers.c at master · postgres/postgres · GitHub — https://github.com/postgres/postgres/blob/master/src/backend/utils/adt/ri_triggers.c
- PostgreSQL Source Code: src/backend/utils/adt/ri_triggers.c Source File — https://doxygen.postgresql.org/ri__triggers_8c_source.html
- PostgreSQL Source Code: src/backend/utils/adt/ri_triggers.c File Reference — https://doxygen.postgresql.org/ri__triggers_8c.html
- …and 27 more

Script executed:

nl -ba docker-compose.yml | sed -n '1,20p;44,70p'
rg -n -F -e 'postgres:' -e 'postgresql:' -- docker-compose.yml api/pyproject.toml api/Dockerfile

Repository: Flagsmith/flagsmith

Length of output: 2589


Script executed:

python3 - <<'PY'
import urllib.request
url='https://raw.githubusercontent.com/postgres/postgres/REL_15_5/src/backend/utils/adt/ri_triggers.c'
try:
    text=urllib.request.urlopen(url, timeout=15).read().decode()
except Exception as e:
    print('FETCH FAILED:',type(e).__name__,str(e))
else:
    lines=text.splitlines()
    for i,line in enumerate(lines):
        if line.startswith('RI_FKey_fk_upd_check_required('):
            print(url)
            for j in range(i-12, min(i+85,len(lines))):
                print(f'{j+1}: {lines[j]}')
            break
    else:
        print('Function not found')
PY

Repository: Flagsmith/flagsmith

Length of output: 3544


Lock the feature when resuming an experiment.

If an option update passes its check while an experiment is paused, a concurrent resume can commit before the update. The option change can then commit while the experiment is running. Acquire the feature lock for every transition to RUNNING.

Comment thread api/features/import_export/services.py Resolved
Comment thread api/features/serializers.py Resolved

@Zaimwa9 Zaimwa9 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First of all, thanks for picking this one, that is a very complex one given the model of the MV options and their weight allocation (mix between project and environment level). It also implies some design decisions that are still under discussions

There are some blockers right now, the main changes we need to move it forward for now would be:

  • Unchanged updates to a variant must not be rejected, otherwise saving the flag in another environment breaks
  • The lock should cover paused experiments as well as running ones
  • The import guard is too wide, and the new transaction changes the behaviour of every import
  • The error should come back in one shape and name the environment running the experiment

I described some tests that should help implementing the first point.

This behavior could also be rewarded with a one-line mention/warning in the doc 👍

Thanks again!

Comment on lines +97 to +111
def perform_update(self, serializer): # type: ignore[no-untyped-def]
features = [serializer.instance.feature]
if "feature" in serializer.validated_data:
features.append(serializer.validated_data["feature"])
feature_ids = sorted({feature.id for feature in features})
with transaction.atomic():
list(
Feature.objects.select_for_update()
.filter(pk__in=feature_ids)
.order_by("pk")
.values_list("pk", flat=True)
)
for feature in features:
self._check_no_live_experiment(feature)
super().perform_update(serializer)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the work on this, the direction is right: variants are shared across the project, so adding, deleting or changing a variant's value or key should be blocked while an experiment is running.

There is one case we need to keep working, though. Users must still be able to change the flag's value and variant percentages in the other environments, where no experiment is running.
How it behaves today:

  1. Start an experiment on a multivariate flag in environment A.
  2. In environment B, toggle the flag or change a variant weight and save.
  3. The save fails with "Multivariate options cannot be changed while the feature is in a live experiment."

The cause is that the dashboard sends a PUT mv-options/{id}/ for every existing option on each save, even when nothing changed. perform_update rejects every PUT, so the environment-level save that follows never runs.

Could you make perform_update reject only real changes?

  • Compare type, string_value, integer_value, boolean_value, key and feature against the stored option, and only raise if one of them differs.
  • Leave default_percentage_allocation out of the comparison. The dashboard always sends it as 0, and it only seeds new environments, so it does not affect a running experiment.

Create and delete stay blocked as they are.

Here are some test ideas that could help implementing the behavior imo.

Test What it proves
test_update_mv_option__running_experiment_unchanged_payload__returns_200 A PUT with the stored values, sent the way the dashboard sends it (default_percentage_allocation: 0), is not treated as an edit.
test_update_mv_option__running_experiment_default_allocation_only__returns_200 Changing only default_percentage_allocation is allowed, and the option's value and key are untouched.
test_update_mv_option__running_experiment_changed_field__returns_400 Parametrised over string_value and key: a real change to a variant is still rejected and nothing is saved. This replaces the current test_update_mv_option__live_experiment__returns_400.
test_update_mv_option__experiment_not_running__returns_200 Parametrised over created, and completed: only a running experiment locks the variants.
test_save_feature_value__experiment_running_in_other_environment__returns_200 Replays the dashboard's save in environment B while an experiment runs in environment A: a PUT mv-options/{id}/ for each option with its variant value and key unchanged, then the feature state update with a new control value and new variant percentages for environment B. Both succeed, environment B stores the new control value and percentages, and environment A's are unchanged.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is another related change to do here.

We need to lock it on a PAUSED experiment too, because once the experimentation resumes, if the variants have changed, all the results and values identities will receive will be skewed.

We will ask the users to complete experiments if they need to change the variants so in the code that means:

  • has_live_experiment should match both RUNNING and PAUSED and for the tests PAUSED behaves the same as running

Comment thread api/features/constants.py
# Multivariate variant reported when an identity falls through to the control value
CONTROL_VARIANT_KEY = "control"
RESERVED_VARIANT_KEY_MESSAGE = f'"{CONTROL_VARIANT_KEY}" is a reserved variant key.'
LIVE_EXPERIMENT_MV_OPTIONS_MESSAGE = (

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DRF will return this error in three different shapes: a bare list from the view, and both a list and a string under multivariate_options from the serializer. None say which environment is running the experiment.

Could we use one shape, {"detail": "<message>"} like the experiment endpoints, and name the environment in the message?

Comment thread api/features/import_export/tasks.py Outdated

feature_import.status = SUCCESS
feature_import.save()
with transaction.atomic():

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This transaction changes the behaviour of every import, not only the ones touching an experiment: one rejected feature now rolls back the whole batch.

Could we revert this function to how it was, and instead run the experiment check as a first pass over all the features in the import, before anything is written? A rejected import then changes nothing, and imports unrelated to experiments behave exactly as before.

Comment thread api/features/import_export/services.py Outdated
affecting other environments' feature states or the Feature definition.
"""
if has_live_experiment(existing_feature.id):
raise ValidationError(LIVE_EXPERIMENT_MV_OPTIONS_MESSAGE)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is too wide, in the sense that we should only reject the import that would add a variant, or when the target environment is the one with the experiment. In that second case the import deletes the environment's segment overrides, which includes the experiment's rollout.

Importing existing variants into another environment only changes that environment's value and percentages, so it should still work.

Comment thread api/features/serializers.py Outdated
Comment on lines +346 to +351
if (
instance
and self._changes_multivariate_options(multivariate_options, instance)
and has_live_experiment(instance.id)
):
raise serializers.ValidationError(LIVE_EXPERIMENT_MV_OPTIONS_MESSAGE)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the same check as the one in update, but it runs before the lock is taken and returns the error in a different shape. Can we keep only the one in update?

@Zaimwa9
Zaimwa9 removed the request for review from emyller November 11, 2026 11:33
- mv-options PUT only checks for an experiment when the variation itself
  changes (type, value, key or feature). default_percentage_allocation is
  ignored, so the dashboard can still save other environments.
- Paused experiments lock the variations as well as running ones.
- Imports keep their original per-feature behaviour. A first pass rejects
  the import before anything is written, and only when it would add a
  variation or overwrite the environment running the experiment.
- One error shape, {"detail": ...}, naming the experiment environment.
- Drop the duplicate check in validate_multivariate_options.
- Mention the lock in the experiment docs.
@TejasLamba2006
TejasLamba2006 requested review from a team as code owners November 12, 2026 06:36
@TejasLamba2006
TejasLamba2006 requested review from Holmus and removed request for a team November 12, 2026 06:36
@github-actions github-actions Bot added api Issue related to the REST API docs Documentation updates labels Nov 12, 2026
@TejasLamba2006

Copy link
Copy Markdown
Author

@Zaimwa9 thanks for the detailed review, the test table made the first point easy to pin down. Pushed a91b390:

  • perform_update only checks for an experiment when the type, value, key or feature of the variation actually changes. default_percentage_allocation is left out, and create and delete stay blocked. The tests from your table are in, including the replay of the dashboard save in environment B while A runs the experiment.
  • Paused experiments lock the variations too. The live_experiment fixture runs every test for both running and paused.
  • _import_features_for_environment is back to how it was. A first pass over the import rejects it before anything is written, and only when it would add a variation or when the target is the experiment's environment. Importing existing variations into another environment goes through.
  • The error is {"detail": "..."} everywhere and names the environment.
  • The check in validate_multivariate_options is gone, only update checks now.
  • Added a sentence about the lock to the Create an Experiment page.

Could you take another look when you get a chance?

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

Review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: af3e240a-f26f-4022-a0c8-b9a0a4bae9db
Commits

Reviewing files that changed from the base of the PR and between d564c8d and a91b390.

Files selected for processing: 9

  • api/experimentation/feature_state_metadata.py
  • api/features/constants.py
  • api/features/import_export/tasks.py
  • api/features/multivariate/views.py
  • api/features/serializers.py
  • api/tests/unit/features/import_export/test_unit_features_import_export_tasks.py
  • api/tests/unit/features/multivariate/test_unit_multivariate_live_experiment.py
  • docs/docs/deployment-self-hosting/observability/_events-catalogue.md
  • docs/docs/experimentation/create-an-experiment.md

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.

Comment thread api/features/import_export/tasks.py Outdated
Comment on lines +155 to +156
if feature_import.strategy != SKIP:
_check_import_against_live_experiments(input_data, environment)

@coderabbitai coderabbitai Bot Nov 12, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Major · Data Integrity & Integration · Heavy lift

Hold the feature lock through the import write.

If this check sees no live experiment, an experiment can start before overwrite_feature_for_environment creates an imported variation. The experiment start takes a feature-row lock, but this import does not take that lock or retain a transaction across its check and write. The import can therefore add a variation after the experiment starts. Lock each affected feature, recheck its experiment status, and keep the lock until its import writes finish. (raw.githubusercontent.com)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overkill, no need to address this one. The window is very small, and closing it would bring back a transaction around the whole import

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Zaimwa9 Understood. The race remains between the experiment check and the import write, but I accept your decision not to add a transaction for it.

I’m not marking the finding as addressed.

You are interacting with an AI system.

Comment thread api/features/import_export/tasks.py Resolved
Comment on lines +112 to +116
if not any(
field in data and data[field] != getattr(instance, field)
for field in VARIATION_FIELDS
):
super().perform_update(serializer)

@coderabbitai coderabbitai Bot Nov 12, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Major · Data Integrity & Integration · Heavy lift

Lock and reload the option before taking the unchanged-update path.

Suppose an unchanged PUT loads value A. Another request changes the option to B, and an experiment then starts. The first PUT still compares A with its cached instance, takes this branch, and saves A without checking the now-running experiment. The dashboard’s unchanged-option saves can therefore change a live variation. Lock the feature and reload the option before deciding whether the submitted fields changed. DRF’s model update saves the supplied instance after setting its validated fields. (raw.githubusercontent.com)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wrong qualification of major, kind of rare scenario

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Zaimwa9 I agree. The race requires a narrow overlap, so Minor is a better qualification than Major.

The race could still overwrite a variation after an experiment starts. I’ll leave the decision on whether to address it to you. I’m not marking the finding as addressed.

You are interacting with an AI system.

super().perform_update(serializer)
return

features = {instance.feature, data.get("feature", instance.feature)}

@coderabbitai coderabbitai Bot Nov 12, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Major · Security & Privacy · Quick win

Authorise the replacement feature before moving an option.

If a requester can edit an option in project A, the requester can supply a feature ID from project B without permission to edit project B. The new branch checks that target for experiments, but neither it nor the source-object permission check authorises the target. The update can attach the option to project B. Require permission on the target project, or reject cross-project moves. The serializer’s relation uses the related model’s default manager. (raw.githubusercontent.com)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of scope, it's already existing and should be addressed as a follow up

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Zaimwa9 Understood. We can address the authorisation concern in a follow-up. Would you like me to open a GitHub issue for it?

I’m not marking the finding as addressed.

You are interacting with an AI system.

@Zaimwa9 Zaimwa9 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fast changes. However we had to re-think the behavior around destructive imports in an environment with features running an experiment, which would be great to adapt.

👍

status__in=(ExperimentStatus.RUNNING, ExperimentStatus.PAUSED),
)
.select_related("environment")
.first()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Problem here, it only looks at one experiment per feature and its environment is the one compared to the one in the import.

Meaning if my_feature has an experiment running in envA and envB, an import in environment B would still succeed and potentially delete the segment overrides.

But actually, after thinking about the underlying customer experience (having a feature in experiment would block any overriding import until the experiment is completed), we are leaning towards another option:
We should skip a feature, and keep going with the rest of the import, in two cases:

  • it is under an experiment in the environment we import into;
  • the import would add a variation to it, and it is under an experiment in any environment.

On my side I will add a frontend warning that features under experiment are being skipped.

I think the code should look something like this (the loop already exists):

def get_live_experiment(
    feature_id: int, environment_id: int | None = None
) -> Experiment | None:
    experiments = Experiment.objects.filter(
        feature_id=feature_id,
        status__in=(ExperimentStatus.RUNNING, ExperimentStatus.PAUSED),
    )
    if environment_id is not None:
        experiments = experiments.filter(environment_id=environment_id)
    return experiments.select_related("environment").first()

and in the import loop:

if existing_feature and (
    get_live_experiment(existing_feature.id, environment.id)
    or (adds_variation and get_live_experiment(existing_feature.id))
):
    continue

If you can please update and add the related test cases, that'd be great 👍

super().perform_update(serializer)
return

features = {instance.feature, data.get("feature", instance.feature)}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of scope, it's already existing and should be addressed as a follow up

Comment thread api/features/import_export/tasks.py Outdated
Comment on lines +155 to +156
if feature_import.strategy != SKIP:
_check_import_against_live_experiments(input_data, environment)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overkill, no need to address this one. The window is very small, and closing it would bring back a transaction around the whole import

Comment on lines +112 to +116
if not any(
field in data and data[field] != getattr(instance, field)
for field in VARIATION_FIELDS
):
super().perform_update(serializer)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wrong qualification of major, kind of rare scenario

…e import

Skip individual features in _import_features_for_environment instead of
failing the whole import when a feature has a running or paused experiment
in the target environment or when the import would add a variation while an
experiment is running or paused in any environment.
@TejasLamba2006

Copy link
Copy Markdown
Author

@Zaimwa9 updated in 8682e20: get_live_experiment now takes an optional environment_id, and destructive imports skip features that have a running or paused experiment in the target environment (including when another environment also has an experiment) or would add a variation while an experiment is active anywhere, letting the rest of the batch import.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

5 issues found across 10 files

Confidence score: 2/5

  • tasks.py can create a variation after a concurrent experiment start because the feature is not locked across the live-experiment check and import write. Hold the feature-row lock across both.
  • multivariate/views.py can miss locking feature B if an option is reassigned from A to B before the save; perform_destroy also locks a stale association. Resolve and lock the current association before mutating the option.
  • The unchanged-value fast path in multivariate/views.py can use a stale option and overwrite a variant after a concurrent edit and experiment start. Lock and reload the option before comparing and saving.
  • tasks.py can overwrite earlier rows, skip a later conflicting feature, and still report the import as SUCCESS. Preflight the full payload and fail before writing if it conflicts.
Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="api/tests/unit/features/multivariate/test_unit_multivariate_live_experiment.py">

<violation number="1" location="api/tests/unit/features/multivariate/test_unit_multivariate_live_experiment.py:139">
P2: This test covers only `string_value` and `key`; a regression that stops locking `type`, `integer_value`, or `boolean_value` would go unnoticed. Add cases that mutate those fields on matching option types.

(Based on your team's feedback about backend diff coverage.) .</violation>
</file>

<file name="api/features/multivariate/views.py">

<violation number="1" location="api/features/multivariate/views.py:116">
P1: This fast path saves without locking or reloading the option, so its “unchanged” comparison can become stale. A concurrent variant edit followed by experiment start lets this PUT overwrite the variant during the live run; recheck against locked current state before bypassing the experiment guard.

(Based on your team's feedback about locking experiment variants.)</violation>

<violation number="2" location="api/features/multivariate/views.py:119">
P1: This feature set can use the option’s stale source association: after a concurrent reassignment from A to B, the lock set can omit B while the save mutates the option now attached to B. `perform_destroy` also locks stale `instance.feature_id`, allowing a live variant to be deleted; lock and reload the option before checking its current feature.

(Based on your team's feedback about locking experiment variants.)</violation>
</file>

<file name="api/features/import_export/tasks.py">

<violation number="1" location="api/features/import_export/tasks.py:160">
P1: This per-row check can skip a conflicting feature after earlier rows have already been overwritten, then mark the import `SUCCESS`, leaving a partial import that looks complete. Preflight the full payload and fail before applying any rows when it contains a locked variant mutation.

(Based on your team's feedback about preflighting imports with experiment conflicts.)</violation>

<violation number="2" location="api/features/import_export/tasks.py:193">
P1: The import checks for a live experiment without locking the feature, then can create a variation after a concurrent start commits. Hold the feature-row lock across the experiment check and the import write so the start transition and import serialize.</violation>
</file>

Tip: cubic used a learning from your PR history. Let your coding agent read cubic learnings directly with the cubic MCP.

View guided diff | Re-trigger cubic

super().perform_update(serializer)
return

features = {instance.feature, data.get("feature", instance.feature)}

@cubic-dev-ai cubic-dev-ai Bot Nov 14, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: This feature set can use the option’s stale source association: after a concurrent reassignment from A to B, the lock set can omit B while the save mutates the option now attached to B. perform_destroy also locks stale instance.feature_id, allowing a live variant to be deleted; lock and reload the option before checking its current feature.

(Based on your team's feedback about locking experiment variants.)

View Feedback

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At api/features/multivariate/views.py, line 119:

<comment>This feature set can use the option’s stale source association: after a concurrent reassignment from A to B, the lock set can omit B while the save mutates the option now attached to B. `perform_destroy` also locks stale `instance.feature_id`, allowing a live variant to be deleted; lock and reload the option before checking its current feature.

(Based on your team's feedback about locking experiment variants.) </comment>

<file context>
@@ -69,9 +83,57 @@ def get_permissions(self):  # type: ignore[no-untyped-def]
+            super().perform_update(serializer)
+            return
+
+        features = {instance.feature, data.get("feature", instance.feature)}
+        with transaction.atomic():
+            list(
</file context>
Fix with cubic

field in data and data[field] != getattr(instance, field)
for field in VARIATION_FIELDS
):
super().perform_update(serializer)

@cubic-dev-ai cubic-dev-ai Bot Nov 14, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: This fast path saves without locking or reloading the option, so its “unchanged” comparison can become stale. A concurrent variant edit followed by experiment start lets this PUT overwrite the variant during the live run; recheck against locked current state before bypassing the experiment guard.

(Based on your team's feedback about locking experiment variants.)

View Feedback

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At api/features/multivariate/views.py, line 116:

<comment>This fast path saves without locking or reloading the option, so its “unchanged” comparison can become stale. A concurrent variant edit followed by experiment start lets this PUT overwrite the variant during the live run; recheck against locked current state before bypassing the experiment guard.

(Based on your team's feedback about locking experiment variants.) </comment>

<file context>
@@ -69,9 +83,57 @@ def get_permissions(self):  # type: ignore[no-untyped-def]
+            field in data and data[field] != getattr(instance, field)
+            for field in VARIATION_FIELDS
+        ):
+            super().perform_update(serializer)
+            return
+
</file context>
Fix with cubic

if existing_feature and feature_import.strategy == SKIP:
if existing_feature and (
feature_import.strategy == SKIP
or _should_skip_for_live_experiment(

@cubic-dev-ai cubic-dev-ai Bot Nov 14, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: This per-row check can skip a conflicting feature after earlier rows have already been overwritten, then mark the import SUCCESS, leaving a partial import that looks complete. Preflight the full payload and fail before applying any rows when it contains a locked variant mutation.

(Based on your team's feedback about preflighting imports with experiment conflicts.)

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At api/features/import_export/tasks.py, line 160:

<comment>This per-row check can skip a conflicting feature after earlier rows have already been overwritten, then mark the import `SUCCESS`, leaving a partial import that looks complete. Preflight the full payload and fail before applying any rows when it contains a locked variant mutation.

(Based on your team's feedback about preflighting imports with experiment conflicts.) </comment>

<file context>
@@ -153,7 +155,12 @@ def _import_features_for_environment(feature_import: FeatureImport) -> None:
-        if existing_feature and feature_import.strategy == SKIP:
+        if existing_feature and (
+            feature_import.strategy == SKIP
+            or _should_skip_for_live_experiment(
+                feature_data, existing_feature, environment
+            )
</file context>
Fix with cubic

(mv_data["type"], mv_data["value"]) not in existing_variations
for mv_data in feature_data["multivariate"]
)
return bool(adds_variation and get_live_experiment(existing_feature.id))

@cubic-dev-ai cubic-dev-ai Bot Nov 14, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: The import checks for a live experiment without locking the feature, then can create a variation after a concurrent start commits. Hold the feature-row lock across the experiment check and the import write so the start transition and import serialize.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At api/features/import_export/tasks.py, line 193:

<comment>The import checks for a live experiment without locking the feature, then can create a variation after a concurrent start commits. Hold the feature-row lock across the experiment check and the import write so the start transition and import serialize.</comment>

<file context>
@@ -166,6 +173,26 @@ def _import_features_for_environment(feature_import: FeatureImport) -> None:
+        (mv_data["type"], mv_data["value"]) not in existing_variations
+        for mv_data in feature_data["multivariate"]
+    )
+    return bool(adds_variation and get_live_experiment(existing_feature.id))
+
+
</file context>
Fix with cubic

Comment thread api/features/serializers.py Resolved

@pytest.mark.parametrize(
"field, new_value",
[("string_value", "edited"), ("key", "edited-key")],

@cubic-dev-ai cubic-dev-ai Bot Nov 14, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This test covers only string_value and key; a regression that stops locking type, integer_value, or boolean_value would go unnoticed. Add cases that mutate those fields on matching option types.

(Based on your team's feedback about backend diff coverage.) .

View Feedback

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At api/tests/unit/features/multivariate/test_unit_multivariate_live_experiment.py, line 139:

<comment>This test covers only `string_value` and `key`; a regression that stops locking `type`, `integer_value`, or `boolean_value` would go unnoticed. Add cases that mutate those fields on matching option types.

(Based on your team's feedback about backend diff coverage.) .</comment>

<file context>
@@ -0,0 +1,461 @@
+
+@pytest.mark.parametrize(
+    "field, new_value",
+    [("string_value", "edited"), ("key", "edited-key")],
+)
+def test_update_mv_option__running_experiment_changed_field__returns_400(
</file context>
Fix with cubic

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 2 files (changes from recent commits).

Confidence score: 3/5

  • In api/features/serializers.py, a PATCH that only changes default_percentage_allocation can remove all multivariate_options, silently losing existing feature data. Preserve those options on partial updates.
Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="api/features/serializers.py">

<violation number="1" location="api/features/serializers.py:350">
P2: A feature PATCH that changes only `default_percentage_allocation` now passes this comparison, but `update()` then removes all `multivariate_options` before saving, so the successful response silently leaves the allocation unchanged. Persist the permitted allocation edits while skipping variant changes; otherwise new environments inherit the old allocation.</violation>
</file>

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

instance: Feature,
) -> bool:
# The frontend resubmits the whole feature, so only a real change to
# variation value/type counts. `default_percentage_allocation` only

@cubic-dev-ai cubic-dev-ai Bot Nov 17, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: A feature PATCH that changes only default_percentage_allocation now passes this comparison, but update() then removes all multivariate_options before saving, so the successful response silently leaves the allocation unchanged. Persist the permitted allocation edits while skipping variant changes; otherwise new environments inherit the old allocation.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At api/features/serializers.py, line 350:

<comment>A feature PATCH that changes only `default_percentage_allocation` now passes this comparison, but `update()` then removes all `multivariate_options` before saving, so the successful response silently leaves the allocation unchanged. Persist the permitted allocation edits while skipping variant changes; otherwise new environments inherit the old allocation.</comment>

<file context>
@@ -347,13 +347,13 @@ def _changes_multivariate_options(
     ) -> bool:
         # The frontend resubmits the whole feature, so only a real change to
-        # writable option fields counts. Compare on content, not ids.
+        # variation value/type counts. `default_percentage_allocation` only
+        # seeds new options or environments and does not affect live variants.
         fields = (
</file context>
Fix with cubic

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api Issue related to the REST API docs Documentation updates

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants