Skip to content

Add First Minute incident drill and Flagsmith experiment - #9016

Open
arpitkjaiswal wants to merge 3 commits into
Flagsmith:mainfrom
arpitkjaiswal:main
Open

arpitkjaiswal wants to merge 3 commits into
Flagsmith:mainfrom
arpitkjaiswal:main

Conversation

@arpitkjaiswal

@arpitkjaiswal arpitkjaiswal commented Nov 19, 2026 •

Copy link
Copy Markdown

Thanks for submitting a PR! Please check the boxes below:

  • I have read the Contributing Guide.
  • I have added information to docs/ if required so people know about the feature.
  • I have filled in the "Changes" section below.
  • I have filled in the "How did you test this code" section below.

Changes

Contributes to

Please describe.

How did you test this code?

Please describe.

@arpitkjaiswal
arpitkjaiswal requested a review from a team as a code owner November 19, 2026 12:36
@arpitkjaiswal
arpitkjaiswal requested review from khvn26 and removed request for a team November 19, 2026 12:36
@vercel

vercel Bot commented Nov 19, 2026

Copy link
Copy Markdown

@arpitkrjaiswal-eng is attempting to deploy a commit to the Flagsmith Team on Vercel.

A member of the Team first needs to authorize it.

@coderabbitai

coderabbitai Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Explain Change →

Walkthrough

This change adds a timed checkout incident drill with evidence panels, diagnosis submission, a connection-pool calculator, and result views. It adds Flagsmith experiment setup, browser-side assignment and event tracking, and a rehearsal that samples traffic, checks warehouse results, and applies a verified rollout. The project also includes a Vite proxy, a standalone practice HTML build, automated tests, verification data, and setup and use instructions.

Estimated code review effort: 4 (Complex) | ~45 minutes

Merge Risk: Moderate · up to 3d727

The drill and rehearsal are mostly in place. Before merge, fix the first-attempt measurement gaps: offline practice, lost conversions, unconfirmed delivery and failed saves. Also make experiment setup reject a reused experiment with the wrong allocation, and host the downloadable build somewhere the project controls.

Finishing Touches

Fix failing CI checks

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

3 issues found across 16 files

Confidence score: 3/5

  • In setup_experiment.py, the SRM check gets no expected shares after the guided allocation is reset, so it may not run for the 50/50 experiment. Read the active rollout allocation.
  • In setup_experiment.py, a paused experiment exits as “not running” instead of resuming. Call start/ for paused experiments.
  • In README.md, both download links point to arpitkjaiswal/flagsmith1 rather than this repository. Link to Flagsmith/flagsmith so the downloads stay available here.
Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="examples/first-minute/setup_experiment.py">

<violation number="1" location="examples/first-minute/setup_experiment.py:78">
P2: This configures a 50/50 experiment, but the backend reads expected shares from the environment defaults after resetting the guided allocation to 0%, so its SRM check returns `None`. Read the active rollout allocation when calculating expected shares; otherwise the advertised sample-ratio warning cannot catch assignment drift for this experiment.</violation>

<violation number="2" location="examples/first-minute/setup_experiment.py:81">
P2: Could this also call `start/` for a paused experiment? The API allows `paused` → `running`, but this branch exits with “The experiment is not running” instead of resuming it.</violation>
</file>

<file name="README.md">

<violation number="1" location="README.md:5">
P2: Both advertised downloads point to `arpitkjaiswal/flagsmith1`, although the ZIP is committed in this repository. Link to `Flagsmith/flagsmith` so the public download cannot drift from or disappear independently of the shipped artifact.</violation>
</file>

Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Re-trigger cubic

Comment thread examples/first-minute/README.md Resolved
Comment thread examples/first-minute/index.html Resolved
Comment thread examples/first-minute/index.html Resolved
"multivariate_feature_state_values": [{"multivariate_feature_option": variant["id"], "percentage_allocation": 50}]},
}, lambda e: e["name"] == name)
detail = path + str(experiment["id"]) + "/"
if experiment["status"] == "created":

@cubic-dev-ai cubic-dev-ai Bot Nov 19, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Could this also call start/ for a paused experiment? The API allows paused → running, but this branch exits with “The experiment is not running” instead of resuming it.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At examples/first-minute/setup_experiment.py, line 81:

<comment>Could this also call `start/` for a paused experiment? The API allows `paused` → `running`, but this branch exits with “The experiment is not running” instead of resuming it.</comment>

<file context>
@@ -0,0 +1,93 @@
+            "multivariate_feature_state_values": [{"multivariate_feature_option": variant["id"], "percentage_allocation": 50}]},
+    }, lambda e: e["name"] == name)
+    detail = path + str(experiment["id"]) + "/"
+    if experiment["status"] == "created":
+        request(detail + "start/", {})
+    if request(detail)["status"] != "running":
</file context>
Fix with cubic

Comment thread README.md Outdated

Play the on-call engineer for 60 seconds. Inspect a checkout outage, submit a diagnosis, then adjust the connection pool to see how it affects the queue. This hackathon entry uses Flagsmith to test whether a three-step runbook improves first-attempt diagnoses.

[Download the offline drill](https://raw.githubusercontent.com/arpitkjaiswal/flagsmith1/main/examples/first-minute/media/first-minute.zip) · [Watch the walkthrough](examples/first-minute/media/walkthrough.mp4) · [Run the live experiment](examples/first-minute/README.md)

@cubic-dev-ai cubic-dev-ai Bot Nov 19, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Both advertised downloads point to arpitkjaiswal/flagsmith1, although the ZIP is committed in this repository. Link to Flagsmith/flagsmith so the public download cannot drift from or disappear independently of the shipped artifact.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At README.md, line 5:

<comment>Both advertised downloads point to `arpitkjaiswal/flagsmith1`, although the ZIP is committed in this repository. Link to `Flagsmith/flagsmith` so the public download cannot drift from or disappear independently of the shipped artifact.</comment>

<file context>
@@ -1,3 +1,11 @@
+
+Play the on-call engineer for 60 seconds. Inspect a checkout outage, submit a diagnosis, then adjust the connection pool to see how it affects the queue. This hackathon entry uses Flagsmith to test whether a three-step runbook improves first-attempt diagnoses.
+
+[Download the offline drill](https://raw.githubusercontent.com/arpitkjaiswal/flagsmith1/main/examples/first-minute/media/first-minute.zip) · [Watch the walkthrough](examples/first-minute/media/walkthrough.mp4) · [Run the live experiment](examples/first-minute/README.md)
+
+Unzip the download and open `first-minute.html` in a browser. Practice rounds run entirely on your device.
</file context>
Fix with cubic

"metrics": [{"metric": metric["id"], "expected_direction": "increase"}],
"experiment_rollout": {"enabled": True, "rollout_percentage": 100,
"feature_state_value": {"type": "string", "value": "control"},
"multivariate_feature_state_values": [{"multivariate_feature_option": variant["id"], "percentage_allocation": 50}]},

@cubic-dev-ai cubic-dev-ai Bot Nov 19, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This configures a 50/50 experiment, but the backend reads expected shares from the environment defaults after resetting the guided allocation to 0%, so its SRM check returns None. Read the active rollout allocation when calculating expected shares; otherwise the advertised sample-ratio warning cannot catch assignment drift for this experiment.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At examples/first-minute/setup_experiment.py, line 78:

<comment>This configures a 50/50 experiment, but the backend reads expected shares from the environment defaults after resetting the guided allocation to 0%, so its SRM check returns `None`. Read the active rollout allocation when calculating expected shares; otherwise the advertised sample-ratio warning cannot catch assignment drift for this experiment.</comment>

<file context>
@@ -0,0 +1,93 @@
+        "metrics": [{"metric": metric["id"], "expected_direction": "increase"}],
+        "experiment_rollout": {"enabled": True, "rollout_percentage": 100,
+            "feature_state_value": {"type": "string", "value": "control"},
+            "multivariate_feature_state_values": [{"multivariate_feature_option": variant["id"], "percentage_allocation": 50}]},
+    }, lambda e: e["name"] == name)
+    detail = path + str(experiment["id"]) + "/"
</file context>
Fix with cubic

Comment thread examples/first-minute/main.js Resolved

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

Review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: a5222e38-40b1-4246-bd21-bdb01c4f8d3b
Commits

Reviewing files that changed from the base of the PR and between fe9b4cd and ffc0d02.

Files ignored due to path filters: 4

  • examples/first-minute/media/first-minute.png is excluded by !**/*.png
  • examples/first-minute/media/first-minute.zip is excluded by !**/*.zip
  • examples/first-minute/media/walkthrough.mp4 is excluded by !**/*.mp4
  • examples/first-minute/package-lock.json is excluded by !**/package-lock.json
Files selected for processing: 12

  • README.md
  • examples/first-minute/.env.example
  • examples/first-minute/.gitignore
  • examples/first-minute/README.md
  • examples/first-minute/build-practice.js
  • examples/first-minute/index.html
  • examples/first-minute/main.js
  • examples/first-minute/package.json
  • examples/first-minute/setup_experiment.py
  • examples/first-minute/style.css
  • examples/first-minute/verification.json
  • examples/first-minute/vite.config.js

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.

Comment thread examples/first-minute/main.js Outdated
lastLayout = flag?.enabled && flag.value === 'guided' ? 'guided' : 'control';
}
active = { layout: lastLayout, measured, deadline: Date.now() + 60000 };
if (measured) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Major · Data Integrity & Integration · Quick win

Record offline practice so it cannot later count as a first attempt.

When initialisation fails or the experiment is not configured, begin() starts the attempt with measured set to false. The localStorage write at line 165 runs only when measured is true, so a practice attempt stores no participation record. The participant identity persists across reloads. After connectivity returns, a reload can enrol the same participant in the experiment. A correct answer then records drill_resolved for someone who already practised.

Store a participation record when a practice attempt starts while the experiment is unavailable. Mark it so that initialise() does not resume it as an active attempt. With that record present, the enrolment check in begin() must not start a measured attempt for that participant. Keep connected first attempts unchanged. This preserves the first-time-participant measurement described in the First Minute README.

Comment thread examples/first-minute/main.js Resolved
Comment thread examples/first-minute/main.js Outdated
$('start').onclick = () => begin(true);
$('mode').textContent = 'Debrief ready';
$('result').focus();
if (attempt.measured && correct && connected) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Major · Data Integrity & Integration · Heavy lift

Retain correct results when a restored attempt is offline.

If the flags request fails after a reload, initialise() still restores the measured attempt. A correct answer before expiry sets done: true, but this condition skips the conversion because connected is false. Later reloads offer only practice, so the conversion is permanently lost.

Persist a pending conversion with the original participant and submission time. Deliver it after recovery without counting practice attempts.

Comment thread examples/first-minute/main.js Outdated
Comment on lines +222 to +224
await flagsmith.flushEvents();
} catch {
$('outcome').append(' · Event delivery unconfirmed');

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Major · Data Integrity & Integration · Heavy lift

Do not treat flush resolution as delivery success.

flushEvents() resolves after the SDK finishes its delivery attempt, including the single retry. A failed batch is dropped and logged only through the SDK logger. The catch at main.js:222-224 therefore never runs for a delivery failure, so Event delivery unconfirmed is never shown.

Events still in the SDK buffer are lost if the participant reloads before the flush completes. The attempt is already saved as done: true before the flush, so a reload does not retry the conversion.

Show the unconfirmed state only when the SDK reports a failed delivery. flushEvents() gives no failure signal, so the change needs a way to detect that failure before the warning can appear. If the conversion must survive a reload, keep the stored result pending until delivery succeeds.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 existing issue remains and 2 new issues found across 21 files (changes from recent commits).

Confidence score: 3/5

  • In main.js, an offline resume can save a result as done without emitting drill_resolved, and the next load only displays the result, so the conversion is permanently lost. Persist and retry the event.
  • In rehearse.py, the fallback assumes a 50/50 split even when it reuses an experiment with different rollout weights, so the allocation check can give the wrong result. Verify the existing weights before checking allocation.
  • In README.md, the smoke tests mock Flagsmith responses and capture events rather than using a local instance, which could lead readers to overestimate what the tests verify. Clarify that only the rehearsal uses the instance.
Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="examples/first-minute/README.md">

<violation number="1" location="examples/first-minute/README.md:30">
P2: The browser smoke tests do not use a local Flagsmith instance: they mock identity responses and capture events with Playwright route handlers. Clarify that only the rehearsal uses the instance, so the evidence does not overstate what the smoke tests verified.</violation>
</file>

<file name="examples/first-minute/main.js">

<violation number="1" location="examples/first-minute/main.js:230">
P2: An offline resume persists a measured result as `done` but skips `drill_resolved` here. On the next successful load, `initialise()` only displays that result, so the conversion is permanently lost; persist and retry a pending conversion after initialization.</violation>
</file>

Requires human review: Auto-approval blocked because this review re-detected 1 unresolved issue already reported by Cubic.

Re-trigger cubic


**Measurement.** The browser stores an anonymous identity and the first attempt's deadline. Opening the desk records an SDK exposure. Only a correct first answer before expiry records the `drill_resolved` occurrence event. Reloading preserves the deadline and restores a finished debrief. Tabs sharing the same browser identity share one first answer. Replays are practice. A new browser profile is a new identity.

**Evidence.** The [browser smoke-test snapshot](verification.json) recorded three exposures and two conversions. The in-app **Experiment replay** follows a larger synthetic run through assignment, measurement and rollout. Both runs used the real Flagsmith SDK and a local Flagsmith instance.

@cubic-dev-ai cubic-dev-ai Bot Nov 19, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: The browser smoke tests do not use a local Flagsmith instance: they mock identity responses and capture events with Playwright route handlers. Clarify that only the rehearsal uses the instance, so the evidence does not overstate what the smoke tests verified.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At examples/first-minute/README.md, line 30:

<comment>The browser smoke tests do not use a local Flagsmith instance: they mock identity responses and capture events with Playwright route handlers. Clarify that only the rehearsal uses the instance, so the evidence does not overstate what the smoke tests verified.</comment>

<file context>
@@ -15,20 +15,44 @@ To run from source, run `npm ci` and `npm run dev` with Node 24. Open the addres
+**Measurement.** The browser stores an anonymous identity and the first attempt's deadline. Opening the desk records an SDK exposure. Only a correct first answer before expiry records the `drill_resolved` occurrence event. Reloading preserves the deadline and restores a finished debrief. Tabs sharing the same browser identity share one first answer. Replays are practice. A new browser profile is a new identity.
 
-**Evidence.** The notes panel contains a dated snapshot of a separate browser verification run. Flagsmith reported two guided exposures, one control exposure and two conversions. The automated checks deliberately exercised correct and incorrect answers. These counts establish that assignment, event delivery and aggregation work. They cannot estimate the effect on people.
+**Evidence.** The [browser smoke-test snapshot](verification.json) recorded three exposures and two conversions. The in-app **Experiment replay** follows a larger synthetic run through assignment, measurement and rollout. Both runs used the real Flagsmith SDK and a local Flagsmith instance.
 
 **Rollout.** Collect independent participant attempts and inspect sample-ratio warnings. Flagsmith shows “Collecting data” below its inference thresholds. Once sufficient evidence supports a decision, end the participant experiment and use the same flag to roll out the selected desk. If the evidence stays inconclusive, continue collecting data.
</file context>
Suggested change
**Evidence.** The [browser smoke-test snapshot](verification.json) recorded three exposures and two conversions. The in-app **Experiment replay** follows a larger synthetic run through assignment, measurement and rollout. Both runs used the real Flagsmith SDK and a local Flagsmith instance.
**Evidence.** The [browser smoke-test snapshot](verification.json) recorded three exposures and two conversions. The in-app **Experiment replay** follows a larger synthetic run through assignment, measurement and rollout. The replay used the real Flagsmith SDK and a local Flagsmith instance; the browser checks intercept SDK traffic locally.
Fix with cubic

}
displayResult(first.finishedAt ? first : outcome);
if (first === outcome && outcome.measured && outcome.answer === 'pool') {
if (!connected) {

@cubic-dev-ai cubic-dev-ai Bot Nov 19, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: An offline resume persists a measured result as done but skips drill_resolved here. On the next successful load, initialise() only displays that result, so the conversion is permanently lost; persist and retry a pending conversion after initialization.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At examples/first-minute/main.js, line 230:

<comment>An offline resume persists a measured result as `done` but skips `drill_resolved` here. On the next successful load, `initialise()` only displays that result, so the conversion is permanently lost; persist and retry a pending conversion after initialization.</comment>

<file context>
@@ -216,7 +205,32 @@ async function finish(answer) {
+    }
+    displayResult(first.finishedAt ? first : outcome);
+    if (first === outcome && outcome.measured && outcome.answer === 'pool') {
+        if (!connected) {
+            $('outcome').append(' · Event delivery unconfirmed');
+            return;
</file context>
Fix with cubic

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 5

Review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: 97bda201-0672-4f77-8dce-93a7a03ae8f6
Commits

Reviewing files that changed from the base of the PR and between ffc0d02 and 80aab9d.

Files ignored due to path filters: 5

  • examples/first-minute/media/first-minute.png is excluded by !**/*.png
  • examples/first-minute/media/first-minute.zip is excluded by !**/*.zip
  • examples/first-minute/media/flagsmith-results.png is excluded by !**/*.png
  • examples/first-minute/media/walkthrough.mp4 is excluded by !**/*.mp4
  • examples/first-minute/package-lock.json is excluded by !**/package-lock.json
Files selected for processing: 16

  • README.md
  • examples/first-minute/.env.example
  • examples/first-minute/.gitignore
  • examples/first-minute/README.md
  • examples/first-minute/attempt-store.js
  • examples/first-minute/experiment-replay.js
  • examples/first-minute/index.html
  • examples/first-minute/main.js
  • examples/first-minute/package.json
  • examples/first-minute/playwright.config.js
  • examples/first-minute/rehearsal-traffic.js
  • examples/first-minute/rehearsal.json
  • examples/first-minute/rehearse.py
  • examples/first-minute/setup_experiment.py
  • examples/first-minute/style.css
  • examples/first-minute/tests/first-attempt.spec.js

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 6 remain after this review.

if (active || starting) return;
starting = true;
try {
const candidate = { layout: lastLayout, measured: false, deadline: Date.now() + 60000 };

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor · Functional Correctness · Quick win

Start the deadline after storage admits the new attempt.

If updateRecord takes five seconds, the desk opens with about 55 seconds remaining. The deadline already includes the storage wait. This contradicts the briefing, which says the clock begins when the participant enters the desk.

Create the new deadline when the start claim can proceed. Keep the stored deadline when another tab joins an existing attempt.

Comment on lines +221 to +223
displayResult(outcome);
$('outcome').append(' · First attempt could not be saved');
return;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Major · Data Integrity & Integration · Heavy lift

Do not restore a revealed diagnosis as an unfinished measured attempt.

If saving an incorrect answer fails, this branch displays the debrief but leaves the stored attempt unfinished. After storage recovers, a reload resumes that attempt. The participant can then submit pool before the original deadline and emit drill_resolved, despite the incorrect first answer.

Preserve and retry the original completion, or invalidate measurement when completion cannot be saved. Do not expose the debrief and then resume the attempt as unfinished. An aborted IndexedDB transaction does not retain its writes. (w3.org)

Comment thread examples/first-minute/README.md Resolved
Comment thread README.md

Play the on-call engineer for 60 seconds. Inspect a checkout outage, submit a diagnosis, then adjust the connection pool to see how it affects the queue. This hackathon entry uses Flagsmith to test whether a three-step runbook improves first-attempt diagnoses.

[Download the offline drill](https://raw.githubusercontent.com/arpitkjaiswal/flagsmith1/main/examples/first-minute/media/first-minute.zip) · [Watch the walkthrough](examples/first-minute/media/walkthrough.mp4) · [See the Flagsmith results](examples/first-minute/media/flagsmith-results.png) · [Run the live experiment](examples/first-minute/README.md)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Major · Security & Privacy · Quick win

Do not host the download on a personal fork's main branch.

The download link points to the main branch of arpitkjaiswal/flagsmith1. The project does not control that repository. The fork owner can change the archive at any time, and the change does not pass project review. Readers who follow the README open first-minute.html on their own device. A changed archive can therefore run unreviewed JavaScript on that device.

Host the archive at a project-controlled location with a fixed, reviewable reference. The file that readers download must be the file that the project reviewed.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 issues found across 10 files (changes from recent commits).

Confidence score: 3/5

  • setup_experiment.py can crash during interactive setup when cookie authentication returns no token body. Handle a missing or non-dict response, or preserve the authenticated cookie session.
  • main.js skips flag retrieval for every saved attempt, even when the saved record does not confirm that exposure was recorded. A reload or termination before exposure recording can leave the attempt without that exposure; track whether recording succeeded before skipping retrieval.
Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="examples/first-minute/setup_experiment.py">

<violation number="1" location="examples/first-minute/setup_experiment.py:82">
P2: Interactive setup crashes when the local Flagsmith instance uses cookie authentication because `/auth/login/` returns no token body. Handle a missing/non-dict response (or preserve the authenticated cookie session) before calling `.get()`; otherwise require `FLAGSMITH_MANAGEMENT_TOKEN` up front for cookie-authenticated deployments.</violation>
</file>

<file name="examples/first-minute/main.js">

<violation number="1" location="examples/first-minute/main.js:270">
P2: This skips flag retrieval for every saved attempt, but the saved record does not indicate that the exposure was successfully recorded. If the page is reloaded or terminated after the attempt is persisted but before `getExperimentFlag()` runs, the resumed attempt never records an exposure and the experiment metrics undercount it; persist exposure completion or make resume retry that operation safely.</violation>
</file>

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

email = input().strip()
password = getpass.getpass("Local Flagsmith password: ")
response = request("auth/login/", {"email": email, "password": password})
if not response.get("key"):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Interactive setup crashes when the local Flagsmith instance uses cookie authentication because /auth/login/ returns no token body. Handle a missing/non-dict response (or preserve the authenticated cookie session) before calling .get(); otherwise require FLAGSMITH_MANAGEMENT_TOKEN up front for cookie-authenticated deployments.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At examples/first-minute/setup_experiment.py, line 82:

<comment>Interactive setup crashes when the local Flagsmith instance uses cookie authentication because `/auth/login/` returns no token body. Handle a missing/non-dict response (or preserve the authenticated cookie session) before calling `.get()`; otherwise require `FLAGSMITH_MANAGEMENT_TOKEN` up front for cookie-authenticated deployments.</comment>

<file context>
@@ -34,6 +64,39 @@ def records(path):
+        email = input().strip()
+        password = getpass.getpass("Local Flagsmith password: ")
+        response = request("auth/login/", {"email": email, "password": password})
+        if not response.get("key"):
+            parser.error(
+                "Sign in to local Flagsmith and supply your user token in FLAGSMITH_MANAGEMENT_TOKEN."
</file context>
Suggested change
if not response.get("key"):
if not isinstance(response, dict) or not response.get("key"):

fetch: (url, options) =>
fetch(url, { ...options, signal: options?.signal ?? AbortSignal.timeout(flagLookupTimeoutMs) }),
// An exposed player already has an assignment. Restore it while enabling the SDK to send their answer.
...(attempt && { preventFetch: true, defaultFlags: { [flagName]: { enabled: true, value: attempt.layout } } }),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This skips flag retrieval for every saved attempt, but the saved record does not indicate that the exposure was successfully recorded. If the page is reloaded or terminated after the attempt is persisted but before getExperimentFlag() runs, the resumed attempt never records an exposure and the experiment metrics undercount it; persist exposure completion or make resume retry that operation safely.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At examples/first-minute/main.js, line 270:

<comment>This skips flag retrieval for every saved attempt, but the saved record does not indicate that the exposure was successfully recorded. If the page is reloaded or terminated after the attempt is persisted but before `getExperimentFlag()` runs, the resumed attempt never records an exposure and the experiment metrics undercount it; persist exposure completion or make resume retry that operation safely.</comment>

<file context>
@@ -251,6 +256,22 @@ $('form').onsubmit = (event) => {
+        fetch: (url, options) =>
+            fetch(url, { ...options, signal: options?.signal ?? AbortSignal.timeout(flagLookupTimeoutMs) }),
+        // An exposed player already has an assignment. Restore it while enabling the SDK to send their answer.
+        ...(attempt && { preventFetch: true, defaultFlags: { [flagName]: { enabled: true, value: attempt.layout } } }),
+    });
+    return Boolean(flagsmith.getAllFlags()[flagName]);
</file context>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: 44a3daab-76ef-4ead-aa77-712111c36247
Commits

Reviewing files that changed from the base of the PR and between 80aab9d and 3d727e5.

Files ignored due to path filters: 1

  • examples/first-minute/media/first-minute.zip is excluded by !**/*.zip
Files selected for processing: 9

  • examples/first-minute/README.md
  • examples/first-minute/index.html
  • examples/first-minute/main.js
  • examples/first-minute/package.json
  • examples/first-minute/rehearsal-traffic.js
  • examples/first-minute/rehearse.py
  • examples/first-minute/setup_experiment.py
  • examples/first-minute/tests/first-attempt.spec.js
  • examples/first-minute/tests/test_rehearsal.py

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 5 remain after this review.

Comment on lines +209 to +210
if experiment["feature"]["id"] != flag["id"] or not any(
m["metric"] == metric["id"] for m in experiment["metrics"]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor · Data Integrity & Integration · Quick win

Reject an existing experiment whose allocation does not give both variants a share.

ensure returns a matching experiment without checking its rollout. The check after it covers only the feature and metric, so an existing experiment with 100% allocated to guided passes. rehearse() then samples that experiment, and the variant and split checks cannot compare the two arms. A missing control variant stops the rehearsal before the experiment is paused, so the experiment can remain running.

Before sampling, preparation must check the existing experiment's rollout allocation. It must stop with an explicit error when the allocation does not match what the rehearsal expects. An existing experiment with the expected 50/50 split must still be accepted.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants