You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Add First Minute incident drill and Flagsmith experiment - #9016
This change adds a timed checkout incident drill with evidence panels, diagnosis submission, a connection-pool calculator, and result views. It adds Flagsmith experiment setup, browser-side assignment and event tracking, and a rehearsal that samples traffic, checks warehouse results, and applies a verified rollout. The project also includes a Vite proxy, a standalone practice HTML build, automated tests, verification data, and setup and use instructions.
The drill and rehearsal are mostly in place. Before merge, fix the first-attempt measurement gaps: offline practice, lost conversions, unconfirmed delivery and failed saves. Also make experiment setup reject a reused experiment with the wrong allocation, and host the downloadable build somewhere the project controls.
Finishing Touches
Fix failing CI checks
Create a new PR
Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
The reason will be displayed to describe this comment to others. Learn more.
3 issues found across 16 files
Confidence score: 3/5
In setup_experiment.py, the SRM check gets no expected shares after the guided allocation is reset, so it may not run for the 50/50 experiment. Read the active rollout allocation.
In setup_experiment.py, a paused experiment exits as “not running” instead of resuming. Call start/ for paused experiments.
In README.md, both download links point to arpitkjaiswal/flagsmith1 rather than this repository. Link to Flagsmith/flagsmith so the downloads stay available here.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="examples/first-minute/setup_experiment.py">
<violation number="1" location="examples/first-minute/setup_experiment.py:78">
P2: This configures a 50/50 experiment, but the backend reads expected shares from the environment defaults after resetting the guided allocation to 0%, so its SRM check returns `None`. Read the active rollout allocation when calculating expected shares; otherwise the advertised sample-ratio warning cannot catch assignment drift for this experiment.</violation>
<violation number="2" location="examples/first-minute/setup_experiment.py:81">
P2: Could this also call `start/` for a paused experiment? The API allows `paused` → `running`, but this branch exits with “The experiment is not running” instead of resuming it.</violation>
</file>
<file name="README.md">
<violation number="1" location="README.md:5">
P2: Both advertised downloads point to `arpitkjaiswal/flagsmith1`, although the ZIP is committed in this repository. Link to `Flagsmith/flagsmith` so the public download cannot drift from or disappear independently of the shipped artifact.</violation>
</file>
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.
The reason will be displayed to describe this comment to others. Learn more.
P2: Could this also call start/ for a paused experiment? The API allows paused → running, but this branch exits with “The experiment is not running” instead of resuming it.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At examples/first-minute/setup_experiment.py, line 81:
<comment>Could this also call `start/` for a paused experiment? The API allows `paused` → `running`, but this branch exits with “The experiment is not running” instead of resuming it.</comment>
<file context>
@@ -0,0 +1,93 @@
+ "multivariate_feature_state_values": [{"multivariate_feature_option": variant["id"], "percentage_allocation": 50}]},
+ }, lambda e: e["name"] == name)
+ detail = path + str(experiment["id"]) + "/"
+ if experiment["status"] == "created":
+ request(detail + "start/", {})
+ if request(detail)["status"] != "running":
</file context>
Play the on-call engineer for 60 seconds. Inspect a checkout outage, submit a diagnosis, then adjust the connection pool to see how it affects the queue. This hackathon entry uses Flagsmith to test whether a three-step runbook improves first-attempt diagnoses.
[Download the offline drill](https://raw.githubusercontent.com/arpitkjaiswal/flagsmith1/main/examples/first-minute/media/first-minute.zip) · [Watch the walkthrough](examples/first-minute/media/walkthrough.mp4) · [Run the live experiment](examples/first-minute/README.md)
The reason will be displayed to describe this comment to others. Learn more.
P2: Both advertised downloads point to arpitkjaiswal/flagsmith1, although the ZIP is committed in this repository. Link to Flagsmith/flagsmith so the public download cannot drift from or disappear independently of the shipped artifact.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At README.md, line 5:
<comment>Both advertised downloads point to `arpitkjaiswal/flagsmith1`, although the ZIP is committed in this repository. Link to `Flagsmith/flagsmith` so the public download cannot drift from or disappear independently of the shipped artifact.</comment>
<file context>
@@ -1,3 +1,11 @@
+
+Play the on-call engineer for 60 seconds. Inspect a checkout outage, submit a diagnosis, then adjust the connection pool to see how it affects the queue. This hackathon entry uses Flagsmith to test whether a three-step runbook improves first-attempt diagnoses.
+
+[Download the offline drill](https://raw.githubusercontent.com/arpitkjaiswal/flagsmith1/main/examples/first-minute/media/first-minute.zip) · [Watch the walkthrough](examples/first-minute/media/walkthrough.mp4) · [Run the live experiment](examples/first-minute/README.md)
+
+Unzip the download and open `first-minute.html` in a browser. Practice rounds run entirely on your device.
</file context>
The reason will be displayed to describe this comment to others. Learn more.
P2: This configures a 50/50 experiment, but the backend reads expected shares from the environment defaults after resetting the guided allocation to 0%, so its SRM check returns None. Read the active rollout allocation when calculating expected shares; otherwise the advertised sample-ratio warning cannot catch assignment drift for this experiment.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At examples/first-minute/setup_experiment.py, line 78:
<comment>This configures a 50/50 experiment, but the backend reads expected shares from the environment defaults after resetting the guided allocation to 0%, so its SRM check returns `None`. Read the active rollout allocation when calculating expected shares; otherwise the advertised sample-ratio warning cannot catch assignment drift for this experiment.</comment>
<file context>
@@ -0,0 +1,93 @@
+ "metrics": [{"metric": metric["id"], "expected_direction": "increase"}],
+ "experiment_rollout": {"enabled": True, "rollout_percentage": 100,
+ "feature_state_value": {"type": "string", "value": "control"},
+ "multivariate_feature_state_values": [{"multivariate_feature_option": variant["id"], "percentage_allocation": 50}]},
+ }, lambda e: e["name"] == name)
+ detail = path + str(experiment["id"]) + "/"
</file context>
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 4
Review info
Run configuration
Source mode: Auto
Repository: Unavailable
Branch: Unavailable
File: No configuration file from this repository applied
Commit: Unavailable
Layers: Organization UI
Review profile: ASSERTIVE
Plan: Advanced
Run ID: a5222e38-40b1-4246-bd21-bdb01c4f8d3b
Commits
Reviewing files that changed from the base of the PR and between fe9b4cd and ffc0d02.
Files ignored due to path filters: 4
examples/first-minute/media/first-minute.png is excluded by !**/*.png
examples/first-minute/media/first-minute.zip is excluded by !**/*.zip
examples/first-minute/media/walkthrough.mp4 is excluded by !**/*.mp4
examples/first-minute/package-lock.json is excluded by !**/package-lock.json
Files selected for processing: 12
README.md
examples/first-minute/.env.example
examples/first-minute/.gitignore
examples/first-minute/README.md
examples/first-minute/build-practice.js
examples/first-minute/index.html
examples/first-minute/main.js
examples/first-minute/package.json
examples/first-minute/setup_experiment.py
examples/first-minute/style.css
examples/first-minute/verification.json
examples/first-minute/vite.config.js
Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.
The reason will be displayed to describe this comment to others. Learn more.
Major · Data Integrity & Integration · Quick win
Record offline practice so it cannot later count as a first attempt.
When initialisation fails or the experiment is not configured, begin() starts the attempt with measured set to false. The localStorage write at line 165 runs only when measured is true, so a practice attempt stores no participation record. The participant identity persists across reloads. After connectivity returns, a reload can enrol the same participant in the experiment. A correct answer then records drill_resolved for someone who already practised.
Store a participation record when a practice attempt starts while the experiment is unavailable. Mark it so that initialise() does not resume it as an active attempt. With that record present, the enrolment check in begin() must not start a measured attempt for that participant. Keep connected first attempts unchanged. This preserves the first-time-participant measurement described in the First Minute README.
The reason will be displayed to describe this comment to others. Learn more.
Major · Data Integrity & Integration · Heavy lift
Retain correct results when a restored attempt is offline.
If the flags request fails after a reload, initialise() still restores the measured attempt. A correct answer before expiry sets done: true, but this condition skips the conversion because connected is false. Later reloads offer only practice, so the conversion is permanently lost.
Persist a pending conversion with the original participant and submission time. Deliver it after recovery without counting practice attempts.
The reason will be displayed to describe this comment to others. Learn more.
Major · Data Integrity & Integration · Heavy lift
Do not treat flush resolution as delivery success.
flushEvents() resolves after the SDK finishes its delivery attempt, including the single retry. A failed batch is dropped and logged only through the SDK logger. The catch at main.js:222-224 therefore never runs for a delivery failure, so Event delivery unconfirmed is never shown.
Events still in the SDK buffer are lost if the participant reloads before the flush completes. The attempt is already saved as done: true before the flush, so a reload does not retry the conversion.
Show the unconfirmed state only when the SDK reports a failed delivery. flushEvents() gives no failure signal, so the change needs a way to detect that failure before the warning can appear. If the conversion must survive a reload, keep the stored result pending until delivery succeeds.
The reason will be displayed to describe this comment to others. Learn more.
1 existing issue remains and 2 new issues found across 21 files (changes from recent commits).
Confidence score: 3/5
In main.js, an offline resume can save a result as done without emitting drill_resolved, and the next load only displays the result, so the conversion is permanently lost. Persist and retry the event.
In rehearse.py, the fallback assumes a 50/50 split even when it reuses an experiment with different rollout weights, so the allocation check can give the wrong result. Verify the existing weights before checking allocation.
In README.md, the smoke tests mock Flagsmith responses and capture events rather than using a local instance, which could lead readers to overestimate what the tests verify. Clarify that only the rehearsal uses the instance.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="examples/first-minute/README.md">
<violation number="1" location="examples/first-minute/README.md:30">
P2: The browser smoke tests do not use a local Flagsmith instance: they mock identity responses and capture events with Playwright route handlers. Clarify that only the rehearsal uses the instance, so the evidence does not overstate what the smoke tests verified.</violation>
</file>
<file name="examples/first-minute/main.js">
<violation number="1" location="examples/first-minute/main.js:230">
P2: An offline resume persists a measured result as `done` but skips `drill_resolved` here. On the next successful load, `initialise()` only displays that result, so the conversion is permanently lost; persist and retry a pending conversion after initialization.</violation>
</file>
Requires human review: Auto-approval blocked because this review re-detected 1 unresolved issue already reported by Cubic.
**Measurement.** The browser stores an anonymous identity and the first attempt's deadline. Opening the desk records an SDK exposure. Only a correct first answer before expiry records the `drill_resolved` occurrence event. Reloading preserves the deadline and restores a finished debrief. Tabs sharing the same browser identity share one first answer. Replays are practice. A new browser profile is a new identity.
**Evidence.** The [browser smoke-test snapshot](verification.json) recorded three exposures and two conversions. The in-app **Experiment replay** follows a larger synthetic run through assignment, measurement and rollout. Both runs used the real Flagsmith SDK and a local Flagsmith instance.
The reason will be displayed to describe this comment to others. Learn more.
P2: The browser smoke tests do not use a local Flagsmith instance: they mock identity responses and capture events with Playwright route handlers. Clarify that only the rehearsal uses the instance, so the evidence does not overstate what the smoke tests verified.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At examples/first-minute/README.md, line 30:
<comment>The browser smoke tests do not use a local Flagsmith instance: they mock identity responses and capture events with Playwright route handlers. Clarify that only the rehearsal uses the instance, so the evidence does not overstate what the smoke tests verified.</comment>
<file context>
@@ -15,20 +15,44 @@ To run from source, run `npm ci` and `npm run dev` with Node 24. Open the addres
+**Measurement.** The browser stores an anonymous identity and the first attempt's deadline. Opening the desk records an SDK exposure. Only a correct first answer before expiry records the `drill_resolved` occurrence event. Reloading preserves the deadline and restores a finished debrief. Tabs sharing the same browser identity share one first answer. Replays are practice. A new browser profile is a new identity.
-**Evidence.** The notes panel contains a dated snapshot of a separate browser verification run. Flagsmith reported two guided exposures, one control exposure and two conversions. The automated checks deliberately exercised correct and incorrect answers. These counts establish that assignment, event delivery and aggregation work. They cannot estimate the effect on people.
+**Evidence.** The [browser smoke-test snapshot](verification.json) recorded three exposures and two conversions. The in-app **Experiment replay** follows a larger synthetic run through assignment, measurement and rollout. Both runs used the real Flagsmith SDK and a local Flagsmith instance.
**Rollout.** Collect independent participant attempts and inspect sample-ratio warnings. Flagsmith shows “Collecting data” below its inference thresholds. Once sufficient evidence supports a decision, end the participant experiment and use the same flag to roll out the selected desk. If the evidence stays inconclusive, continue collecting data.
</file context>
Suggested change
**Evidence.** The [browser smoke-test snapshot](verification.json) recorded three exposures and two conversions. The in-app **Experiment replay** follows a larger synthetic run through assignment, measurement and rollout. Both runs used the real Flagsmith SDK and a local Flagsmith instance.
**Evidence.** The [browser smoke-test snapshot](verification.json) recorded three exposures and two conversions. The in-app **Experiment replay** follows a larger synthetic run through assignment, measurement and rollout. The replay used the real Flagsmith SDK and a local Flagsmith instance; the browser checks intercept SDK traffic locally.
The reason will be displayed to describe this comment to others. Learn more.
P2: An offline resume persists a measured result as done but skips drill_resolved here. On the next successful load, initialise() only displays that result, so the conversion is permanently lost; persist and retry a pending conversion after initialization.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At examples/first-minute/main.js, line 230:
<comment>An offline resume persists a measured result as `done` but skips `drill_resolved` here. On the next successful load, `initialise()` only displays that result, so the conversion is permanently lost; persist and retry a pending conversion after initialization.</comment>
<file context>
@@ -216,7 +205,32 @@ async function finish(answer) {
+ }
+ displayResult(first.finishedAt ? first : outcome);
+ if (first === outcome && outcome.measured && outcome.answer === 'pool') {
+ if (!connected) {
+ $('outcome').append(' · Event delivery unconfirmed');
+ return;
</file context>
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 5
Review info
Run configuration
Source mode: Auto
Repository: Unavailable
Branch: Unavailable
File: No configuration file from this repository applied
Commit: Unavailable
Layers: Organization UI
Review profile: ASSERTIVE
Plan: Advanced
Run ID: 97bda201-0672-4f77-8dce-93a7a03ae8f6
Commits
Reviewing files that changed from the base of the PR and between ffc0d02 and 80aab9d.
Files ignored due to path filters: 5
examples/first-minute/media/first-minute.png is excluded by !**/*.png
examples/first-minute/media/first-minute.zip is excluded by !**/*.zip
examples/first-minute/media/flagsmith-results.png is excluded by !**/*.png
examples/first-minute/media/walkthrough.mp4 is excluded by !**/*.mp4
examples/first-minute/package-lock.json is excluded by !**/package-lock.json
Files selected for processing: 16
README.md
examples/first-minute/.env.example
examples/first-minute/.gitignore
examples/first-minute/README.md
examples/first-minute/attempt-store.js
examples/first-minute/experiment-replay.js
examples/first-minute/index.html
examples/first-minute/main.js
examples/first-minute/package.json
examples/first-minute/playwright.config.js
examples/first-minute/rehearsal-traffic.js
examples/first-minute/rehearsal.json
examples/first-minute/rehearse.py
examples/first-minute/setup_experiment.py
examples/first-minute/style.css
examples/first-minute/tests/first-attempt.spec.js
Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 6 remain after this review.
The reason will be displayed to describe this comment to others. Learn more.
Minor · Functional Correctness · Quick win
Start the deadline after storage admits the new attempt.
If updateRecord takes five seconds, the desk opens with about 55 seconds remaining. The deadline already includes the storage wait. This contradicts the briefing, which says the clock begins when the participant enters the desk.
Create the new deadline when the start claim can proceed. Keep the stored deadline when another tab joins an existing attempt.
The reason will be displayed to describe this comment to others. Learn more.
Major · Data Integrity & Integration · Heavy lift
Do not restore a revealed diagnosis as an unfinished measured attempt.
If saving an incorrect answer fails, this branch displays the debrief but leaves the stored attempt unfinished. After storage recovers, a reload resumes that attempt. The participant can then submit pool before the original deadline and emit drill_resolved, despite the incorrect first answer.
Preserve and retry the original completion, or invalidate measurement when completion cannot be saved. Do not expose the debrief and then resume the attempt as unfinished. An aborted IndexedDB transaction does not retain its writes. (w3.org)
Play the on-call engineer for 60 seconds. Inspect a checkout outage, submit a diagnosis, then adjust the connection pool to see how it affects the queue. This hackathon entry uses Flagsmith to test whether a three-step runbook improves first-attempt diagnoses.
[Download the offline drill](https://raw.githubusercontent.com/arpitkjaiswal/flagsmith1/main/examples/first-minute/media/first-minute.zip) · [Watch the walkthrough](examples/first-minute/media/walkthrough.mp4) · [See the Flagsmith results](examples/first-minute/media/flagsmith-results.png) · [Run the live experiment](examples/first-minute/README.md)
The reason will be displayed to describe this comment to others. Learn more.
Major · Security & Privacy · Quick win
Do not host the download on a personal fork's main branch.
The download link points to the main branch of arpitkjaiswal/flagsmith1. The project does not control that repository. The fork owner can change the archive at any time, and the change does not pass project review. Readers who follow the README open first-minute.html on their own device. A changed archive can therefore run unreviewed JavaScript on that device.
Host the archive at a project-controlled location with a fixed, reviewable reference. The file that readers download must be the file that the project reviewed.
The reason will be displayed to describe this comment to others. Learn more.
2 issues found across 10 files (changes from recent commits).
Confidence score: 3/5
setup_experiment.py can crash during interactive setup when cookie authentication returns no token body. Handle a missing or non-dict response, or preserve the authenticated cookie session.
main.js skips flag retrieval for every saved attempt, even when the saved record does not confirm that exposure was recorded. A reload or termination before exposure recording can leave the attempt without that exposure; track whether recording succeeded before skipping retrieval.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="examples/first-minute/setup_experiment.py">
<violation number="1" location="examples/first-minute/setup_experiment.py:82">
P2: Interactive setup crashes when the local Flagsmith instance uses cookie authentication because `/auth/login/` returns no token body. Handle a missing/non-dict response (or preserve the authenticated cookie session) before calling `.get()`; otherwise require `FLAGSMITH_MANAGEMENT_TOKEN` up front for cookie-authenticated deployments.</violation>
</file>
<file name="examples/first-minute/main.js">
<violation number="1" location="examples/first-minute/main.js:270">
P2: This skips flag retrieval for every saved attempt, but the saved record does not indicate that the exposure was successfully recorded. If the page is reloaded or terminated after the attempt is persisted but before `getExperimentFlag()` runs, the resumed attempt never records an exposure and the experiment metrics undercount it; persist exposure completion or make resume retry that operation safely.</violation>
</file>
Reply with feedback, questions, or to request a fix.
The reason will be displayed to describe this comment to others. Learn more.
P2: Interactive setup crashes when the local Flagsmith instance uses cookie authentication because /auth/login/ returns no token body. Handle a missing/non-dict response (or preserve the authenticated cookie session) before calling .get(); otherwise require FLAGSMITH_MANAGEMENT_TOKEN up front for cookie-authenticated deployments.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At examples/first-minute/setup_experiment.py, line 82:
<comment>Interactive setup crashes when the local Flagsmith instance uses cookie authentication because `/auth/login/` returns no token body. Handle a missing/non-dict response (or preserve the authenticated cookie session) before calling `.get()`; otherwise require `FLAGSMITH_MANAGEMENT_TOKEN` up front for cookie-authenticated deployments.</comment>
<file context>
@@ -34,6 +64,39 @@ def records(path):
+ email = input().strip()
+ password = getpass.getpass("Local Flagsmith password: ")
+ response = request("auth/login/", {"email": email, "password": password})
+ if not response.get("key"):
+ parser.error(
+ "Sign in to local Flagsmith and supply your user token in FLAGSMITH_MANAGEMENT_TOKEN."
</file context>
The reason will be displayed to describe this comment to others. Learn more.
P2: This skips flag retrieval for every saved attempt, but the saved record does not indicate that the exposure was successfully recorded. If the page is reloaded or terminated after the attempt is persisted but before getExperimentFlag() runs, the resumed attempt never records an exposure and the experiment metrics undercount it; persist exposure completion or make resume retry that operation safely.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At examples/first-minute/main.js, line 270:
<comment>This skips flag retrieval for every saved attempt, but the saved record does not indicate that the exposure was successfully recorded. If the page is reloaded or terminated after the attempt is persisted but before `getExperimentFlag()` runs, the resumed attempt never records an exposure and the experiment metrics undercount it; persist exposure completion or make resume retry that operation safely.</comment>
<file context>
@@ -251,6 +256,22 @@ $('form').onsubmit = (event) => {
+ fetch: (url, options) =>
+ fetch(url, { ...options, signal: options?.signal ?? AbortSignal.timeout(flagLookupTimeoutMs) }),
+ // An exposed player already has an assignment. Restore it while enabling the SDK to send their answer.
+ ...(attempt && { preventFetch: true, defaultFlags: { [flagName]: { enabled: true, value: attempt.layout } } }),
+ });
+ return Boolean(flagsmith.getAllFlags()[flagName]);
</file context>
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 1
Review info
Run configuration
Source mode: Auto
Repository: Unavailable
Branch: Unavailable
File: No configuration file from this repository applied
Commit: Unavailable
Layers: Organization UI
Review profile: ASSERTIVE
Plan: Advanced
Run ID: 44a3daab-76ef-4ead-aa77-712111c36247
Commits
Reviewing files that changed from the base of the PR and between 80aab9d and 3d727e5.
Files ignored due to path filters: 1
examples/first-minute/media/first-minute.zip is excluded by !**/*.zip
Files selected for processing: 9
examples/first-minute/README.md
examples/first-minute/index.html
examples/first-minute/main.js
examples/first-minute/package.json
examples/first-minute/rehearsal-traffic.js
examples/first-minute/rehearse.py
examples/first-minute/setup_experiment.py
examples/first-minute/tests/first-attempt.spec.js
examples/first-minute/tests/test_rehearsal.py
Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 5 remain after this review.
The reason will be displayed to describe this comment to others. Learn more.
Minor · Data Integrity & Integration · Quick win
Reject an existing experiment whose allocation does not give both variants a share.
ensure returns a matching experiment without checking its rollout. The check after it covers only the feature and metric, so an existing experiment with 100% allocated to guided passes. rehearse() then samples that experiment, and the variant and split checks cannot compare the two arms. A missing control variant stops the rehearsal before the experiment is paused, so the experiment can remain running.
Before sampling, preparation must check the existing experiment's rollout allocation. It must stop with an explicit error when the allocation does not match what the rehearsal expects. An existing experiment with the expected 50/50 split must still be accepted.
This branch has not been deployed
No deployments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thanks for submitting a PR! Please check the boxes below:
docs/if required so people know about the feature.Changes
Contributes to
Please describe.
How did you test this code?
Please describe.