Repository navigation
Add First Minute incident drill and Flagsmith experiment #9016
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
arpitkjaiswal
wants to merge
3
commits into
Flagsmith:main
Choose a base branch
from
arpitkjaiswal:main
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+3,311
−0
Open
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
ffc0d02
Add First Minute incident drill and Flagsmith experiment
arpitkrjaiswal-eng 80aab9d
Replay a verified Flagsmith rollout and protect first-attempt metrics
arpitkrjaiswal-eng 3d727e5
Keep First Minute playable through stalled assignments and reloads
arpitkrjaiswal-eng File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| # Client-side environment key for the project containing the triage_guide flag. | ||
| VITE_FLAGSMITH_ENVIRONMENT_ID= | ||
| # Optional API origin for self-hosted deployments. Local Vite uses its SDK proxy. | ||
| VITE_FLAGSMITH_ORIGIN= | ||
| # Optional management dashboard link shown in the connected build. | ||
| VITE_FLAGSMITH_DASHBOARD_URL= |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| node_modules/ | ||
| dist/ | ||
| .env* | ||
| !.env.example | ||
| test-results/ | ||
| playwright-report/ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,62 @@ | ||||||
| # First Minute | ||||||
|
|
||||||
| A 60-second incident drill that tests whether a short runbook helps people diagnose a checkout outage. Participants inspect a release log, service metrics and request traces, choose one diagnosis, and see where the slow request spent its time. | ||||||
|
|
||||||
|  | ||||||
|
|
||||||
| [Watch the 38-second captioned walkthrough](media/walkthrough.mp4) | ||||||
|
|
||||||
| ## Play | ||||||
|
|
||||||
| For a no-install practice round, [download the drill](https://raw.githubusercontent.com/arpitkjaiswal/flagsmith1/main/examples/first-minute/media/first-minute.zip), unzip it and open `first-minute.html`. The debrief includes a capacity model where you can adjust the pool size and see the resulting queue growth. | ||||||
|
|
||||||
| To run from source, run `npm ci` and `npm run dev` with Node 24. Open the address Vite prints. Without a client-side environment key, the drill runs in practice mode. To make a portable, offline copy, run `npm run build:practice` and open `dist/first-minute.html`. Finish a round and choose **Try the other desk** to compare the layouts. | ||||||
|
|
||||||
| To rebuild the ZIP distributed on GitHub, run `npm run package:practice` with Node 24 and Python 3 installed. | ||||||
|
|
||||||
| ## Run the experiment locally | ||||||
|
|
||||||
| 1. From the repository root, run `docker compose up -d`. Open `http://localhost:8000`, sign in and create an organisation if this is a fresh install. The supplied Compose file enables Experimentation and stores SDK events in PostgreSQL. | ||||||
| 2. From `examples/first-minute`, run `python3 setup_experiment.py` in a terminal. Sign in with your local Flagsmith email and password. If your account belongs to several organisations, the script lists their names and IDs so you can rerun it with the chosen ID. It creates or reuses the First Minute project, Participant drill environment, warehouse connection, flag, metric and experiment. Copy the two printed `VITE_` lines into `.env.local`. For unattended setup, provide your local user token in `FLAGSMITH_MANAGEMENT_TOKEN` and run `python3 setup_experiment.py ORGANISATION_ID`. | ||||||
| 3. Start or restart Vite. Its local SDK proxy forwards flags, identities and events to Flagsmith on port 8000. The briefing should say **Connected to Flagsmith**. Start the drill to enrol. For a shared deployment, set `VITE_FLAGSMITH_ORIGIN` to the HTTPS origin serving the Flagsmith API and SDK events. Set `VITE_FLAGSMITH_DASHBOARD_URL` to the address where your team signs in to view that experiment. Run `npm run build` and serve `dist` over HTTPS. | ||||||
| 4. Open **Experiments** in the Participant drill environment and refresh the results after events arrive. The guide is a keyed variation of the `triage_guide` multivariate flag. All eligible identities enter the experiment, with equal allocation between control and guided. | ||||||
|
|
||||||
| If the self-hosted dashboard shows **Maintenance** because its bootstrap request is blocked, configure `FLAGSMITH_ON_FLAGSMITH_API_URL` and `FLAGSMITH_ON_FLAGSMITH_API_KEY` on the web service to read from a local dashboard configuration environment, then restart the service. [Self-hosting guide](../../docs/docs/deployment-self-hosting/core-configuration/running-flagsmith-on-flagsmith.md) | ||||||
|
|
||||||
| ## What the experiment measures | ||||||
|
|
||||||
| **Hypothesis.** A three-step runbook increases the fraction of first-time participants who correctly diagnose connection-pool exhaustion within 60 seconds. Both variants use identical evidence, answer choices, time limit and feedback. | ||||||
|
|
||||||
| **Measurement.** The browser stores an anonymous identity, the assigned desk and the first attempt's deadline. Opening the desk records an SDK exposure. Only a correct first answer before expiry records the `drill_resolved` occurrence event. A returning player resumes the same desk and deadline directly from the browser. Tabs sharing the same browser identity share one first answer. New players can practise if Flagsmith takes more than four seconds to assign a desk. Practice and replays are excluded from the experiment. A new browser profile is a new identity. | ||||||
|
|
||||||
| **Evidence.** The [browser smoke-test snapshot](verification.json) recorded three exposures and two conversions. The in-app **Experiment replay** follows a larger synthetic run through assignment, measurement and rollout. Both runs used the real Flagsmith SDK and a local Flagsmith instance. | ||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. P2: The browser smoke tests do not use a local Flagsmith instance: they mock identity responses and capture events with Playwright route handlers. Clarify that only the rehearsal uses the instance, so the evidence does not overstate what the smoke tests verified. Prompt for AI agents
Suggested change
|
||||||
|
|
||||||
| **Rollout.** Collect independent participant attempts and inspect sample-ratio warnings. Flagsmith shows “Collecting data” below its inference thresholds. Once sufficient evidence supports a decision, end the participant experiment and use the same flag to roll out the selected desk. If the evidence stays inconclusive, continue collecting data. | ||||||
|
|
||||||
| ## Replay the Flagsmith experiment | ||||||
|
|
||||||
| Open **Experiment replay** in the drill. Assignment shows how the multivariate flag split 240 simulated identities. Results compares 83 conversions from 114 guided attempts with 54 from 126 standard attempts. The script programmed a 72% success chance for guided and 45% for standard. Flagsmith calculated the conversion rates, relative lift and win probability for this synthetic sample. Human outcomes are measured in the **Participant drill** environment. | ||||||
|
|
||||||
| Rollout shows what happened after the synthetic experiment closed. The same `triage_guide` flag was set to 100% guided in **Synthetic rollout rehearsal**. All 16 post-rollout SDK checks received the guided desk outside the experiment. The replay and its downloadable data work offline. | ||||||
|
|
||||||
|  | ||||||
|
|
||||||
| To reproduce the capture, run the following from this directory against the local Flagsmith instance. It accepts the same sign-in and optional organisation ID as the participant setup. | ||||||
|
|
||||||
| ```sh | ||||||
| python3 rehearse.py | ||||||
| ``` | ||||||
|
|
||||||
| This command provisions the dedicated rehearsal environment. It sends seeded traffic through the JavaScript SDK, waits for the warehouse counts to match, and uses the Flagsmith results to select a desk. Results refreshes respect the server's five-minute cooldown. The script pauses enrolment before closing the rehearsal and updating the environment default. Each run against a completed rehearsal verifies the rollout with 16 fresh identities. The JSON contains aggregate evidence only. The allocation check uses an exact two-sided binomial test against 50/50 when Flagsmith leaves its sample-ratio statistic empty. | ||||||
|
|
||||||
| The [captured JSON](rehearsal.json) drives the replay. To replace it after your own successful run, save the command output to a temporary file, then move it to `rehearsal.json`. The dashboard capture is a separate image taken from the original run shown here. | ||||||
|
|
||||||
| ## Browser checks | ||||||
|
|
||||||
| Run `npx playwright install chromium` once, then `npm test`. To use an existing Chromium installation, set `CHROMIUM_EXECUTABLE` to its path. The browser tests intercept SDK traffic locally and cover simultaneous submissions, interrupted connections, keyboard navigation, timeouts and practice exclusions. The rehearsal command above exercises the real API and warehouse. | ||||||
|
|
||||||
| With pytest installed, run `python3 -m pytest tests/test_rehearsal.py` to check recovery when Flagsmith asks the rehearsal to wait before refreshing results. | ||||||
|
|
||||||
| ## Submission copy | ||||||
|
|
||||||
| When checkout slows down, a responder often has a dashboard full of signals and has to work out what changed. First Minute lets anyone take the on-call seat. Players have 60 seconds to trace a slowdown through a release, service metrics and a slow request. Flagsmith assigns a plain desk or a three-step runbook and measures whether the player finds the root cause on their first attempt. The result helps the team decide whether every new responder should get the guide. | ||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,45 @@ | ||
| // IndexedDB serialises read/write transactions across tabs. A first answer can | ||
| // therefore claim the conversion even when two windows submit at the same time. | ||
| let connection; | ||
| function openDatabase() { | ||
| if (!connection) { | ||
| connection = new Promise((resolve, reject) => { | ||
| const request = indexedDB.open('first-minute', 1); | ||
| request.onupgradeneeded = () => request.result.createObjectStore('rounds'); | ||
| request.onsuccess = () => resolve(request.result); | ||
| request.onerror = () => reject(request.error); | ||
| }); | ||
| } | ||
| return connection; | ||
| } | ||
|
|
||
| function readLegacy(key, parse) { | ||
| try { | ||
| const value = localStorage.getItem(key); | ||
| return value === null ? undefined : parse(value); | ||
| } catch { | ||
| return undefined; | ||
| } | ||
| } | ||
|
|
||
| export async function updateRecord(key, change, parseLegacy = JSON.parse) { | ||
| const database = await openDatabase(); | ||
| return new Promise((resolve, reject) => { | ||
| const transaction = database.transaction('rounds', 'readwrite'); | ||
| const store = transaction.objectStore('rounds'); | ||
| const read = store.get(key); | ||
| let next; | ||
| let failure; | ||
| transaction.oncomplete = () => resolve(next); | ||
| transaction.onabort = () => reject(failure || transaction.error); | ||
| read.onsuccess = () => { | ||
| try { | ||
| next = change(read.result ?? readLegacy(key, parseLegacy)); | ||
| if (next !== undefined && next !== read.result) store.put(next, key); | ||
| } catch (error) { | ||
| failure = error; | ||
| transaction.abort(); | ||
| } | ||
| }; | ||
| }); | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| import { build } from 'vite'; | ||
| import { mkdir, writeFile } from 'node:fs/promises'; | ||
|
|
||
| // Package the practice desk for judges who want to open it directly from a file. | ||
| const bundle = await build({ mode: 'practice', build: { write: false } }); | ||
| const html = bundle.output.find((file) => file.fileName === 'index.html').source; | ||
| const script = bundle.output.find((file) => file.type === 'chunk' && file.isEntry).code; | ||
| const css = bundle.output.find((file) => file.fileName.endsWith('.css')).source; | ||
| const document = html | ||
| .replace( | ||
| /<script type="module"[^>]*src="[^"]+"[^>]*><\/script>/, | ||
| () => '<script type="module">' + script.replace(/<\/script/gi, '<\\/script') + '</script>', | ||
| ) | ||
| .replace(/<link rel="stylesheet"[^>]*>/, () => '<style>' + css + '</style>'); | ||
| await mkdir('dist', { recursive: true }); | ||
| await writeFile('dist/first-minute.html', document); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,88 @@ | ||
| import report from './rehearsal.json'; | ||
| import dashboardCapture from './media/flagsmith-results.png?inline'; | ||
|
|
||
| const $ = (id) => document.getElementById(id); | ||
| const labels = { control: 'Standard desk', guided: 'Guided desk' }; | ||
| const { control, guided } = report.variants; | ||
| const enrolled = control.exposed + guided.exposed; | ||
| const percent = new Intl.NumberFormat('en-GB', { style: 'percent', maximumFractionDigits: 1 }); | ||
| const signed = new Intl.NumberFormat('en-GB', { signDisplay: 'exceptZero', maximumFractionDigits: 1 }); | ||
| const rate = (desk) => desk.converted / desk.exposed; | ||
|
|
||
| $( | ||
| 'replay-source', | ||
| ).textContent = `Flagsmith measured ${enrolled} synthetic attempts. The rehearsal programmed a ${percent.format( | ||
| report.simulated_success_rates.control, | ||
| )} success chance for the standard desk and ${percent.format(report.simulated_success_rates.guided)} for guided.`; | ||
|
|
||
| $( | ||
| 'allocation-description', | ||
| ).textContent = `The ${report.flag} flag allocated ${report.configured_split.control}% of traffic to standard and ${report.configured_split.guided}% to guided. Each identity kept its assigned desk.`; | ||
| $('split-standard').style.width = (100 * control.exposed) / enrolled + '%'; | ||
| $('assigned-control').textContent = control.exposed + ' standard'; | ||
| $('assigned-guided').textContent = guided.exposed + ' guided'; | ||
|
|
||
| for (const key of ['control', 'guided']) { | ||
| const row = report.variants[key]; | ||
| const group = document.createElement('div'); | ||
| group.className = 'rate-row ' + key; | ||
| const title = document.createElement('strong'); | ||
| title.textContent = labels[key]; | ||
| const count = document.createElement('span'); | ||
| count.textContent = `${row.converted} of ${row.exposed} · ${percent.format(rate(row))}`; | ||
| const track = document.createElement('div'); | ||
| track.className = 'rate-track'; | ||
| track.setAttribute('aria-hidden', 'true'); | ||
| const fill = document.createElement('span'); | ||
| fill.style.width = 100 * rate(row) + '%'; | ||
| track.append(fill); | ||
| group.append(title, count, track); | ||
| $('result-bars').append(group); | ||
| } | ||
|
|
||
| const inference = report.guided_inference; | ||
| $('difference').textContent = signed.format(100 * (rate(guided) - rate(control))) + ' pp'; | ||
| $('win-chance').textContent = | ||
| inference.chance_to_win > 0.999 && inference.chance_to_win < 1 ? '>99.9%' : percent.format(inference.chance_to_win); | ||
| $('stat-detail').textContent = `Relative lift ${signed.format( | ||
| 100 * inference.lift, | ||
| )}% (95% credible interval ${signed.format(100 * inference.ci_low)}% to ${signed.format(100 * inference.ci_high)}%). ${ | ||
| report.allocation_check.method | ||
| }, p = ${report.allocation_check.p_value.toFixed(3)}.`; | ||
| const shipped = report.rollout; | ||
| $('rollout-checks').textContent = `${shipped.new_identities_verified} / ${shipped.new_identities_verified}`; | ||
| $('rollout-description').textContent = `After the rehearsal closed, the same flag was set to serve the ${labels[ | ||
| shipped.desk | ||
| ].toLowerCase()} to ${shipped.percentage}% of users in that environment. All ${ | ||
| shipped.new_identities_verified | ||
| } fresh test identities received that desk outside the experiment.`; | ||
|
|
||
| const date = new Date(report.as_of); | ||
| $('replay-date').dateTime = report.as_of; | ||
| $('replay-date').textContent = | ||
| 'Captured ' + new Intl.DateTimeFormat('en-GB', { dateStyle: 'medium', timeZone: 'UTC' }).format(date); | ||
| $('download-results').href = | ||
| 'data:application/json;charset=utf-8,' + encodeURIComponent(JSON.stringify(report, null, 2) + '\n'); | ||
|
|
||
| $('dashboard-capture').src = dashboardCapture; | ||
| const liveDashboard = import.meta.env.MODE !== 'practice' && import.meta.env.VITE_FLAGSMITH_DASHBOARD_URL; | ||
| if (liveDashboard) { | ||
| $('participant-dashboard').href = liveDashboard; | ||
| $('participant-dashboard').hidden = false; | ||
| } | ||
|
|
||
| const steps = document.querySelectorAll('.replay-stages button'); | ||
| function selectStage(stage) { | ||
| for (const step of steps) { | ||
| const panel = step.getAttribute('aria-controls'); | ||
| step.setAttribute('aria-pressed', String(panel === stage)); | ||
| $(panel).hidden = panel !== stage; | ||
| } | ||
| } | ||
| for (const step of steps) step.onclick = () => selectStage(step.getAttribute('aria-controls')); | ||
| function openResults() { | ||
| selectStage('measurement'); | ||
| $('notes').showModal(); | ||
| } | ||
| $('method').onclick = openResults; | ||
| $('show-experiment').onclick = openResults; |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do not host the download on a personal fork's
mainbranch.The download link points to the
mainbranch ofarpitkjaiswal/flagsmith1. The project does not control that repository. The fork owner can change the archive at any time, and the change does not pass project review. Readers who follow the README openfirst-minute.htmlon their own device. A changed archive can therefore run unreviewed JavaScript on that device.Host the archive at a project-controlled location with a fixed, reviewable reference. The file that readers download must be the file that the project reviewed.