Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,11 @@
# First Minute

Play the on-call engineer for 60 seconds. Inspect a checkout outage, submit a diagnosis, then adjust the connection pool to see how it affects the queue. This hackathon entry uses Flagsmith to test whether a three-step runbook improves first-attempt diagnoses.

[Download the offline drill](https://raw.githubusercontent.com/arpitkjaiswal/flagsmith1/main/examples/first-minute/media/first-minute.zip) · [Watch the walkthrough](examples/first-minute/media/walkthrough.mp4) · [See the Flagsmith results](examples/first-minute/media/flagsmith-results.png) · [Run the live experiment](examples/first-minute/README.md)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Major · Security & Privacy · Quick win

Do not host the download on a personal fork's main branch.

The download link points to the main branch of arpitkjaiswal/flagsmith1. The project does not control that repository. The fork owner can change the archive at any time, and the change does not pass project review. Readers who follow the README open first-minute.html on their own device. A changed archive can therefore run unreviewed JavaScript on that device.

Host the archive at a project-controlled location with a fixed, reviewable reference. The file that readers download must be the file that the project reviewed.


Unzip the download and open `first-minute.html` in a browser. Practice rounds run entirely on your device. Select **Experiment replay** to inspect the synthetic results and the verified flag rollout.

[![Feature Flag, Remote Config and A/B Testing platform, Flagsmith](static-files/flagsmith-cover.png)](https://www.flagsmith.com/)

[![Stars](https://img.shields.io/github/stars/flagsmith/flagsmith)](https://github.com/Flagsmith/flagsmith/stargazers)
Expand Down
6 changes: 6 additions & 0 deletions examples/first-minute/.env.example
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# Client-side environment key for the project containing the triage_guide flag.
VITE_FLAGSMITH_ENVIRONMENT_ID=
# Optional API origin for self-hosted deployments. Local Vite uses its SDK proxy.
VITE_FLAGSMITH_ORIGIN=
# Optional management dashboard link shown in the connected build.
VITE_FLAGSMITH_DASHBOARD_URL=
6 changes: 6 additions & 0 deletions examples/first-minute/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
node_modules/
dist/
.env*
!.env.example
test-results/
playwright-report/
62 changes: 62 additions & 0 deletions examples/first-minute/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
# First Minute

A 60-second incident drill that tests whether a short runbook helps people diagnose a checkout outage. Participants inspect a release log, service metrics and request traces, choose one diagnosis, and see where the slow request spent its time.

![Checkout incident briefing](media/first-minute.png)

[Watch the 38-second captioned walkthrough](media/walkthrough.mp4)

## Play

For a no-install practice round, [download the drill](https://raw.githubusercontent.com/arpitkjaiswal/flagsmith1/main/examples/first-minute/media/first-minute.zip), unzip it and open `first-minute.html`. The debrief includes a capacity model where you can adjust the pool size and see the resulting queue growth.

To run from source, run `npm ci` and `npm run dev` with Node 24. Open the address Vite prints. Without a client-side environment key, the drill runs in practice mode. To make a portable, offline copy, run `npm run build:practice` and open `dist/first-minute.html`. Finish a round and choose **Try the other desk** to compare the layouts.

To rebuild the ZIP distributed on GitHub, run `npm run package:practice` with Node 24 and Python 3 installed.

## Run the experiment locally

1. From the repository root, run `docker compose up -d`. Open `http://localhost:8000`, sign in and create an organisation if this is a fresh install. The supplied Compose file enables Experimentation and stores SDK events in PostgreSQL.
2. From `examples/first-minute`, run `python3 setup_experiment.py` in a terminal. Sign in with your local Flagsmith email and password. If your account belongs to several organisations, the script lists their names and IDs so you can rerun it with the chosen ID. It creates or reuses the First Minute project, Participant drill environment, warehouse connection, flag, metric and experiment. Copy the two printed `VITE_` lines into `.env.local`. For unattended setup, provide your local user token in `FLAGSMITH_MANAGEMENT_TOKEN` and run `python3 setup_experiment.py ORGANISATION_ID`.
3. Start or restart Vite. Its local SDK proxy forwards flags, identities and events to Flagsmith on port 8000. The briefing should say **Connected to Flagsmith**. Start the drill to enrol. For a shared deployment, set `VITE_FLAGSMITH_ORIGIN` to the HTTPS origin serving the Flagsmith API and SDK events. Set `VITE_FLAGSMITH_DASHBOARD_URL` to the address where your team signs in to view that experiment. Run `npm run build` and serve `dist` over HTTPS.
4. Open **Experiments** in the Participant drill environment and refresh the results after events arrive. The guide is a keyed variation of the `triage_guide` multivariate flag. All eligible identities enter the experiment, with equal allocation between control and guided.

If the self-hosted dashboard shows **Maintenance** because its bootstrap request is blocked, configure `FLAGSMITH_ON_FLAGSMITH_API_URL` and `FLAGSMITH_ON_FLAGSMITH_API_KEY` on the web service to read from a local dashboard configuration environment, then restart the service. [Self-hosting guide](../../docs/docs/deployment-self-hosting/core-configuration/running-flagsmith-on-flagsmith.md)

## What the experiment measures

**Hypothesis.** A three-step runbook increases the fraction of first-time participants who correctly diagnose connection-pool exhaustion within 60 seconds. Both variants use identical evidence, answer choices, time limit and feedback.

**Measurement.** The browser stores an anonymous identity, the assigned desk and the first attempt's deadline. Opening the desk records an SDK exposure. Only a correct first answer before expiry records the `drill_resolved` occurrence event. A returning player resumes the same desk and deadline directly from the browser. Tabs sharing the same browser identity share one first answer. New players can practise if Flagsmith takes more than four seconds to assign a desk. Practice and replays are excluded from the experiment. A new browser profile is a new identity.

**Evidence.** The [browser smoke-test snapshot](verification.json) recorded three exposures and two conversions. The in-app **Experiment replay** follows a larger synthetic run through assignment, measurement and rollout. Both runs used the real Flagsmith SDK and a local Flagsmith instance.

@cubic-dev-ai cubic-dev-ai Bot Nov 19, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: The browser smoke tests do not use a local Flagsmith instance: they mock identity responses and capture events with Playwright route handlers. Clarify that only the rehearsal uses the instance, so the evidence does not overstate what the smoke tests verified.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At examples/first-minute/README.md, line 30:

<comment>The browser smoke tests do not use a local Flagsmith instance: they mock identity responses and capture events with Playwright route handlers. Clarify that only the rehearsal uses the instance, so the evidence does not overstate what the smoke tests verified.</comment>

<file context>
@@ -15,20 +15,44 @@ To run from source, run `npm ci` and `npm run dev` with Node 24. Open the addres
+**Measurement.** The browser stores an anonymous identity and the first attempt's deadline. Opening the desk records an SDK exposure. Only a correct first answer before expiry records the `drill_resolved` occurrence event. Reloading preserves the deadline and restores a finished debrief. Tabs sharing the same browser identity share one first answer. Replays are practice. A new browser profile is a new identity.
 
-**Evidence.** The notes panel contains a dated snapshot of a separate browser verification run. Flagsmith reported two guided exposures, one control exposure and two conversions. The automated checks deliberately exercised correct and incorrect answers. These counts establish that assignment, event delivery and aggregation work. They cannot estimate the effect on people.
+**Evidence.** The [browser smoke-test snapshot](verification.json) recorded three exposures and two conversions. The in-app **Experiment replay** follows a larger synthetic run through assignment, measurement and rollout. Both runs used the real Flagsmith SDK and a local Flagsmith instance.
 
 **Rollout.** Collect independent participant attempts and inspect sample-ratio warnings. Flagsmith shows “Collecting data” below its inference thresholds. Once sufficient evidence supports a decision, end the participant experiment and use the same flag to roll out the selected desk. If the evidence stays inconclusive, continue collecting data.
</file context>
Suggested change
**Evidence.** The [browser smoke-test snapshot](verification.json) recorded three exposures and two conversions. The in-app **Experiment replay** follows a larger synthetic run through assignment, measurement and rollout. Both runs used the real Flagsmith SDK and a local Flagsmith instance.
**Evidence.** The [browser smoke-test snapshot](verification.json) recorded three exposures and two conversions. The in-app **Experiment replay** follows a larger synthetic run through assignment, measurement and rollout. The replay used the real Flagsmith SDK and a local Flagsmith instance; the browser checks intercept SDK traffic locally.
Fix with cubic


**Rollout.** Collect independent participant attempts and inspect sample-ratio warnings. Flagsmith shows “Collecting data” below its inference thresholds. Once sufficient evidence supports a decision, end the participant experiment and use the same flag to roll out the selected desk. If the evidence stays inconclusive, continue collecting data.

## Replay the Flagsmith experiment

Open **Experiment replay** in the drill. Assignment shows how the multivariate flag split 240 simulated identities. Results compares 83 conversions from 114 guided attempts with 54 from 126 standard attempts. The script programmed a 72% success chance for guided and 45% for standard. Flagsmith calculated the conversion rates, relative lift and win probability for this synthetic sample. Human outcomes are measured in the **Participant drill** environment.

Rollout shows what happened after the synthetic experiment closed. The same `triage_guide` flag was set to 100% guided in **Synthetic rollout rehearsal**. All 16 post-rollout SDK checks received the guided desk outside the experiment. The replay and its downloadable data work offline.

![Completed synthetic experiment in the native Flagsmith dashboard](media/flagsmith-results.png)

To reproduce the capture, run the following from this directory against the local Flagsmith instance. It accepts the same sign-in and optional organisation ID as the participant setup.

```sh
python3 rehearse.py
```

This command provisions the dedicated rehearsal environment. It sends seeded traffic through the JavaScript SDK, waits for the warehouse counts to match, and uses the Flagsmith results to select a desk. Results refreshes respect the server's five-minute cooldown. The script pauses enrolment before closing the rehearsal and updating the environment default. Each run against a completed rehearsal verifies the rollout with 16 fresh identities. The JSON contains aggregate evidence only. The allocation check uses an exact two-sided binomial test against 50/50 when Flagsmith leaves its sample-ratio statistic empty.

The [captured JSON](rehearsal.json) drives the replay. To replace it after your own successful run, save the command output to a temporary file, then move it to `rehearsal.json`. The dashboard capture is a separate image taken from the original run shown here.

## Browser checks

Run `npx playwright install chromium` once, then `npm test`. To use an existing Chromium installation, set `CHROMIUM_EXECUTABLE` to its path. The browser tests intercept SDK traffic locally and cover simultaneous submissions, interrupted connections, keyboard navigation, timeouts and practice exclusions. The rehearsal command above exercises the real API and warehouse.

With pytest installed, run `python3 -m pytest tests/test_rehearsal.py` to check recovery when Flagsmith asks the rehearsal to wait before refreshing results.

## Submission copy

When checkout slows down, a responder often has a dashboard full of signals and has to work out what changed. First Minute lets anyone take the on-call seat. Players have 60 seconds to trace a slowdown through a release, service metrics and a slow request. Flagsmith assigns a plain desk or a three-step runbook and measures whether the player finds the root cause on their first attempt. The result helps the team decide whether every new responder should get the guide.
45 changes: 45 additions & 0 deletions examples/first-minute/attempt-store.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
// IndexedDB serialises read/write transactions across tabs. A first answer can
// therefore claim the conversion even when two windows submit at the same time.
let connection;
function openDatabase() {
if (!connection) {
connection = new Promise((resolve, reject) => {
const request = indexedDB.open('first-minute', 1);
request.onupgradeneeded = () => request.result.createObjectStore('rounds');
request.onsuccess = () => resolve(request.result);
request.onerror = () => reject(request.error);
});
}
return connection;
}

function readLegacy(key, parse) {
try {
const value = localStorage.getItem(key);
return value === null ? undefined : parse(value);
} catch {
return undefined;
}
}

export async function updateRecord(key, change, parseLegacy = JSON.parse) {
const database = await openDatabase();
return new Promise((resolve, reject) => {
const transaction = database.transaction('rounds', 'readwrite');
const store = transaction.objectStore('rounds');
const read = store.get(key);
let next;
let failure;
transaction.oncomplete = () => resolve(next);
transaction.onabort = () => reject(failure || transaction.error);
read.onsuccess = () => {
try {
next = change(read.result ?? readLegacy(key, parseLegacy));
if (next !== undefined && next !== read.result) store.put(next, key);
} catch (error) {
failure = error;
transaction.abort();
}
};
});
}
16 changes: 16 additions & 0 deletions examples/first-minute/build-practice.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
import { build } from 'vite';
import { mkdir, writeFile } from 'node:fs/promises';

// Package the practice desk for judges who want to open it directly from a file.
const bundle = await build({ mode: 'practice', build: { write: false } });
const html = bundle.output.find((file) => file.fileName === 'index.html').source;
const script = bundle.output.find((file) => file.type === 'chunk' && file.isEntry).code;
const css = bundle.output.find((file) => file.fileName.endsWith('.css')).source;
const document = html
.replace(
/<script type="module"[^>]*src="[^"]+"[^>]*><\/script>/,
() => '<script type="module">' + script.replace(/<\/script/gi, '<\\/script') + '</script>',
)
.replace(/<link rel="stylesheet"[^>]*>/, () => '<style>' + css + '</style>');
await mkdir('dist', { recursive: true });
await writeFile('dist/first-minute.html', document);
88 changes: 88 additions & 0 deletions examples/first-minute/experiment-replay.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
import report from './rehearsal.json';
import dashboardCapture from './media/flagsmith-results.png?inline';

const $ = (id) => document.getElementById(id);
const labels = { control: 'Standard desk', guided: 'Guided desk' };
const { control, guided } = report.variants;
const enrolled = control.exposed + guided.exposed;
const percent = new Intl.NumberFormat('en-GB', { style: 'percent', maximumFractionDigits: 1 });
const signed = new Intl.NumberFormat('en-GB', { signDisplay: 'exceptZero', maximumFractionDigits: 1 });
const rate = (desk) => desk.converted / desk.exposed;

$(
'replay-source',
).textContent = `Flagsmith measured ${enrolled} synthetic attempts. The rehearsal programmed a ${percent.format(
report.simulated_success_rates.control,
)} success chance for the standard desk and ${percent.format(report.simulated_success_rates.guided)} for guided.`;

$(
'allocation-description',
).textContent = `The ${report.flag} flag allocated ${report.configured_split.control}% of traffic to standard and ${report.configured_split.guided}% to guided. Each identity kept its assigned desk.`;
$('split-standard').style.width = (100 * control.exposed) / enrolled + '%';
$('assigned-control').textContent = control.exposed + ' standard';
$('assigned-guided').textContent = guided.exposed + ' guided';

for (const key of ['control', 'guided']) {
const row = report.variants[key];
const group = document.createElement('div');
group.className = 'rate-row ' + key;
const title = document.createElement('strong');
title.textContent = labels[key];
const count = document.createElement('span');
count.textContent = `${row.converted} of ${row.exposed} · ${percent.format(rate(row))}`;
const track = document.createElement('div');
track.className = 'rate-track';
track.setAttribute('aria-hidden', 'true');
const fill = document.createElement('span');
fill.style.width = 100 * rate(row) + '%';
track.append(fill);
group.append(title, count, track);
$('result-bars').append(group);
}

const inference = report.guided_inference;
$('difference').textContent = signed.format(100 * (rate(guided) - rate(control))) + ' pp';
$('win-chance').textContent =
inference.chance_to_win > 0.999 && inference.chance_to_win < 1 ? '>99.9%' : percent.format(inference.chance_to_win);
$('stat-detail').textContent = `Relative lift ${signed.format(
100 * inference.lift,
)}% (95% credible interval ${signed.format(100 * inference.ci_low)}% to ${signed.format(100 * inference.ci_high)}%). ${
report.allocation_check.method
}, p = ${report.allocation_check.p_value.toFixed(3)}.`;
const shipped = report.rollout;
$('rollout-checks').textContent = `${shipped.new_identities_verified} / ${shipped.new_identities_verified}`;
$('rollout-description').textContent = `After the rehearsal closed, the same flag was set to serve the ${labels[
shipped.desk
].toLowerCase()} to ${shipped.percentage}% of users in that environment. All ${
shipped.new_identities_verified
} fresh test identities received that desk outside the experiment.`;

const date = new Date(report.as_of);
$('replay-date').dateTime = report.as_of;
$('replay-date').textContent =
'Captured ' + new Intl.DateTimeFormat('en-GB', { dateStyle: 'medium', timeZone: 'UTC' }).format(date);
$('download-results').href =
'data:application/json;charset=utf-8,' + encodeURIComponent(JSON.stringify(report, null, 2) + '\n');

$('dashboard-capture').src = dashboardCapture;
const liveDashboard = import.meta.env.MODE !== 'practice' && import.meta.env.VITE_FLAGSMITH_DASHBOARD_URL;
if (liveDashboard) {
$('participant-dashboard').href = liveDashboard;
$('participant-dashboard').hidden = false;
}

const steps = document.querySelectorAll('.replay-stages button');
function selectStage(stage) {
for (const step of steps) {
const panel = step.getAttribute('aria-controls');
step.setAttribute('aria-pressed', String(panel === stage));
$(panel).hidden = panel !== stage;
}
}
for (const step of steps) step.onclick = () => selectStage(step.getAttribute('aria-controls'));
function openResults() {
selectStage('measurement');
$('notes').showModal();
}
$('method').onclick = openResults;
$('show-experiment').onclick = openResults;
Loading
Loading