diff --git a/README.md b/README.md index 649f4ec..d86cb6d 100644 --- a/README.md +++ b/README.md @@ -131,7 +131,7 @@ End-to-end wall-clock per run, measured from `report.json`'s `meta.durationMs` o | `1` | + full `xcodebuild build` + `./gradlew assembleDebug` + home-screen capture + Stage 1 vision judge | ~2.5 min (barbershop-queue · 2026-05-24) | ~$0.13 (derived) | | `2` | + Rails server boot + scripted-CRUD walk via `mobile-mcp` + Stage 2 vision judge | ~7–8 min (sentova / sushi / task-tracker · 2026-05-23; orange-baton 7m 12s · 2026-10-17) | ~$0.22 (orange-baton, measured 2026-10-17) | -Cold builds, first-run cocoapods/gradle dependency resolution, or unbooted simulators add a one-time minute or two. `--tests` adds a few minutes and no API cost: the three suites run in parallel, so the same orange-baton VISUAL=2 run took 9m 52s with them (2026-10-18), about 2.5 min more. The self-repair loop (opt-in, hard-capped at 5 iterations) can extend a failing run substantially — budget for it if you set `NATIVEAPPTEMPLATE_REPAIR=on`. +Cold builds, first-run cocoapods/gradle dependency resolution, or unbooted simulators add a one-time minute or two. `--tests` adds a few minutes and no API cost: the three suites run in parallel, so the same orange-baton VISUAL=2 run took 9m 52s with them (2026-10-18), about 2.5 min more; a 0.4.2 re-run with `--domain=orangebaton.com` and `--rename-ja` took 9m 06s (Rails 24 s, Android 23 s and iOS 85 s of tests). The self-repair loop (opt-in, hard-capped at 5 iterations) can extend a failing run substantially — budget for it if you set `NATIVEAPPTEMPLATE_REPAIR=on`. **Cost.** The agent makes real `claude-opus-5-5` API calls ($4 / $20 per million input / output tokens; 5-minute cache writes $5, cache reads $0.40; rates in [`src/models.ts`](./src/models.ts)). How the numbers above were made, on 2026-10-15: @@ -140,13 +140,14 @@ Cold builds, first-run cocoapods/gradle dependency resolution, or unbooted simul - **Derived, per run:** VISUAL=1 = planner + Stage 1 on iOS and Android ≈ $0.13; VISUAL=2 adds Stage 2 on both ≈ $0.24. A judge retry after a transient render failure adds ~$0.05 per retried platform. - **Measured, per run (2026-10-17):** a passing paid-edition VISUAL=2 run of the Orange Baton spec (see [Validation report](#validation-report)) cost **$0.223** — planner 2,761 in / 874 out = $0.029, plus 24 judge calls (Stage 1 + Stage 2 on iOS and Android) = 3,456 in / 2,976 out / 17,320 cache-write / 86,600 cache-read = $0.195. That's in line with the derived ~$0.24. - **Checked against billing (2026-10-18):** three Orange Baton runs, as `tmp/trace/` logged them, summed to 17,835 in / 10,235 out / 47,584 cache-write / 237,920 cache-read tokens, ≈ $0.61 — the Claude Console's usage export for that key and day showed exactly the same token counts. Another VISUAL=2 run that day came to the same 24 judge calls, $0.2197 in all. +- **Reported per run (0.4.2, 2026-10-18):** a passing VISUAL=2 + `--tests` run of the same spec, with `--domain=orangebaton.com` and `--rename-ja Shop=案件 --rename-ja ItemTag=バトン`, recorded **$0.2163** in its own `report.json` — planner 2,761 in / 820 out = $0.0274, plus 24 judge calls = 3,456 in / 2,692 out / 17,320 cache-write / 86,600 cache-read = $0.1889. Same input and cache token counts as the trace-based measurement above; only the output tokens differ. - **Self-repair** (`NATIVEAPPTEMPLATE_REPAIR`): a toy fix converged in 3 turns for ~$0.10 (measured). A real Compose/Hilt build fix can take many more turns (up to 40); estimate ~$0.10–$1 per attempt, at most 5 attempts per run. Re-validation after a repair does not re-run the judge. **Check your own spend:** every run reports its own estimate. The CLI's closing summary prints it after the verdict: ``` overall: PASS -cost: ≈ $0.2197 (25 API calls: planner $0.0273, judge $0.1924) +cost: ≈ $0.2163 (25 API calls: planner $0.0274, judge $0.1889) ``` `report.json` has the per-agent breakdown under `cost` (calls, input / output / cache-write / cache-read tokens, USD; repair is the Agent SDK's own `total_cost_usd`), the HTML report shows it as an "API cost (estimate)" table, and the MCP tool returns it with the report. It is priced at the list rates above; for billed amounts, see usage per API key in the Claude Console, and set a workspace spend cap (see [Security](#security)) as a backstop — the agent exits non-zero on validation failure, but doesn't gate on spend itself. To total several runs, `npm --prefix run cost` (or `bash /scripts/cost.sh [trace_dir]`) adds up the per-call lines in `tmp/trace/` of the directory you ran the agent from; those logs append across runs, so clear `tmp/trace/` first to cost a single one. @@ -293,9 +294,9 @@ See [`docs/SPEC.md`](./docs/SPEC.md) for the full design. Every run writes a report of the validation results to the output directory: -[![Example validation report (paid edition, VISUAL=2): a self-contained HTML page showing the overall PASS verdict, a platform×layer matrix, per-layer detail, embedded iOS + Android home-screen captures with the vision judge's rationales, the Stage 2 walk-through filmstrip, the contract-parity check, and the domain rename plan](./docs/images/validation-report.png)](./docs/images/validation-report.png) +[![Example validation report (paid edition, VISUAL=2): a self-contained HTML page showing the overall PASS verdict, a platform×layer matrix, per-layer detail, formatter and test-suite results, embedded iOS + Android home-screen captures with the vision judge's rationales, the Stage 2 walk-through filmstrip, the contract-parity check, the API cost table, the domain rename plan and the app identifiers](./docs/images/validation-report.png)](./docs/images/validation-report.png) -*Example report from a paid-edition `NATIVEAPPTEMPLATE_VISUAL=2` run of `"a job hand-off tracker for small field-service companies: each project always has exactly one open baton held by one teammate; finishing it means choosing the next stage and the next holder"` with `--project-name="Orange Baton" --rename Shop=Project --rename ItemTag=Baton --rename Shopkeeper=Teammate` — all three layers green, Stage 2 walked 46/46 steps on iOS and 48/48 on Android, 7m 12s, $0.22. Click to view full size; the real artifact is a live, self-contained HTML file you open in a browser.* +*Example report from a paid-edition `NATIVEAPPTEMPLATE_VISUAL=2` run of `"a job hand-off tracker for small field-service companies: each project always has exactly one open baton held by one teammate; finishing it means choosing the next stage and the next holder"` with `--project-name="Orange Baton" --domain=orangebaton.com --rename Shop=Project --rename ItemTag=Baton --rename Shopkeeper=Teammate --rename-ja Shop=案件 --rename-ja ItemTag=バトン --tests` (agent 0.4.2) — all three layers green, Stage 2 walked 46/46 steps on iOS and 48/48 on Android, all three test suites passed, 9m 06s, $0.22. Click to view full size; the real artifact is a live, self-contained HTML file you open in a browser.* - **`out//validation-report.html`** — a self-contained HTML report (screenshots base64-embedded, no external assets, no JavaScript) you can open in a browser, attach to a PR, or drop into a demo. It shows the overall verdict — **PASS**, **FAIL**, or **PARTIAL** (nothing failed, but Layer 3 didn't judge the UI, as in a default `NATIVEAPPTEMPLATE_VISUAL=0` run) — a platform×layer matrix, Layer 1 leftover-token findings, Layer 2 build commands + `stderr`, Layer 3 home-screen screenshots with the vision judge's per-criterion rationales (plus the Stage 2 filmstrip when `NATIVEAPPTEMPLATE_VISUAL=2`), the reviewer's contract diff, the run's API cost, and the domain rename plan. - **`out//report.json`** — the same data, machine-readable, for CI gating or programmatic use. The full schema lives in [`docs/validation-report.md`](./docs/validation-report.md). diff --git a/docs/images/validation-report.png b/docs/images/validation-report.png index 7156ee0..b503836 100644 Binary files a/docs/images/validation-report.png and b/docs/images/validation-report.png differ diff --git a/docs/validation-report.md b/docs/validation-report.md index 3191391..8715ca0 100644 --- a/docs/validation-report.md +++ b/docs/validation-report.md @@ -193,7 +193,7 @@ Reuse the social-preview palette for brand coherence (`docs/social-preview.svg`) - Always write `out//report.json` + `validation-report.html`. - Print the report path as a `file://` URL (clickable in terminals) alongside the existing summary lines. -- When the run made API calls, print `cost: ≈ $0.2197 (25 API calls: planner $0.0273, judge $0.1924)` after the `overall:` line. +- When the run made API calls, print `cost: ≈ $0.2163 (25 API calls: planner $0.0274, judge $0.1889)` after the `overall:` line. Flags: