Skip to content

Publish eval evidence on the Lucid site #62

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
triggeredcode merged 1 commit into from
Oct 20, 2026
Merged

Publish eval evidence on the Lucid site #62

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions apps/desktop/electron/site.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,28 @@ describe("site/index.html", () => {
expect(tag).toContain('href="https://github.com/triggeredcode/lucid/releases/latest"');
});

it("publishes the routing experiment with its decision and measurement context", () => {
const evidence = html.match(/<section\s+id="evals">([\s\S]*?)<\/section>/)?.[1] ?? "";
expect(evidence).toContain("Measured before it ships.");
expect(evidence).toContain("Routing experiment, dataset v2");
expect(evidence).toContain("Current explicit-state heuristic");
expect(evidence).toContain("Nomic Embed Text");
expect(evidence).toContain("Qwen3 Embedding 0.6B");
expect(evidence).toContain("Gemma 3 1B");
expect(evidence).toContain("Apple M4 Pro");
expect(evidence).toContain("synthetic fixtures");
expect(evidence).toContain("not every computer or model");
expect(evidence).toContain(
'href="https://github.com/triggeredcode/lucid/blob/main/docs/evals.md"',
);

const headings = evidence.match(/<th\s+scope="col">/g) ?? [];
const rows = [...evidence.matchAll(/<tr>([\s\S]*?)<\/tr>/g)].slice(1);
expect(headings).toHaveLength(7);
expect(rows).toHaveLength(3);
for (const row of rows) expect(row[1]?.match(/<td(?:\s+class="[^"]+")?>/g)).toHaveLength(7);
});

it("keeps every command and its Copy button side by side, with the command written exactly as shown", () => {
// The commands are in <pre>: any space or line break the formatter adds inside shows on the page, and a Copy button placed over
// the text hides it. So each box is a <pre><code> that starts at its first character, then the button, never over it.
Expand Down
90 changes: 90 additions & 0 deletions site/index.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@
<nav class="nav" aria-label="Sections">
<a href="#idea">The idea</a>
<a href="#how">How it works</a>
<a href="#evals">Evals</a>
<a href="#formats">Formats</a>
<a class="keep" href="#download">Download</a>
<a href="guide.html">Guide</a>
Expand Down Expand Up @@ -494,6 +495,95 @@ <h2>Typeset, not screenshotted.</h2>
</div>
</section>

<!-- ============ evidence ============ -->
<section id="evals">
<div class="wrap">
<div class="head">
<p class="kicker">Evidence</p>
<h2>Measured before it ships.</h2>
<p class="section-lead">
Lucid keeps frozen routing cases in the repository. A candidate stays out of the app
until it improves the baseline without weakening safety or the final task.
</p>
</div>

<div class="eval-decision reveal">
<strong>Current decision</strong>
<p>
Keep deterministic routing. Nomic handled more requests, but 55.6% semantic-scope
accuracy is not reliable enough to add a runtime model and its management cost.
</p>
</div>

<div class="eval-table-wrap reveal" style="--d: 0.08s" tabindex="0">
<table class="eval-table">
<caption>
Routing experiment, dataset v2
</caption>
<thead>
<tr>
<th scope="col">Candidate</th>
<th scope="col">Routing pass</th>
<th scope="col">Semantic scope</th>
<th scope="col">Vague near-miss safety</th>
<th scope="col">Warm p50 / p95</th>
<th scope="col">Cold max</th>
<th scope="col">Verdict</th>
</tr>
</thead>
<tbody>
<tr>
<td>Current explicit-state heuristic</td>
<td>37 / 46</td>
<td>0.0%</td>
<td>100.0%</td>
<td>&lt;1 ms</td>
<td>&lt;1 ms</td>
<td class="eval-verdict">Shipped baseline</td>
</tr>
<tr>
<td>Nomic Embed Text <small>margin 0.04</small></td>
<td>42 / 46</td>
<td>55.6%</td>
<td>100.0%</td>
<td>9.6 / 11.5 ms</td>
<td>175 ms</td>
<td class="eval-verdict">Experiment only</td>
</tr>
<tr>
<td>Qwen3 Embedding 0.6B <small>margin 0.04</small></td>
<td>39 / 46</td>
<td>22.2%</td>
<td>100.0%</td>
<td>24.7 / 34.3 ms</td>
<td>415 ms</td>
<td class="eval-verdict">Not advanced</td>
</tr>
</tbody>
</table>
</div>

<div class="eval-foot reveal" style="--d: 0.16s">
<div>
<strong>Decision model rejected on calibration</strong>
<p>
Gemma 3 1B passed 12 of 16 routes but scored 0% on vague near-miss safety. It took
381 / 483 ms warm and up to 1.69 s cold, so its holdout set was never opened.
</p>
</div>
<div class="eval-notes">
<p>
Dataset v2 · synthetic fixtures · Apple M4 Pro. These measurements describe this
experiment, not every computer or model.
</p>
<a href="https://github.com/triggeredcode/lucid/blob/main/docs/evals.md"
>Read the methodology and reproduce it</a
>
</div>
</div>
</div>
</section>

<!-- ============ formats ============ -->
<section id="formats">
<div class="wrap">
Expand Down
102 changes: 102 additions & 0 deletions site/style.css
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -274,6 +274,12 @@ pre {
display: none;
}
}
@media (min-width: 761px) and (max-width: 1100px) {
.bar,
.nav {
gap: var(--s4);
}
}

/* ---- buttons ---- */
.btn {
Expand Down Expand Up @@ -1044,6 +1050,102 @@ details[open] summary::after {
}
}

/* ---- evaluation evidence ---- */
.eval-decision {
display: grid;
grid-template-columns: minmax(140px, 1fr) minmax(0, 4fr);
gap: var(--s5);
padding: var(--s5) 0;
border-block: 2px solid var(--ink);
}
.eval-decision p {
max-width: 72ch;
color: var(--ink-2);
}
.eval-table-wrap {
margin-top: var(--s6);
overflow-x: auto;
border: 1px solid var(--line-strong);
border-radius: var(--r2);
background: var(--bg);
}
.eval-table {
width: 100%;
min-width: 920px;
border-collapse: collapse;
font-size: 14px;
line-height: 1.45;
}
.eval-table caption {
padding: var(--s4);
border-bottom: 1px solid var(--line-strong);
color: var(--ink-2);
font-weight: 700;
text-align: left;
}
.eval-table th,
.eval-table td {
padding: var(--s4);
border-right: 1px solid var(--line);
border-bottom: 1px solid var(--line);
text-align: left;
vertical-align: top;
}
.eval-table th:last-child,
.eval-table td:last-child {
border-right: 0;
}
.eval-table tbody tr:last-child td {
border-bottom: 0;
}
.eval-table th {
background: var(--panel);
font-size: var(--t-1);
}
.eval-table td:first-child {
min-width: 220px;
font-weight: 700;
}
.eval-table small {
display: block;
color: var(--ink-3);
font-size: 12px;
font-weight: 400;
}
.eval-verdict {
font-weight: 700;
}
.eval-foot {
display: grid;
grid-template-columns: minmax(0, 3fr) minmax(0, 2fr);
gap: var(--s7);
margin-top: var(--s6);
}
.eval-foot p {
margin-top: var(--s2);
color: var(--ink-2);
font-size: 14px;
}
.eval-notes {
padding-left: var(--s5);
border-left: 1px solid var(--line-strong);
}
.eval-notes p {
margin: 0 0 var(--s3);
}
@media (max-width: 680px) {
.eval-decision,
.eval-foot {
grid-template-columns: minmax(0, 1fr);
gap: var(--s3);
}
.eval-notes {
padding: var(--s4) 0 0;
border-top: 1px solid var(--line-strong);
border-left: 0;
}
}

/* ---- faq and footer ---- */
.faq {
grid-template-columns: minmax(0, 1fr);
Expand Down
Loading