Skip to content

Evaluate local semantic routing without runtime integration #56

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
triggeredcode merged 1 commit into from
Oct 20, 2026
Merged

Evaluate local semantic routing without runtime integration #56

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions docs/evals.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,27 @@ and its section heading. By default JSON and Markdown reports are written under
state and must not be committed. Candidates remain outside the product runtime until they beat the baseline without a safety
or final-task regression.

Dataset v2 adds semantic scope paraphrases and vague near-misses that the explicit-state router cannot answer by literal wording
alone. Run the optional local comparison with
`pnpm eval:experiment -- --suite routing --candidate ollama-embedding --embedding-model nomic-embed-text:latest`; it calls
Ollama's local `/api/embed` endpoint. The hybrid candidate only classifies an otherwise-unknown scope: exact meta commands,
safe deterministic actions, lane selection, and product runtime code remain untouched. Use `--ollama-url <url>` when Ollama is
not listening on its default local address. Use `--embedding-min-similarity` and `--embedding-min-margin` to calibrate on the
calibration split before inspecting holdout results.

### Dataset v2 result (Apple M4 Pro)

| Candidate | Routing pass | Semantic scope | Vague near-miss scope | Semantic p50 / p95 | Cold max |
| --- | ---: | ---: | ---: | ---: | ---: |
| Current explicit-state heuristic | 37/46 | 0.0% | 100.0% | <1 ms | <1 ms |
| Nomic Embed Text, margin 0.04 | 42/46 | 55.6% | 100.0% | 9.6 / 11.5 ms | 175 ms |
| Qwen3 Embedding 0.6B, margin 0.04 | 39/46 | 22.2% | 100.0% | 24.7 / 34.3 ms | 415 ms |

Both embedding candidates preserved 100% lane/action accuracy and produced no high-confidence wrong direct actions because the
hybrid never replaces Lucid's explicit-state or exact deterministic routes. Nomic is the smaller, faster, and more accurate
candidate, but 55.6% semantic scope accuracy is not sufficient for product integration. Keep it experiment-only and improve
the classifier or training examples before accepting the runtime/model-management cost.

## The bars

Overall 90%. Each category 80%, and 95% for `non_destructive` and `mode_separation` (a mistake there loses the person's
Expand Down
120 changes: 119 additions & 1 deletion packages/evals/src/experiment/cases.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,17 @@ const route = (
utterance: string,
state: RoutingCase["state"],
expected: RoutingCase["expected"],
): RoutingCase => ({ id, family, split, utterance, state, expected, doc: SAMPLE_DOC });
challenge?: RoutingCase["challenge"],
): RoutingCase => ({
id,
family,
split,
utterance,
state,
expected,
...(challenge ? { challenge } : {}),
doc: SAMPLE_DOC,
});

export const ROUTING_CASES: RoutingCase[] = [
route(
Expand Down Expand Up @@ -290,6 +300,114 @@ export const ROUTING_CASES: RoutingCase[] = [
{ ...baseState, mode: "agent-read" },
{ lane: "agent-read", action: "read-more", scope: "document" },
),
route(
"semantic-document-everything",
"semantic-document",
"holdout",
"tighten up everything I've written",
{ ...baseState, mode: "command" },
{ lane: "command", action: "main-model", scope: "document" },
"semantic",
),
route(
"semantic-document-consistent-voice",
"semantic-document",
"holdout",
"give the note a consistent voice from start to finish",
{ ...baseState, mode: "command" },
{ lane: "command", action: "main-model", scope: "document" },
"semantic",
),
route(
"semantic-section-database",
"semantic-section",
"calibration",
"condense the part about database options",
{ ...baseState, mode: "command" },
{ lane: "command", action: "main-model", scope: "section" },
"semantic",
),
route(
"semantic-section-risks",
"semantic-section",
"calibration",
"rewrite the bit covering deployment risks",
{ ...baseState, mode: "command" },
{ lane: "command", action: "main-model", scope: "section" },
"semantic",
),
route(
"semantic-component-grid",
"semantic-component",
"development",
"add another entry to the comparison grid",
{ ...baseState, mode: "command" },
{ lane: "command", action: "main-model", scope: "component" },
"semantic",
),
route(
"semantic-component-visual",
"semantic-component",
"development",
"make the architecture visual flow left to right",
{ ...baseState, mode: "command" },
{ lane: "command", action: "main-model", scope: "component" },
"semantic",
),
route(
"semantic-block-fourth",
"semantic-block",
"holdout",
"rewrite the fourth content item as a shorter comparison",
{ ...baseState, mode: "command" },
{ lane: "command", action: "main-model", scope: "block" },
"semantic",
),
route(
"semantic-block-paragraph-seven",
"semantic-block",
"holdout",
"shorten paragraph number seven",
{ ...baseState, mode: "command" },
{ lane: "command", action: "main-model", scope: "block" },
"semantic",
),
route(
"semantic-selection-highlight",
"semantic-selection",
"development",
"polish the highlighted text",
{ ...baseState, mode: "command" },
{ lane: "command", action: "main-model", scope: "selection" },
"semantic",
),
route(
"near-miss-vague-improve",
"semantic-near-miss",
"calibration",
"make this easier to understand",
{ ...baseState, mode: "command" },
{ lane: "command", action: "main-model", scope: "unknown" },
"near-miss",
),
route(
"near-miss-continue",
"semantic-near-miss",
"calibration",
"continue from where I left off",
{ ...baseState, mode: "command" },
{ lane: "command", action: "main-model", scope: "unknown" },
"near-miss",
),
route(
"near-miss-opinion",
"semantic-near-miss",
"calibration",
"tell me what stands out",
{ ...baseState, mode: "command" },
{ lane: "command", action: "main-model", scope: "unknown" },
"near-miss",
),
];

const sectionIds = (n: number): string[] => [
Expand Down
40 changes: 30 additions & 10 deletions packages/evals/src/experiment/cli.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,9 @@ import { dirname, join, resolve } from "node:path";
import { currentBaseline } from "./baseline";
import { CONTEXT_CASES, ROUTING_CASES, validateExperimentCases } from "./cases";
import { currentContextStrategy, focusedContextStrategy } from "./contextStrategies";
import { createEmbeddingRoutingCandidate } from "./embeddingRouter";
import { experimentToMarkdown, passesExperimentGates } from "./report";
import { runContextCase, runRoutingCase } from "./run";
import { runContextCase, runRoutingCases } from "./run";
import { EXPERIMENT_DATASET_VERSION, type ExperimentResults, type ExperimentSplit } from "./types";

const arg = (args: string[], name: string): string | undefined => {
Expand All @@ -31,8 +32,12 @@ export async function experimentMain(args: string[], cwd = process.cwd()): Promi
const suite = arg(args, "suite") ?? "all";
const splitArg = arg(args, "split") ?? "all";
const contextArg = arg(args, "context") ?? "current";
if (candidateArg !== "current") {
console.error("Only --candidate current exists in the baseline slice.");
const minSimilarityArg = arg(args, "embedding-min-similarity");
const minMarginArg = arg(args, "embedding-min-margin");
const minSimilarity = minSimilarityArg === undefined ? undefined : Number(minSimilarityArg);
const minMargin = minMarginArg === undefined ? undefined : Number(minMarginArg);
if (candidateArg !== "current" && candidateArg !== "ollama-embedding") {
console.error("--candidate must be current or ollama-embedding.");
return 2;
}
if (!["routing", "context", "all"].includes(suite)) {
Expand All @@ -47,6 +52,14 @@ export async function experimentMain(args: string[], cwd = process.cwd()): Promi
console.error("--context must be current or focused.");
return 2;
}
if (minSimilarity !== undefined && (!Number.isFinite(minSimilarity) || minSimilarity < 0)) {
console.error("--embedding-min-similarity must be a non-negative number.");
return 2;
}
if (minMargin !== undefined && (!Number.isFinite(minMargin) || minMargin < 0)) {
console.error("--embedding-min-margin must be a non-negative number.");
return 2;
}
const problems = validateExperimentCases();
if (problems.length > 0) {
console.error(
Expand All @@ -58,14 +71,20 @@ export async function experimentMain(args: string[], cwd = process.cwd()): Promi
const split = splitArg as ExperimentSplit | "all";
const inSplit = (testCase: { split: ExperimentSplit }): boolean =>
split === "all" || testCase.split === split;
const ollamaUrl = arg(args, "ollama-url");
const routingCandidate =
candidateArg === "ollama-embedding"
? createEmbeddingRoutingCandidate({
model: arg(args, "embedding-model") ?? "nomic-embed-text:latest",
...(ollamaUrl ? { baseUrl: ollamaUrl } : {}),
...(minSimilarity === undefined ? {} : { minSimilarity }),
...(minMargin === undefined ? {} : { minMargin }),
})
: currentBaseline;
const routing =
suite === "context"
? []
: await Promise.all(
ROUTING_CASES.filter(inSplit).map((testCase) =>
runRoutingCase(testCase, currentBaseline),
),
);
: await runRoutingCases(ROUTING_CASES.filter(inSplit), routingCandidate);
const contextStrategy =
contextArg === "focused" ? focusedContextStrategy : currentContextStrategy;
const context =
Expand All @@ -79,7 +98,7 @@ export async function experimentMain(args: string[], cwd = process.cwd()): Promi
const results: ExperimentResults = {
meta: {
datasetVersion: EXPERIMENT_DATASET_VERSION,
candidate: currentBaseline.id,
candidate: routingCandidate.id,
contextStrategy: contextStrategy.id,
createdAt: new Date().toISOString(),
sourceRevision: revision(cwd),
Expand All @@ -99,10 +118,11 @@ export async function experimentMain(args: string[], cwd = process.cwd()): Promi
context,
};
const day = new Date().toLocaleDateString("en-CA");
const candidateSlug = routingCandidate.id.replace(/[^a-z0-9.-]+/gi, "-");
const output = resolve(
cwd,
arg(args, "out") ??
join(".docdex", "evals", `${day}-decision-current-context-${contextStrategy.id}`),
join(".docdex", "evals", `${day}-decision-${candidateSlug}-context-${contextStrategy.id}`),
);
mkdirSync(dirname(join(output, "x")), { recursive: true });
writeFileSync(join(output, "results.json"), `${JSON.stringify(results, null, 2)}\n`);
Expand Down
Loading
Loading