lbruton · Oct 31, 2026 · Oct 30, 2026 · Oct 30, 2026 · Oct 30, 2026 · Oct 30, 2026
diff --git a/AGENTS.md b/AGENTS.md
@@ -42,6 +42,7 @@
 - **Hosted embeddings deferred** — SESF-6 keeps embedding fully local (MLX). No hosted/OpenAI setup steps, credentials, or collections exist. Future hosted path would require a separate identity/collection to avoid vector mixing.
 - **Optional LaunchAgent for multi-harness startup** — `./sessionflow-server.sh install-agent` writes `~/Library/LaunchAgents/cc.lbruton.sessionflow.plist` so the server starts at login before any harness hook races. OPTIONAL — `start/stop/status/restart` behavior is unchanged when not installed. `agent-status` / `uninstall-agent` manage it. `setup.sh` honors `SESSIONFLOW_INSTALL_AGENT=1` for non-interactive installs.
 - **Optional hourly backfill LaunchAgent (SESF-7)** — `./sessionflow-server.sh install-backfill-agent` writes `cc.lbruton.sessionflow-backfill.plist` with a `StartInterval` (defaults to 3600s; override via `SESSIONFLOW_BACKFILL_INTERVAL_SECONDS`). It runs `python cleanup.py backfill enqueue --provider claude_code_cli --mode incremental`, decoupling catch-up indexing from MCP session activity. MCP-client SessionStart hooks should connect to the already-running server rather than racing to start it.
+- **Retrieval eval (SESF-82)** — `eval_retrieval.py` talks to the running server over MCP and never embeds. The real query set and reports live in the private companion (`DocVault/Projects/SessionFlow/eval/` in Devops), never in this public repo; `tests/eval/fixture_queries.jsonl` is synthetic. `run --env KEY=VALUE` refuses when `/health` `ranking` disagrees, so ranking env must reach the server process. `sweep --yes` does that with `launchctl setenv` and `kickstart -k`, then unsets and restarts. A plist `EnvironmentVariables` entry overrides `setenv`. `freshness` writes a `fact` with `topic_key` `eval:freshness:<host>` and deletes it; probes on one host take turns on `~/.sessionflow/eval-freshness.lock`. `SESSIONFLOW_SEARCH_LOG=1` (off by default) logs searches and follow-ups to `~/.sessionflow/search-log.jsonl` (`search_log.py`). `install-eval-agent` needs `SESSIONFLOW_EVAL_QUERIES` and `SESSIONFLOW_EVAL_OUT`.
 - **Schema drift (SESF-11)** — `_ensure_collection` refuses to start when the live Milvus schema differs from `_expected_schema_fields()`. Drop+recreate is opt-in via `SESSIONFLOW_AUTO_MIGRATE_SCHEMA=1`, or run `python cleanup.py migrate-schema` explicitly (destructive).
 - **`SESSIONFLOW_BACKFILL_MAX_TURNS_PER_RUN` (SESF-12)** — default raised from 200 to 100,000 so a long-running server isn't silently capped after a few minutes. The budget logs a single WARN the first time the cap is hit so operators don't have to grep for "0 turns" lines.
 - **FTS5 thread affinity (SESF-13)** — `FTSIndex` keeps per-thread persistent connections (`threading.local`). Server-mode connections opened on the embed executor and request threads are isolated, and cross-thread `close_all()` is a no-op rather than a noisy WARN.

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -9,6 +9,15 @@ have no git tags or GitHub Releases. For changes before the fork, see [UPSTREAM-
 
 ## [Unreleased]
 
+## [0.9.0] - 2026-10-30
+
+### Added
+
+- `eval_retrieval.py` scores a labelled JSONL query set against the running server: recall@5, MRR and nDCG@10 per query group, plus a current-first rate for superseded decisions and the top hit score for no-answer queries. `run`, `sweep` (restarts the LaunchAgent once per env value; needs `--yes`), `freshness` (a canary memory, deleted after the probe), `usage` and `nightly` subcommands (SESF-82).
+- `SESSIONFLOW_SEARCH_LOG=1` appends each search, `get_memory` and `get_turns` call to `~/.sessionflow/search-log.jsonl` with redacted queries and result IDs in rank order. It is off by default (SESF-82).
+- `/health` reports the hybrid ranking settings under `ranking` (SESF-82).
+- `sessionflow-server.sh install-eval-agent`, `uninstall-eval-agent` and `eval-agent-status` manage a nightly LaunchAgent that runs `eval_retrieval.py nightly` (SESF-82).
+
 ## [0.8.1] - 2026-10-30
 
 ### Fixed

diff --git a/CLAUDE.md b/CLAUDE.md
@@ -42,6 +42,7 @@ Semantic search over Claude Code session transcripts. Independent project, origi
 - **Hosted embeddings deferred** — SESF-6 keeps embedding fully local (MLX). No hosted/OpenAI setup steps, credentials, or collections exist. Future hosted path would require a separate identity/collection to avoid vector mixing.
 - **Optional LaunchAgent for multi-harness startup** — `./sessionflow-server.sh install-agent` writes `~/Library/LaunchAgents/cc.lbruton.sessionflow.plist` so the server starts at login before any harness hook races. OPTIONAL — `start/stop/status/restart` behavior is unchanged when not installed. `agent-status` / `uninstall-agent` manage it. `setup.sh` honors `SESSIONFLOW_INSTALL_AGENT=1` for non-interactive installs.
 - **Optional hourly backfill LaunchAgent (SESF-7)** — `./sessionflow-server.sh install-backfill-agent` writes `cc.lbruton.sessionflow-backfill.plist` with a `StartInterval` (defaults to 3600s; override via `SESSIONFLOW_BACKFILL_INTERVAL_SECONDS`). It runs `python cleanup.py backfill enqueue --provider claude_code_cli --mode incremental`, decoupling catch-up indexing from MCP session activity. MCP-client SessionStart hooks should connect to the already-running server rather than racing to start it.
+- **Retrieval eval (SESF-82)** — `eval_retrieval.py` talks to the running server over MCP and never embeds. The real query set and reports live in the private companion (`DocVault/Projects/SessionFlow/eval/` in Devops), never in this public repo; `tests/eval/fixture_queries.jsonl` is synthetic. `run --env KEY=VALUE` refuses when `/health` `ranking` disagrees, so ranking env must reach the server process. `sweep --yes` does that with `launchctl setenv` and `kickstart -k`, then unsets and restarts. A plist `EnvironmentVariables` entry overrides `setenv`. `freshness` writes a `fact` with `topic_key` `eval:freshness:<host>` and deletes it; probes on one host take turns on `~/.sessionflow/eval-freshness.lock`. `SESSIONFLOW_SEARCH_LOG=1` (off by default) logs searches and follow-ups to `~/.sessionflow/search-log.jsonl` (`search_log.py`). `install-eval-agent` needs `SESSIONFLOW_EVAL_QUERIES` and `SESSIONFLOW_EVAL_OUT`.
 - **Schema drift (SESF-11)** — `_ensure_collection` refuses to start when the live Milvus schema differs from `_expected_schema_fields()`. Drop+recreate is opt-in via `SESSIONFLOW_AUTO_MIGRATE_SCHEMA=1`, or run `python cleanup.py migrate-schema` explicitly (destructive).
 - **`SESSIONFLOW_BACKFILL_MAX_TURNS_PER_RUN` (SESF-12)** — default raised from 200 to 100,000 so a long-running server isn't silently capped after a few minutes. The budget logs a single WARN the first time the cap is hit so operators don't have to grep for "0 turns" lines.
 - **FTS5 thread affinity (SESF-13)** — `FTSIndex` keeps per-thread persistent connections (`threading.local`). Server-mode connections opened on the embed executor and request threads are isolated, and cross-thread `close_all()` is a no-op rather than a noisy WARN.

diff --git a/README.md b/README.md
@@ -68,6 +68,7 @@ Restart Claude Code after setup to load the MCP server.
 | `SESSIONFLOW_EXPIRE_DAYS` | `365` | Prune turns older than this many days. |
 | `SESSIONFLOW_WATCH` | `true` | Turn the transcript file watcher on or off. |
 | `SESSIONFLOW_URL` | `http://127.0.0.1:7102` | Server URL that the hooks call. |
+| `SESSIONFLOW_SEARCH_LOG` | off | Set to `1` to log each search and follow-up call to `~/.sessionflow/search-log.jsonl`. Queries are redacted. |
 
 ## Running
 
@@ -93,6 +94,19 @@ Several harnesses can start at the same time, and their `SessionStart` hooks the
 
 The `start`, `stop`, `restart` and `status` commands work the same with or without the LaunchAgent. `setup.sh` offers to install it. Set `SESSIONFLOW_INSTALL_AGENT=1` to install it without a prompt.
 
+### Retrieval eval
+
+`eval_retrieval.py` scores a labelled query set against the running server. Each JSONL line names an `id`, a `group`, a `tool`, a `query`, optional `args` and the `expected` IDs (`memory_id` or `session_id:turn_index`). `tests/eval/fixture_queries.jsonl` shows the format.
+
+```bash
+./venv/bin/python eval_retrieval.py run --queries queries.jsonl --sort-by hybrid --sort-by relevance --out reports/
+./venv/bin/python eval_retrieval.py sweep --queries queries.jsonl --var SESSIONFLOW_RECENCY_WEIGHT --values 0,0.15,0.3,0.5 --out reports/ --yes
+./venv/bin/python eval_retrieval.py freshness     # Time a canary memory until search finds it
+./venv/bin/python eval_retrieval.py usage --days 7 # Roll up the search usage log
+```
+
+`sweep` restarts the LaunchAgent once per value. `SESSIONFLOW_EVAL_QUERIES=<file> SESSIONFLOW_EVAL_OUT=<folder> ./sessionflow-server.sh install-eval-agent` runs `nightly` every day at 03:30.
+
 ## Memories
 
 SessionFlow also stores memories: notes an agent writes on purpose, such as session summaries, decisions and handoffs. Postgres holds them verbatim. The Milvus `memories` collection and a local `memories_fts.db` file are search indexes that SessionFlow rebuilds from Postgres. Memory is optional. Without `SESSIONFLOW_MEMORY_DSN`, the server serves transcripts only and `/health` reports memory as not configured.