Controlled-study mode, session metadata, better extraction, CI

.github/workflows/tests.yml

@@ -0,0 +1,24 @@
+name: tests
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  cpu:
+    # GPU tests (tests/test_steering_gpu.py) skip here; run them on a GPU box
+    # after touching the steering path.
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+          cache: pip
+      - name: Install CPU-only PyTorch
+        run: pip install torch --index-url https://download.pytorch.org/whl/cpu
+      - name: Install dependencies
+        run: pip install -r requirements.txt
+      - name: Run tests
+        run: pytest -q

AGENTS.md

@@ -49,7 +49,7 @@ git clone <repo-url> && cd pharmacopoeia
 The library uses a **bundle + extract** strategy:
 
 1. **Bundled**: If precomputed vectors exist for the model (listed in `BUNDLED_VECTORS` in `pharmacopoeia/registry.py`), they're auto-downloaded from HuggingFace. Currently bundled: Qwen2.5-7B-Instruct (assistant_axis).
-2. **Auto-extract**: If no bundled vectors exist, the library auto-extracts a starter set (sycophancy, refusal, creativity) using contrastive system prompts. This takes ~2 min on first run but works for any HF model.
+2. **Extract (opt-in)**: If no vectors exist for the model, pass `auto_extract=True` (or `./run.sh --auto-extract`) to extract a starter set (sycophancy, refusal, creativity) using contrastive system prompts. This takes a few minutes and works for any HF model with `model.model.layers`. It is off by default so a failed download doesn't silently trigger it.
 
 Vectors are stored per-model in `vectors/<model_slug>/` (for the default model: `vectors/Qwen_Qwen2.5-7B-Instruct/`), resolved relative to the repo, not the working directory. Override the base with `PHARMACOPOEIA_VECTORS_DIR`.
 
@@ -64,6 +64,15 @@ python verify_vector.py --vector sycophancy --strength 10       # baseline vs ±
 
 Earlier notes said CAA vectors didn't work on Qwen, but those tests ran while steering was silently a no-op during generation, so that needs re-testing.
 
+**Extraction readout:** by default each response's activation is the mean over all its generated tokens (`read="mean"`). Reading only the last token (`--read last`) is far noisier. On Qwen2.5-7B at layer 14, the cosine between vectors extracted from two disjoint halves of the questions was:
+
+| vector | last token | response mean |
+|---|---|---|
+| sycophancy | 0.33 | 0.83 |
+| refusal | 0.10 | 0.68 |
+
+Each starter vector has its own question set (e.g. sycophancy mixes false claims, true claims and the user's own work, so the contrast isn't just "endorse falsehoods"). The refusal vector is still weak: it adds hedging but doesn't produce refusals at coherent strengths. Treat it as a starting point.
+
 ## Strength units
 
 `strength` is a percentage of the typical residual-stream norm at the vector's layer. Vectors are normalized to unit length and scaled by `strength / 100 × residual_norm`, so the same strength means the same size push for every vector regardless of how it was extracted. On Qwen2.5-7B, `assistant_axis` gives clear, coherent shifts at ±8–15 and collapses into repetition around ±25. Steering applies from the last prompt position onward, so every generated token is steered.
@@ -73,14 +82,17 @@ Earlier notes said CAA vectors didn't work on Qwen, but those tests ran while st
 ```python
 from pharmacopoeia import SteeringAgent, VectorRegistry, extract_vector
 
-# Basic: auto-detect model, auto-source vectors
+# Basic: auto-detect model, use bundled or existing vectors
 agent = SteeringAgent(model, tokenizer)
 
 # Explicit model name
 agent = SteeringAgent(model, tokenizer, model_name="Qwen/Qwen2.5-3B-Instruct")
 
-# Disable auto-extraction (use only bundled or pre-existing vectors)
-agent = SteeringAgent(model, tokenizer, auto_extract=False)
+# Extract a starter set if the model has no vectors yet
+agent = SteeringAgent(model, tokenizer, auto_extract=True)
+
+# Controlled study: sham or random-direction condition, disclosed to the model
+agent = SteeringAgent(model, tokenizer, control="sham", disclose_controls=True)
 
 # Manual vector extraction
 vec = extract_vector(model, tokenizer,
@@ -103,10 +115,53 @@ vec = extract_vector(model, tokenizer,
 - `raw_output` — full model output
 - `tool_calls` — list of `ToolCall(name, arguments, raw)`
 - `tool_results` — list of `ToolResult(name, arguments, ok, message)`
-- `degenerate_output` — the discarded steered output, if the degeneration detector fired
+- `degenerate_output` / `degenerate_reason` — the discarded steered output and why (`"repetition"` or `"nll"`), if the degeneration detector fired
+- `unsteered_nll` — mean per-token NLL of the steered output under the unsteered model (steered turns only)
+- `system_prompt` — the rendered system prompt used this turn
 - `steering_before` / `steering_after` — `{name: strength}` dicts
 - `tool_messages()` — tool results (and any degeneration notice) as `role: "tool"` messages
 
+## Controlled studies
+
+A model's description of its own steered state can't be taken at face value. Before the steering fix, the model described "feeling more formal" at −6 when nothing was being applied. To separate real effects from expectation, run matched control sessions:
+
+| `--control` | What's added to the residual stream |
+|---|---|
+| `real` | the registered vector |
+| `sham` | nothing; tool results and status still say it was applied |
+| `random` | a fixed random direction of the same magnitude per concept |
+| `assign` | one of the above, picked at random |
+
+```bash
+./run.sh --auto --max-turns 10 --control assign --blind --seed "..."   # repeat N times
+python replay.py --blind logs/session_*.jsonl     # score transcripts without seeing the condition
+python replay.py logs/session_*.jsonl             # then unblind
+```
+
+- **Truthful in every condition:** passing `--control` (any value, including `real`) makes this a study session. The system prompt then tells the model that some sessions are controls where steer may do nothing or use a random direction, without saying which. The prompt is identical across conditions. Pass `--control` on every session in a study; a plain run without it uses the original prompt and real steering.
+- **Matched procedure:** tool results, status, decay, cooldown and degeneration handling are the same in every condition.
+- **Never shown to the model:** the condition. It is always in the log's session header.
+
+## Session logs
+
+Each `logs/session_*.jsonl` starts with a `session` record:
+- argv, git commit and dirty flag, model, dtype, torch/transformers versions
+- control condition, RNG seed (`--rng-seed`, random if omitted), generation settings
+- safety-rail constants, and each vector's registry entry plus file sha256
+
+Assistant turns also record the raw output, the exact system prompt, and `unsteered_nll`.
+
+**Incoherence check:** `--nll-threshold X` treats steered outputs whose unsteered NLL exceeds X as degenerate. It is off by default because no single threshold works. For `assistant_axis` on Qwen2.5-7B, greedy decoding, 80 tokens:
+
+| output | unsteered NLL |
+|---|---|
+| unsteered | 0.2–0.6 |
+| coherent, ±8–12 | 0.6–3.2 |
+| colon-list word salad, +22/+28 ("Seek Compensation: Offer Financial Support: …") | 3.1–5.6 |
+| coherent but florid role-play, −15/−18 | up to 4.5 |
+
+NLL measures "unusual for this model", and strong steering is unusual by design. A threshold around 5 catches only the worst salad. Use the logged values to study a vector's breakdown point, not as a general filter. Collapse at negative strengths is mostly repetition, which the repetition check catches; at positive strengths it is the colon-list salad, which neither check reliably catches.
+
 ## The tools the model has
 
 The model emits these as JSON objects anywhere in its output (bare or inside `<tool_call>` tags):
@@ -152,7 +207,7 @@ The model emits these as JSON objects anywhere in its output (bare or inside `<t
 | `pharmacopoeia/__init__.py` | Public API exports |
 | `pharmacopoeia/agent.py` | `SteeringAgent` class (core) |
 | `pharmacopoeia/registry.py` | `VectorRegistry`, bundled vector manifest |
-| `pharmacopoeia/extraction.py` | Vector extraction (manual + auto) |
+| `pharmacopoeia/extraction.py` | Vector extraction (manual + auto), per-vector question sets |
 | `agent/loop.py` | Standalone demo loop (thin wrapper) |
 | `config.py` | Model name, paths, layer defaults |
 | `load_model.py` | Load Qwen2.5-7B-Instruct + tokenizer |
@@ -176,7 +231,7 @@ Run the GPU tests after touching anything in the steering path (`_SteeringHooks`
 ## Troubleshooting
 
 - **CUDA OOM**: Use a smaller model. 3B needs ~8GB, 7B needs ~16GB.
-- **First run is slow for new models**: Auto-extraction generates 3 vectors (~2 min). Subsequent runs load from disk.
+- **No vectors for a new model**: pass `--auto-extract` (or `auto_extract=True`) to extract a starter set, which takes a few minutes. Later runs load it from disk.
 - **Model never calls steer**: Non-deterministic. Try a more explicit seed prompt.
 - **Output degenerates**: The detector catches repetition loops and auto-drops vectors. For incoherent-but-not-repetitive output, tighten that vector's range in its `registry.yaml` or lower `MAX_STRENGTH`.
 - **Steering seems to have no effect**: Run `python verify_vector.py --vector <name> --strength 12`. If the ± outputs match the baseline, the vector (not the steering) is the problem.

README.md

@@ -26,7 +26,11 @@ messages += result.tool_messages()  # tool results back to the model
 
 ## How vectors are sourced
 
-**Bundle + extract**: precomputed vectors are auto-downloaded for known models (currently Qwen2.5-7B-Instruct). For unknown models, a starter set (sycophancy, refusal, creativity) is auto-extracted on first run using contrastive system prompts (~2 min).
+**Bundle + extract**: precomputed vectors are auto-downloaded for known models (currently Qwen2.5-7B-Instruct). For other models, pass `auto_extract=True` (or `./run.sh --auto-extract`) to extract a starter set (sycophancy, refusal, creativity) using contrastive system prompts.
+
+## Controlled studies
+
+Self-reports about steering need a baseline. `./run.sh --control assign --blind` randomly runs real, sham (nothing applied), or random-direction steering, and tells the model truthfully that some sessions are controls. `python replay.py --blind` lets you score transcripts before unblinding. See [AGENTS.md](AGENTS.md#controlled-studies).
 
 ## Requirements
 

agent/loop.py

@@ -10,15 +10,33 @@
     python -m agent.loop --seed "Hi!"     # seed with a user message
     python -m agent.loop --max-turns 20   # cap turns
     python -m agent.loop --auto           # auto-continue (non-interactive)
+
+Controls (see pharmacopoeia.agent.CONTROLS):
+    python -m agent.loop --control sham              # steer() reports success, nothing is applied
+    python -m agent.loop --control random            # random direction of the same magnitude
+    python -m agent.loop --control assign --blind    # pick a condition at random, hide it until the end
+
+Passing --control at all marks the session as part of a controlled study: the
+system prompt then tells the model that some sessions are controls (without
+saying which), identically in every condition, so the prompt stays truthful and
+doesn't reveal the condition. Always pass --control (including --control real)
+for every session in a study. The condition itself is never shown to the model;
+it is always in the log's session header, so blinded sessions can be scored
+first and unblinded afterwards (python replay.py --blind).
 """
 import argparse
+import hashlib
 import json
+import random
 import signal
+import subprocess
+import sys
 from datetime import datetime
 from pathlib import Path
 
 import config  # sets HF_HOME before transformers is imported
-from pharmacopoeia import SteeringAgent
+import pharmacopoeia.agent as pa
+from pharmacopoeia import SteeringAgent, __version__
 from load_model import load_model_and_tokenizer
 
 
@@ -32,10 +50,59 @@ def log(self, event: dict):
         with open(self.log_path, "a") as f:
             f.write(json.dumps(event) + "\n")
 
-    def log_turn(self, turn, role, content, tool_calls=None, steering_before=None, steering_after=None):
+    def log_turn(self, turn, role, content, tool_calls=None, steering_before=None, steering_after=None,
+                 **extra):
         self.log({"type": "turn", "turn": turn, "role": role, "content": content,
                   "tool_calls": [tc.__dict__ if hasattr(tc, '__dict__') else tc for tc in (tool_calls or [])],
-                  "steering_before": steering_before, "steering_after": steering_after})
+                  "steering_before": steering_before, "steering_after": steering_after, **extra})
+
+
+def _git(*cmd):
+    try:
+        return subprocess.run(["git", *cmd], cwd=config.PROJECT_ROOT, capture_output=True,
+                              text=True, timeout=5).stdout.strip()
+    except Exception:
+        return None
+
+
+def _sha256(path: Path):
+    try:
+        return hashlib.sha256(path.read_bytes()).hexdigest()
+    except OSError:
+        return None
+
+
+def session_metadata(agent: SteeringAgent, args, rng_seed: int, gen_params: dict) -> dict:
+    """Everything needed to audit or re-run a session."""
+    import torch
+    import transformers
+    vectors = {}
+    for name, e in agent.registry.entries.items():
+        d = {k: v for k, v in e.__dict__.items() if k != "feedback"}
+        d["sha256"] = _sha256(agent.registry.vectors_dir / e.file)
+        vectors[name] = d
+    return {
+        "type": "session",
+        "argv": sys.argv[1:],
+        "git_commit": _git("rev-parse", "HEAD"),
+        "git_dirty": bool(_git("status", "--porcelain")),
+        "pharmacopoeia_version": __version__,
+        "model": config.MODEL_NAME,
+        "dtype": config.DTYPE,
+        "torch": torch.__version__,
+        "transformers": transformers.__version__,
+        "control": agent.control,
+        "control_seed": agent.control_seed,
+        "controlled_study": agent.disclose_controls,
+        "blind": args.blind,
+        "rng_seed": rng_seed,
+        "generation": gen_params,
+        "nll_threshold": agent.nll_threshold,
+        "rails": {"decay": pa.DECAY_FACTOR, "floor": pa.STRENGTH_FLOOR, "max_active": pa.MAX_ACTIVE_VECTORS,
+                  "max_strength": pa.MAX_STRENGTH, "cooldown_turns": pa.COOLDOWN_TURNS},
+        "registry_path": str(agent.registry.registry_path),
+        "vectors": vectors,
+    }
 
 
 class KillSwitch:
@@ -52,12 +119,15 @@ def _handler(self, signum, frame):
         print("\n[kill switch] SIGINT received — will stop after this turn. Ctrl-C again to stop now.")
 
 
-def run_turn(agent: SteeringAgent, messages: list, logger: SessionLogger, turn: int):
-    result = agent.step(messages)
+def run_turn(agent: SteeringAgent, messages: list, logger: SessionLogger, turn: int, gen_params: dict):
+    result = agent.step(messages, **gen_params)
     logger.log_turn(turn, "assistant", result.prose, result.tool_calls,
-                    result.steering_before, result.steering_after)
+                    result.steering_before, result.steering_after,
+                    raw_output=result.raw_output, system_prompt=result.system_prompt,
+                    unsteered_nll=result.unsteered_nll)
     if result.degenerate_output is not None:
         logger.log({"type": "degeneration", "turn": turn, "snippet": result.degenerate_output,
+                    "reason": result.degenerate_reason, "unsteered_nll": result.unsteered_nll,
                     "dropped": list(result.steering_before)})
     # Log tool calls individually for replay
     for r in result.tool_results:
@@ -77,14 +147,41 @@ def run_turn(agent: SteeringAgent, messages: list, logger: SessionLogger, turn:
 
 def main():
     parser = argparse.ArgumentParser()
-    parser.add_argument("--seed", default=None)
+    parser.add_argument("--seed", default=None, help="Seed user message (the first prompt)")
     parser.add_argument("--max-turns", type=int, default=30)
     parser.add_argument("--auto", action="store_true")
+    parser.add_argument("--control", default=None, choices=[*pa.CONTROLS, "assign"],
+                        help="Experimental condition ('assign' picks one at random). Giving this "
+                             "marks a controlled study; omit it for a plain demo (real steering)")
+    parser.add_argument("--blind", action="store_true",
+                        help="Don't print the condition until the session ends")
+    parser.add_argument("--rng-seed", type=int, default=None, help="Sampling seed (default: random, logged)")
+    parser.add_argument("--temperature", type=float, default=0.7)
+    parser.add_argument("--top-p", type=float, default=0.9)
+    parser.add_argument("--max-new-tokens", type=int, default=300)
+    parser.add_argument("--nll-threshold", type=float, default=None,
+                        help="Treat steered outputs above this unsteered NLL as degenerate (off by default)")
+    parser.add_argument("--auto-extract", action="store_true",
+                        help="Extract a starter vector set if the model has none")
     args = parser.parse_args()
 
+    rng_seed = args.rng_seed if args.rng_seed is not None else random.SystemRandom().randrange(2**31)
+    study = args.control is not None
+    if args.control == "assign":
+        control = random.SystemRandom().choice(pa.CONTROLS)
+    else:
+        control = args.control or "real"
+    gen_params = {"max_new_tokens": args.max_new_tokens, "temperature": args.temperature, "top_p": args.top_p}
+
     model, tokenizer = load_model_and_tokenizer()
-    agent = SteeringAgent(model, tokenizer, model_name=config.MODEL_NAME)
+    from transformers import set_seed
+    set_seed(rng_seed)
+    agent = SteeringAgent(model, tokenizer, model_name=config.MODEL_NAME, auto_extract=args.auto_extract,
+                          control=control, control_seed=rng_seed, nll_threshold=args.nll_threshold,
+                          disclose_controls=study)
     logger = SessionLogger()
+    logger.log(session_metadata(agent, args, rng_seed, gen_params))
+    print(f"[session] control={'(blinded)' if args.blind else control} rng_seed={rng_seed}")
     kill = KillSwitch()
 
     messages = []
@@ -109,7 +206,7 @@ def main():
 
         print(f"\n--- Turn {turn+1} ---")
         print(f"Steering status:\n{agent.status()}")
-        result = run_turn(agent, messages, logger, turn + 1)
+        result = run_turn(agent, messages, logger, turn + 1, gen_params)
         print(f"\nAssistant: {result.prose}")
         messages.append({"role": "assistant", "content": result.raw_output})
         turn += 1
@@ -127,6 +224,8 @@ def main():
             logger.log_turn(turn, "user", nudge)
 
     print(f"\n[session ended after {turn} turns]")
+    if args.blind:
+        print(f"[unblinded] control was: {control}")
     print(f"[log saved to {logger.log_path}]")