Community vector index: automatic checks, GPU verification and a submission form

.github/ISSUE_TEMPLATE/vector-submission.yml

@@ -0,0 +1,134 @@
+name: Submit a steering vector
+description: Tell us about a steering vector you've made or found. No git needed. We'll turn it into an index entry.
+title: "[Vector] "
+labels: ["vector-submission"]
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Thanks for contributing! Fill in what you know. Only the fields marked required are needed, and partial information is still useful.
+
+        Modulo lets a model steer its own activations. Your vector stays hosted where it is (e.g. your HuggingFace repo). We store a pointer pinned by its sha256 and credit you in the index. New entries start as **community**. After a maintainer runs a GPU check and reviews the outputs, an entry can become **verified**. See [CONTRIBUTING.md](https://github.com/EigenformAI/modulo/blob/main/CONTRIBUTING.md) for details.
+
+  - type: input
+    id: concept
+    attributes:
+      label: Concept name
+      description: Short name the model will use to call it, lowercase with underscores.
+      placeholder: e.g. honesty, formality, joy
+    validations:
+      required: true
+
+  - type: input
+    id: model
+    attributes:
+      label: Model
+      description: HuggingFace model id the vector was extracted for (and a revision, if you know it).
+      placeholder: e.g. meta-llama/Llama-3.1-8B-Instruct
+    validations:
+      required: true
+
+  - type: input
+    id: location
+    attributes:
+      label: Where the vector file is hosted
+      description: Link to the file (a HuggingFace repo is best). .safetensors preferred; .pt and .npz also accepted.
+      placeholder: https://huggingface.co/<user>/<repo>/blob/main/honesty.safetensors
+    validations:
+      required: true
+
+  - type: input
+    id: layer
+    attributes:
+      label: Layer(s)
+      description: Which decoder layer the vector is for. If the file holds one vector per layer, say so and give the recommended layer.
+      placeholder: e.g. 14 (file is [num_layers, hidden]; 14 recommended)
+    validations:
+      required: true
+
+  - type: textarea
+    id: effect
+    attributes:
+      label: What it does
+      description: What adding the vector does (+), and what subtracting it does (-).
+      placeholder: |
+        + more candid; admits uncertainty
+        - more evasive; overclaims
+    validations:
+      required: true
+
+  - type: textarea
+    id: method
+    attributes:
+      label: How it was extracted
+      description: e.g. mean difference of activations over contrast pairs, CAA, PCA, SAE feature, probe direction.
+    validations:
+      required: true
+
+  - type: input
+    id: license
+    attributes:
+      label: License
+      placeholder: e.g. MIT, Apache-2.0, CC-BY-4.0
+    validations:
+      required: true
+
+  - type: input
+    id: credit
+    attributes:
+      label: How you'd like to be credited
+      description: Names, handles or affiliations to list as contributors.
+    validations:
+      required: true
+
+  - type: textarea
+    id: original_setup
+    attributes:
+      label: Strength and setup you used (strongly encouraged)
+      description: |
+        Modulo measures strength as a % of the residual-stream norm at the layer, after normalizing the vector, so your numbers won't transfer directly. Tell us what you used and we'll calibrate: the coefficient, whether the vector was normalized, which positions were steered (all tokens / generated only), and which strengths stayed coherent.
+      placeholder: e.g. unnormalized vector, coefficient 4.0, all positions, coherent up to ~6
+
+  - type: textarea
+    id: failure_modes
+    attributes:
+      label: Known failure modes (strongly encouraged)
+      placeholder: e.g. collapses into repetition above 8; also makes the model more verbose
+
+  - type: input
+    id: paper
+    attributes:
+      label: Paper or write-up
+      placeholder: https://arxiv.org/abs/...
+
+  - type: input
+    id: contrast_dataset
+    attributes:
+      label: Contrast dataset or prompts
+      description: Link to the data the vector was extracted from, if public.
+
+  - type: textarea
+    id: evaluation
+    attributes:
+      label: Evaluation results
+      description: Any measurements of the effect, e.g. benchmark deltas or behavior rates.
+
+  - type: textarea
+    id: test_prompts
+    attributes:
+      label: Test prompts (up to 5)
+      description: Prompts where the effect should be easy to see, one per line. Used in our GPU check.
+
+  - type: textarea
+    id: notes
+    attributes:
+      label: Anything else
+      description: Other layers you tried, related vectors, caveats, how to reach you.
+
+  - type: checkboxes
+    id: confirm
+    attributes:
+      label: Confirmation
+      options:
+        - label: I have the right to share this vector under the license above.
+          required: true

.github/workflows/community-check.yml

@@ -0,0 +1,44 @@
+name: Community vector check
+
+on:
+  pull_request:
+    paths:
+      - "community/**"
+
+# Read-only token and no secrets, so this is safe to run on PRs from forks.
+permissions:
+  contents: read
+
+jobs:
+  check:
+    runs-on: ubuntu-latest
+    steps:
+      # The checker and library come from the base branch so a PR can't change
+      # the checks that judge it; only the PR's index is read.
+      - uses: actions/checkout@v4
+        with:
+          ref: ${{ github.event.pull_request.base.sha }}
+          path: base
+          persist-credentials: false
+      - uses: actions/checkout@v4
+        with:
+          ref: ${{ github.event.pull_request.head.sha }}
+          path: pr
+          persist-credentials: false
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+      - name: Install dependencies
+        run: |
+          pip install --index-url https://download.pytorch.org/whl/cpu torch
+          pip install pyyaml numpy huggingface_hub safetensors
+      - name: Check index
+        run: |
+          # Until the checker exists on the base branch (the PR adding it), use the PR's copy
+          checker=base/community/check_vectors.py
+          [ -f "$checker" ] || checker=pr/community/check_vectors.py
+          python "$checker" --index pr/community/vectors.yaml \
+            --base-index base/community/vectors.yaml --summary summary.md
+      - name: Summary
+        if: always()
+        run: '[ -f summary.md ] && cat summary.md >> "$GITHUB_STEP_SUMMARY" || true'

AGENTS.md

@@ -166,6 +166,11 @@ The format is defined in `modulo/reports.py`. `validate_report(d)` returns a lis
 | `modulo/registry.py` | `VectorRegistry`, bundled vector manifest |
 | `modulo/extraction.py` | Vector extraction (manual + auto) |
 | `modulo/reports.py` | Report format, validation, `reports.jsonl` read/write |
+| `modulo/community.py` | Community vector index: entry format and file checks |
+| `community/vectors.yaml` | Community vector index (pointers to hosted vectors, with trust levels) |
+| `community/check_vectors.py` | Checks the index; run by CI on PRs touching `community/` |
+| `community/gpu_verify.py` | Maintainer GPU check: strength sweep + suggested safe range |
+| `CONTRIBUTING.md` | How to contribute vectors (issue form or PR) |
 | `agent/loop.py` | Standalone demo loop (thin wrapper) |
 | `config.py` | Model name, paths, layer defaults |
 | `load_model.py` | Load Qwen2.5-7B-Instruct + tokenizer |

CONTRIBUTING.md

@@ -0,0 +1,70 @@
+# Contributing steering vectors
+
+Modulo lets local models steer their own activations, and we'd love more vectors for more models. If you've extracted a steering vector, or know of one, please share it: anything you know about it helps, even if it's partial.
+
+## Two ways to contribute
+
+1. **Fill in the form (no git needed).** [Open a vector submission](https://github.com/EigenformAI/modulo/issues/new?template=vector-submission.yml). A maintainer turns it into an index entry and credits you.
+2. **Open a pull request** adding an entry to [`community/vectors.yaml`](community/vectors.yaml). Automatic checks run on your PR.
+
+Your vector file stays where it's hosted (ideally your own HuggingFace repo). The index stores a pointer pinned by the file's sha256, so the file can't change underneath the entry.
+
+## Trust levels
+
+| Level | Who sets it | What it means |
+|---|---|---|
+| `community` | You (all new entries) | Passed the automatic checks. Opt-in. |
+| `verified` | Maintainers only | A maintainer ran it on the model, reviewed the outputs and set a safe strength range. Will be loaded by default. |
+
+The library doesn't read this index yet. Today only the vectors bundled in `modulo/registry.py` are downloaded automatically; loading the index (verified by default, community opt-in) is the next step.
+
+Entry text (`description`, `sign_convention`, `test_prompts`) ends up in a model's prompt once an entry is verified, so it's reviewed as part of promotion.
+
+## Strength units
+
+Modulo normalizes every vector to unit length and measures `strength` as a **percentage of the typical residual-stream norm at that layer**. Published coefficients use other conventions: raw multipliers, unnormalized vectors, steering all positions instead of generated tokens only. So you don't need to give a range in Modulo units. Describe what you used in `original_setup`; the GPU check sweeps strengths and suggests the Modulo range.
+
+## Index entry fields
+
+Required:
+
+| Field | Example | Notes |
+|---|---|---|
+| `id` | `llama3.1-8b-instruct-honesty` | Unique; lowercase letters, digits, `.` `-` `_` |
+| `name` | `honesty` | What the model calls it with `steer`; lowercase, `_` |
+| `model` | `meta-llama/Llama-3.1-8B-Instruct` | HF model id |
+| `repo_id` / `repo_type` | `you/llama-vectors` / `model` | Where the file lives (`model` or `dataset` repo) |
+| `filename` | `honesty.safetensors` | `.safetensors` preferred; `.pt` and `.npz` accepted |
+| `sha256` | `sha256sum honesty.safetensors` | 64 lowercase hex characters |
+| `format` | `single` | `single` (one vector) or `per_layer` (`[num_layers, hidden]`) |
+| `layer` | `14` | Decoder layer the vector is added at |
+| `description` | | What the vector represents |
+| `sign_convention` | `+ more candid; - more evasive` | What + and − do |
+| `extraction_method` | | How it was made (contrast pairs, CAA, PCA, SAE feature, …) |
+| `license` | `mit` | |
+| `contributors` | `[Your Name]` | Who to credit |
+| `trust` | `community` | Always `community` for new entries |
+
+Optional but very useful: `revision` (the HF commit of your repo; recommended), `model_revision`, `original_setup`, `known_failure_modes`, `paper`, `contrast_dataset`, `evaluation`, `test_prompts` (up to 5), `notes`. Maintainers set `safe_strength_min`/`safe_strength_max` when promoting.
+
+File formats: a `.safetensors` file must contain exactly one tensor; an `.npz` file stores it under the key `vector`; a `.pt` file is a plain tensor (loaded with `weights_only=True`). With `format: per_layer`, the tensor is indexed by `layer`.
+
+## What the automatic checks do
+
+On every PR that touches `community/`, [`community/check_vectors.py`](community/check_vectors.py) runs (from the main branch, not your PR's copy) and:
+
+- validates every entry's fields
+- for new or changed entries: downloads the file (max 100 MB), checks its sha256, loads it with the same code Modulo uses, checks the vector size against the model's `hidden_size`, the layer against its layer count, and that the norm is finite and non-zero
+- flags any change to a `verified` entry for maintainers
+
+Run it yourself first: `python community/check_vectors.py`.
+
+## For maintainers: promoting an entry
+
+On a GPU box with the repo checked out:
+
+```bash
+python community/gpu_verify.py --entry <id> --index <path to the PR's vectors.yaml> --pr <number>
+```
+
+This runs the vector at ±4/8/12/15 on the entry's test prompts plus generic ones, posts a table of where output changes and degenerates, collapsible samples, and a suggested safe range. If the samples match the claimed effect, set `trust: verified` and the safe range, and merge. The repetition detector doesn't catch incoherent but non-repetitive output, so read the samples.

community/check_vectors.py

@@ -0,0 +1,130 @@
+"""Check community/vectors.yaml: entry fields for every entry, plus file checks
+(download, sha256, load, shape vs the model's config, norm) for entries that are
+new or changed relative to a base index.
+
+CI runs this from the base branch against the PR's index, so a PR can't change
+the checks that judge it:
+
+    python community/check_vectors.py --index pr/community/vectors.yaml \
+        --base-index base/community/vectors.yaml --summary summary.md
+
+Run with --all to re-check every entry's file.
+"""
+import argparse
+import json
+import sys
+from pathlib import Path
+
+import yaml
+
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
+
+from modulo.community import MAX_FILE_BYTES, check_vector_file, validate_index  # noqa: E402
+
+
+def load_index(path: Path) -> dict:
+    if not path.exists():
+        return {"entries": []}
+    with open(path) as f:
+        return yaml.safe_load(f) or {"entries": []}
+
+
+def changed_entries(index: dict, base: dict) -> tuple[list[dict], list[str]]:
+    """Entries that are new or differ from base, and ids removed from base."""
+    base_by_id = {e.get("id"): e for e in base.get("entries", []) if isinstance(e, dict)}
+    ids = {e.get("id") for e in index["entries"] if isinstance(e, dict)}
+    changed = [e for e in index["entries"] if base_by_id.get(e.get("id")) != e]
+    removed = sorted(i for i in base_by_id if i not in ids)
+    return changed, removed
+
+
+def maintainer_warnings(changed: list[dict], base: dict) -> list[str]:
+    base_by_id = {e.get("id"): e for e in base.get("entries", []) if isinstance(e, dict)}
+    warnings = []
+    for e in changed:
+        was_verified = base_by_id.get(e["id"], {}).get("trust") == "verified"
+        if e["trust"] == "verified" or was_verified:
+            warnings.append(f"{e['id']}: adds, changes or demotes a verified entry (maintainers only)")
+    return warnings
+
+
+def model_dims(entry: dict) -> tuple[int | None, int | None, str | None]:
+    """(hidden_size, num_layers, note) from the model's config.json on the Hub."""
+    from huggingface_hub import hf_hub_download
+    try:
+        path = hf_hub_download(entry["model"], "config.json", revision=entry.get("model_revision"))
+    except Exception as e:  # gated or missing models: skip the shape checks, don't fail
+        return None, None, f"{entry['id']}: couldn't read {entry['model']} config ({type(e).__name__}); shape not checked"
+    cfg = json.loads(Path(path).read_text())
+    cfg = cfg.get("text_config", cfg) if "hidden_size" not in cfg else cfg
+    return cfg.get("hidden_size"), cfg.get("num_hidden_layers"), None
+
+
+def fetch_vector(entry: dict) -> tuple[Path | None, str | None]:
+    from huggingface_hub import HfApi, hf_hub_download
+    kwargs = dict(repo_id=entry["repo_id"], repo_type=entry["repo_type"], revision=entry.get("revision"))
+    try:
+        [info] = HfApi().get_paths_info(paths=[entry["filename"]], **kwargs)
+        if getattr(info, "size", None) is None:
+            return None, f"{entry['id']}: '{entry['filename']}' is not a file in {entry['repo_id']}"
+        if info.size > MAX_FILE_BYTES:
+            return None, f"{entry['id']}: file is {info.size / 1e6:.0f} MB (max {MAX_FILE_BYTES / 1e6:.0f} MB)"
+        return Path(hf_hub_download(filename=entry["filename"], **kwargs)), None
+    except Exception as e:
+        return None, f"{entry['id']}: couldn't download {entry['repo_id']}/{entry['filename']} ({type(e).__name__}: {e})"
+
+
+def check(index_path: Path, base_path: Path | None, check_all: bool) -> tuple[list[str], list[str], list[str] | None]:
+    """Returns (errors, warnings, checked ids); checked is None if field errors stopped the file checks."""
+    index = load_index(index_path)
+    errors = validate_index(index)
+    if errors:
+        return errors, [], None
+    base = load_index(base_path) if base_path else {"entries": []}
+    changed, removed = (index["entries"], []) if check_all else changed_entries(index, base)
+    warnings = maintainer_warnings(changed, base) + [f"{i}: removed" for i in removed]
+    for entry in changed:
+        path, err = fetch_vector(entry)
+        if err:
+            errors.append(err)
+            continue
+        hidden, layers, note = model_dims(entry)
+        if note:
+            warnings.append(note)
+        errors += check_vector_file(entry, path, hidden_size=hidden, num_layers=layers)
+    return errors, warnings, [e["id"] for e in changed]
+
+
+def summary_markdown(errors, warnings, checked) -> str:
+    lines = ["## Community vector index check", ""]
+    lines.append("**Result: " + ("❌ problems found" if errors else "✅ passed") + "**")
+    lines.append("")
+    if checked is None:
+        lines.append("Files checked: skipped until the problems below are fixed")
+    else:
+        lines.append(f"Files checked: {', '.join(f'`{i}`' for i in checked) or 'none (no new or changed entries)'}")
+    for title, items in (("Problems", errors), ("Notes for maintainers", warnings)):
+        if items:
+            lines += ["", f"### {title}", ""] + [f"- {i}" for i in items]
+    return "\n".join(lines) + "\n"
+
+
+def main():
+    root = Path(__file__).resolve().parent
+    parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    parser.add_argument("--index", type=Path, default=root / "vectors.yaml")
+    parser.add_argument("--base-index", type=Path, default=None, help="Only entries that differ from this get file checks")
+    parser.add_argument("--all", action="store_true", help="File-check every entry")
+    parser.add_argument("--summary", type=Path, default=None, help="Also write a Markdown summary here")
+    args = parser.parse_args()
+
+    errors, warnings, checked = check(args.index, args.base_index, args.all)
+    md = summary_markdown(errors, warnings, checked)
+    print(md)
+    if args.summary:
+        args.summary.write_text(md)
+    sys.exit(1 if errors else 0)
+
+
+if __name__ == "__main__":
+    main()