Skip to content

Commit 9908bb4

Browse files
Hananel-HazanclaudeAHMETHAKANBEZIR1codex
authored
CI: close test-guard bypasses found by independent review (#832)
* fix: IzhikevichNodes and CSRMNodes accept multi-dimensional shapes; CSRMNodes reset clears its spike window IzhikevichNodes sized its parameters by the n argument (None when only shape is given) and never reshaped them; CSRMNodes contracted with 'i,kij->kj'. Both now work for any shape; one-dimensional results are bit-identical (goldens conv2d_if_izh and csrm_window trial 0 unchanged). CSRMNodes.reset_state_variables now zeroes last_spikes, which changes results after a reset (csrm_window golden regenerated with that reason). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: LocalConnection batch > 1; MaxPool connections without decay or with one channel LocalConnection.compute dropped the batch dimension in its final view. MaxPool{1,2,3}d multiplied a missing decay (None) and squeeze()d away a channel of size 1; a missing decay now means plain spike counts, and the firing-rate buffer follows the spike tensor's shape. Results that ran before are unchanged (goldens pass). Reproducers kept as test_fixed_bug_stays_fixed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: sparse MulticompartmentConnection learns and runs with batch > 1; NetworkMonitor reset; tuple target shapes - MCC_LearningRule bounded sparse weights into a new tensor that only the rule held, so DiehlAndCook2015(sparse=True) never learned; the bound and the sparse normalisation now write in place. At batch 1 sparse and dense learn bit-identical weights without norm (test_sparse_dc2015_learns_like_dense); with norm they differ by summation order (max 6.0e-8). - MulticompartmentConnection.compute applies per-sample sparse weight copies ([batch, src, tgt]) with bmm; one copy keeps the old arithmetic. - NetworkMonitor.reset_state_variables looked connections up in layers. - [batch] + target.shape failed for tuple shapes. Results change: DiehlAndCook2015(sparse=True) now learns. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: MSTDP/MSTDPET/Rmax state cleared on reset; Rmax and MCC MSTDPET work with batch > 1 - LearningRule.reset_state_variables deletes the tensors a rule creates on its first update (_step_state); AbstractConnection.reset_state_variables forwards to the rule. Reset then equals a freshly built network (invariant tests for mstdp_mstdpet and rules_mix now pass). Results change after a reset with learning on: goldens mstdp_mstdpet(_dt05), rules_mix(_dt05) regenerated, trial 0 bit-identical. - Rmax and MCC_learning.MSTDPET keep per-sample state and reduce the update over the batch, as MSTDP does. Batch 1 is bit-identical to the old formula (TestMccMstdpetBatched, goldens); batch 2 with sum equals the sum of single samples to summation order. Op counts at batch 1: MCC MSTDPET 24 -> 27, Rmax 24 -> 26 (regenerated with reason). API snapshot: +1 method. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: preserve the mean-field weight parameter during normalization (#813) Adopted from #813 (head 53dd2e3) by AHMETHAKANBEZIR1, reviewed with the bindsnet-pr-review skill (sandboxed canaries, fail-then-pass, API, timing); Hananel adopted it as accountable author because the PR says no human reviewed it. MeanFieldConnection.normalize scales w in place by norm over the absolute weight sum (a shared scalar counted once per target), instead of assigning a plain tensor to the Parameter (TypeError on every call). Co-Authored-By: Hananel Hazan <hananel@hazan.org.il> Co-Authored-By: OpenAI Codex <codex@openai.com> * fix: reset and resize the CSRM connection spike history (#815) Adopted from #815 (head 49c57f8) by AHMETHAKANBEZIR1, reviewed with the bindsnet-pr-review skill; Hananel adopted it as accountable author because the PR says no human reviewed it. Connection.compute_window re-creates s_w when the batch size changes, and reset_state_variables clears it for CSRM targets. Co-Authored-By: Hananel Hazan <hananel@hazan.org.il> Co-Authored-By: OpenAI Codex <codex@openai.com> * test: goldens and invariants after adopting #813 and #815 meanfield_norm and csrm_window goldens regenerated (both used to record a crash). Invariant xfails removed: reset-equals-fresh for csrm_window and meanfield_norm, round trip and batch for meanfield_norm. Still open: the CSRM spike window is not in the state_dict (round trip). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: CSRM spike window saved in the state_dict; MulticompartmentConnection drives CSRMNodes - Connection.s_w is a registered buffer, so state_dict saves it; a load adopts a saved window or drops one the saved state lacked (_load_window). The csrm_window round-trip invariant now passes; golden regenerated (adds s_w, no value changed). - MulticompartmentConnection.compute_window read s_w and self.w, neither of which it has, so MCC into CSRMNodes always raised. It now keeps a window of its own outputs (pipeline runs once per step), resized on a batch change, cleared on reset, saved in the state_dict. With static weights it matches a Connection bit for bit (test_multicompartment_into_csrm_matches_connection). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: MeanFieldConnection keeps batch samples apart and applies weight_decay - compute() took the mean over the whole batch, so one sample's input depended on the others; it now uses each sample's own mean (batch 1 bit-identical: goldens pass without the weight_decay change). - weight_decay was passed by position into AbstractConnection's reduction slot and ignored; it is now passed by name. Results change for anyone who set weight_decay on a MeanFieldConnection (golden meanfield_decay regenerated with that reason). - Two #813 test lines expected the batch-wide mean (0.75); corrected to the per-sample means. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: MeanField feature, Normalization/Updating sub-features, adaptation features at batch > 1, sparse weight bounds - topology_features.MeanField never called the base constructor (no name) and had no initial value; it now has an optional name, a placeholder value, and a per-sample mean ([batch, src, tgt], applied per sample by compute). - Sub-features (Normalization, Updating) get no-op prime_feature, update, normalize and reset_state_variables, which MulticompartmentConnection calls on every pipeline entry; both now work in the constructor and in Network.run. - AdaptationBaseSynapsHistory/OtherSynaps: a synapse is active when its pre-synaptic neuron spiked in any sample (the mask is shared connectivity); batch 1 gives the same values. - Sparse weights with wmin/wmax: Connection.__init__ bounds a dense copy, LearningRule.update bounds in place via a dense copy, and WeightDependentPostPre/DiehlAndCook compute their factors from a dense view. PostPre, WeightDependentPostPre and DiehlAndCook on a bounded SparseConnection learn bit-identical weights to the dense Connection. - Test workarounds for these bugs removed (batch-1-only sets, broken pairs, tuple-shape and sparse-bound special cases). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: known-bug list becomes FIXED_BUGS; workarounds removed; MulticompartmentConnection feature values in the state_dict - MulticompartmentConnection saves its pipeline feature values as feature.<name> and loads them in place, so DiehlAndCook2015 can be saved and restored (the last four strict xfails in test_invariants.py now pass; dc2015 goldens regenerated: new keys, no value changed). - Every KNOWN_BUGS entry is fixed: the dict becomes FIXED_BUGS (reproducers that must keep running); the empty xfail machinery in test_invariants.py and the batch-1, no-learning-check and sub-feature workarounds in test_all_classes.py are removed, so those classes run at batch 2 with full checks. - CONTRIBUTING, PR template and bindsnet/learning/README.md updated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: CHANGELOG for the 0.3.5 bug fixes, results changes listed first Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: issues found by independent review of PR #831 Network (reviewer B): - MulticompartmentConnection state_dict: refuses to save duplicate feature names (they shared one key, and loading put one feature's value into the other); checks shape and sparse layout on load (copy_ broadcast a wrong-shaped tensor silently); saves the adaptation features' mask, spike buffer and counter (only value was saved). - An Updating sub-feature drives its parent's learning; the connection's update skips that parent (it learned twice per step). - A sub-feature whose parent is not in the pipeline is refused at construction (it failed later with an unclear TypeError). insert_pipeline had list.insert's arguments swapped. - Connection.compute_window and MaxPool resets create state on the spikes' device (CUDA device mismatch, also on master). - Docstrings state how the two CSRM windows differ once weights learn. Learning rules (reviewer A): - MCC_learning.DiehlAndCook crashed on sparse weights (twin of the learning.py fix). Sparse MCC weight copies now all get the batch's shared update in PostPre, DiehlAndCook and MSTDPET (as MSTDP did), so sparse equals dense; the default squeeze reduction means a sum over a larger batch, as in the fused dense path (_reduce_batch, _add_to_feature). - Docs: per-sample reward shape [batch, 1, 1]; reset keeps the reduction chosen at construction. CHANGELOG: checkpoint compatibility (strict=False for old checkpoints), new state shapes, sparse and Updating semantics. Each fix has a test that fails on ce98f93 and passes here. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci: close test-guard bypasses found by independent review (WIP part 1) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: op-count workloads for every Nodes class Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci: close the test-guard bypasses found by independent review (part 2), docs Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: AHMETHAKANBEZIR1 <ahmethakan0008@gmail.com> Co-authored-by: OpenAI Codex <codex@openai.com>
1 parent d48dc79 commit 9908bb4

14 files changed

Lines changed: 734 additions & 131 deletions

‎.github/CODEOWNERS‎

Lines changed: 12 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,14 @@
11
# Pull requests that touch these files need a maintainer's approval (branch
22
# protection: "Require review from Code Owners"). They hold recorded results,
3-
# the tests that pin formulas to their papers, and the CI that runs them; a
4-
# pull request must not edit them to make itself pass.
5-
/test/regression/ @Hananel-Hazan
6-
/test/api_surface.json @Hananel-Hazan
7-
/test/regen_api_surface.py @Hananel-Hazan
8-
/test/test_api_surface.py @Hananel-Hazan
9-
/test/network/test_all_classes.py @Hananel-Hazan
10-
/test/network/test_learning_rule_specs.py @Hananel-Hazan
11-
/test/network/test_mstdp_florian.py @Hananel-Hazan
12-
/test/network/test_perf_equivalence.py @Hananel-Hazan
13-
/.github/ @Hananel-Hazan
3+
# the tests and the settings that decide which tests run, and the CI that runs
4+
# them; a pull request must not edit them to make itself pass.
5+
/test/ @Hananel-Hazan
6+
/.github/ @Hananel-Hazan
7+
/pyproject.toml @Hananel-Hazan
8+
/poetry.lock @Hananel-Hazan
9+
/pytest.ini @Hananel-Hazan
10+
/.pytest.ini @Hananel-Hazan
11+
/tox.ini @Hananel-Hazan
12+
/setup.cfg @Hananel-Hazan
13+
/.coveragerc @Hananel-Hazan
14+
/conftest.py @Hananel-Hazan

‎.github/scripts/bench_ab.py‎

Lines changed: 28 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -30,8 +30,12 @@
3030
3131
The report (Markdown) is written to DIR/bench_ab.md, appended to --summary if
3232
given (GitHub's $GITHUB_STEP_SUMMARY), and the raw numbers to DIR/bench_ab.json.
33-
The exit status is always 0: this is a non-required check and a human decides;
34-
the verdict, or a TOOL ERROR line, is in the report.
33+
Without --fail-above the exit status is always 0 and a human decides; the
34+
verdict, or a TOOL ERROR line, is in the report. With --fail-above X the exit
35+
status is 1 when any workload's median ratio is above X and its interval's lower
36+
end is above 1.5 (a large, clear slowdown, far outside the no-change spread of
37+
0.90 to 1.12 per round measured on GitHub runners), or when the tool failed; CI
38+
uses X = 2.
3539
"""
3640

3741
from __future__ import annotations
@@ -318,6 +322,12 @@ def main() -> int:
318322
ap.add_argument("--time", type=int, default=100, help="time steps per run")
319323
ap.add_argument("--threads", type=int, default=2)
320324
ap.add_argument("--threshold", type=float, default=1.25)
325+
ap.add_argument(
326+
"--fail-above",
327+
type=float,
328+
default=None,
329+
help="exit 1 when a median ratio is above this and the interval's lower end above 1.5",
330+
)
321331
ap.add_argument("--min-seconds", type=float, default=1.0)
322332
ap.add_argument("--only", default=",".join(WORKLOADS))
323333
ap.add_argument("--timeout", type=int, default=900, help="seconds per process")
@@ -383,10 +393,25 @@ def log(msg):
383393
indent=1,
384394
)
385395
)
396+
failing = []
397+
if args.fail_above is not None:
398+
failing = [
399+
r["workload"]
400+
for r in rows
401+
if r["ratio_median"] > args.fail_above and r["ratio_ci95"][0] > 1.5
402+
]
403+
if error:
404+
failing.append("tool error")
405+
report += (
406+
f"\n**Failed:** {', '.join(failing)} (median above {args.fail_above}x "
407+
"the base, or the tool failed).\n"
408+
if failing
409+
else f"\nRequired limit: no workload above {args.fail_above}x the base.\n"
410+
)
386411
if args.summary:
387412
with open(args.summary, "a") as f:
388413
f.write(report)
389-
return 0
414+
return 1 if failing else 0
390415

391416

392417
if __name__ == "__main__":

‎.github/scripts/coverage_floor.py‎

Lines changed: 10 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -13,12 +13,19 @@
1313
import sys
1414
import xml.etree.ElementTree as ET
1515

16-
# Measured on CI? No: measured locally on 2026-10-24 with torch 2.14.1 CPU after
17-
# Measured on GitHub CI (build job, Python 3.13, CPU torch), run 40798879593,
18-
# 2026-10-28.
16+
# One point under the coverage measured on GitHub CI (build job, Python 3.13, CPU
17+
# torch): network and learning in run 40798879593, the others in run 40804291700
18+
# (master 33913c7), both 2026-10-28.
1919
FLOORS = {
2020
"network": 89.6, # 90.6% measured
2121
"learning": 94.2, # 95.2% measured
22+
"models": 98.3, # 99.3% measured
23+
"encoding": 73.6, # 74.6% measured
24+
"conversion": 58.8, # 59.8% measured
25+
"analysis": 32.8, # 33.8% measured
26+
"environment": 26.3, # 27.3% measured
27+
"pipeline": 19.8, # 20.8% measured
28+
"datasets": 20.6, # 21.6% measured
2229
}
2330
# Cobertura filenames are relative to the <source> element, which pytest-cov sets to
2431
# the bindsnet package directory, so "network/nodes.py", not "bindsnet/network/...".

‎.github/scripts/fails_before_fix.py‎

Lines changed: 53 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -9,8 +9,13 @@
99
- on the head commit as checked out.
1010
1111
It passes when at least one test fails (or cannot be collected) on the base and
12-
passes on the head. A test file that cannot be imported on the base (for
13-
example, it imports a function the pull request adds) counts as failing there.
12+
passes on the head. Only a test that ran on the base and failed there counts
13+
(any exception in its body, including an assertion or "DID NOT RAISE"). A test
14+
that cannot be imported or set up on the base (for example, it imports a
15+
function the pull request adds) shows no bug and does not count; a pull request
16+
that adds a feature needs the maintainer label. The summary shows each counted
17+
test's failure on the base, because a test can also fail for reasons unrelated
18+
to the bug, and a maintainer must read it.
1419
1520
It also passes, with a note, when the pull request does not touch any
1621
``bindsnet/*.py`` file (documentation, dependencies, workflows), or when the
@@ -53,8 +58,11 @@ def changed(base, head, path, filters="ACMR"):
5358
def run_pytest(tree, files, xml):
5459
"""Run pytest on ``files`` inside ``tree`` with ``tree`` first on the path.
5560
56-
Returns {test id: "pass" | "fail"} and {module: "error"} for files that
57-
could not be collected. Skipped and xfailed tests are left out.
61+
Returns {test id: ("pass" | "fail" | "error", message)} and the modules that
62+
could not be collected. "fail" means the test ran and failed (any exception
63+
in its body); "error" means set-up or collection failed. Skipped and xfailed
64+
tests are left out. The settings come from the tree's pyproject.toml only, so
65+
an added pytest.ini or setup.cfg cannot change what runs.
5866
"""
5967
env = dict(os.environ, PYTHONPATH=str(tree))
6068
where = subprocess.run(
@@ -69,7 +77,8 @@ def run_pytest(tree, files, xml):
6977
raise RuntimeError(f"bindsnet imported from {where}, not from {tree}")
7078
subprocess.run(
7179
[sys.executable, "-m", "pytest", "-q", "-p", "no:cacheprovider"]
72-
+ ["--continue-on-collection-errors", f"--junitxml={xml}", *files],
80+
+ ["-c", "pyproject.toml", "--continue-on-collection-errors"]
81+
+ [f"--junitxml={xml}", *files],
7382
cwd=tree,
7483
env=env,
7584
)
@@ -82,7 +91,12 @@ def run_pytest(tree, files, xml):
8291
test_id = f"{case.get('classname')}::{case.get('name')}"
8392
if "skipped" in tags:
8493
continue
85-
results[test_id] = "fail" if tags & {"failure", "error"} else "pass"
94+
for kind, tag in (("fail", "failure"), ("error", "error"), ("pass", None)):
95+
child = case.find(tag) if tag else None
96+
if tag is None or child is not None:
97+
message = (child.get("message") or "") if child is not None else ""
98+
results[test_id] = (kind, message.splitlines()[0][:200] if message else "")
99+
break
86100
return results, broken
87101

88102

@@ -106,6 +120,10 @@ def finish(code, message):
106120
f.write(text)
107121
return code
108122

123+
# Commit ids, so "master" or "HEAD" mean the same inside the base worktree.
124+
args.base = git("rev-parse", args.base).strip()
125+
args.head = git("rev-parse", args.head).strip()
126+
109127
labels = {label.strip() for label in args.labels.split(",") if label.strip()}
110128
if args.skip_label in labels:
111129
return finish(0, f"Skipped: a maintainer added the label `{args.skip_label}`.")
@@ -137,26 +155,42 @@ def finish(code, message):
137155
subprocess.run(["git", "worktree", "remove", "--force", str(base_tree)])
138156
shutil.rmtree(work, ignore_errors=True)
139157

140-
def failed_on_base(test_id):
158+
def collection_broken(test_id):
141159
module = test_id.split("::")[0]
142-
if any(module == m or module.startswith(m + ".") for m in broken):
143-
return True
144-
return on_base.get(test_id) == "fail"
145-
146-
shown = [t for t, r in on_head.items() if r == "pass" and failed_on_base(t)]
160+
return any(module == m or module.startswith(m + ".") for m in broken)
161+
162+
passing = [t for t, (kind, _) in on_head.items() if kind == "pass"]
163+
# Evidence: the test ran on the base and failed there. A test that could not
164+
# be imported or set up on the base (for example, it imports a name the PR
165+
# adds) shows nothing about a bug; it is listed for the maintainer instead.
166+
shown = [t for t in passing if on_base.get(t, ("", ""))[0] == "fail"]
167+
unshown = [
168+
t
169+
for t in passing
170+
if t not in shown and (collection_broken(t) or on_base.get(t, ("",))[0] == "error")
171+
]
172+
if unshown:
173+
lines.append(
174+
"Not counted (could not be collected or set up on the base, so they "
175+
"show no bug): " + ", ".join(f"`{t}`" for t in sorted(unshown)[:20])
176+
)
177+
lines.append("")
147178
if shown:
148-
lines.append("Tests that fail on the base and pass with this change:")
149-
lines += [f"- `{t}`" for t in sorted(shown)[:50]]
179+
lines.append(
180+
"Tests that fail on the base and pass with this change, with the "
181+
"base failure (a maintainer checks that it is the bug):"
182+
)
183+
lines += [f"- `{t}`: {on_base[t][1]}" for t in sorted(shown)[:50]]
150184
if len(shown) > 50:
151185
lines.append(f"- ... and {len(shown) - 50} more")
152186
return finish(0, "\n**Passed.**")
153187
return finish(
154188
1,
155-
"**Failed:** every test in the changed test files gives the same result "
156-
"with and without the library change, so none of them shows the bug "
157-
"this pull request fixes. Add a test that fails on `master` and passes "
158-
"with this change. If this is not a bug fix (a refactor or a speed-up), "
159-
f"a maintainer can add the label `{args.skip_label}`.",
189+
"**Failed:** no test in the changed test files ran and failed on the "
190+
"base while passing with this change, so none of them shows the bug this "
191+
"pull request fixes. Add a test that fails on `master` and passes with "
192+
"this change. If this is not a bug fix (a refactor, a speed-up, or a new "
193+
f"feature), a maintainer can add the label `{args.skip_label}`.",
160194
)
161195

162196

0 commit comments

Comments
 (0)