Skip to content

Feat: compute metrics with max_spikes_per_batch - #4948

Open
alejoe91 wants to merge 1 commit into
SpikeInterface:mainfrom
alejoe91:optimize-qm
Open

alejoe91 wants to merge 1 commit into
SpikeInterface:mainfrom
alejoe91:optimize-qm

Conversation

@alejoe91

Copy link
Copy Markdown
Member

Closes #4947

Spike-level quality metrics used to build full-size caches of the whole spike vector (to_reordered_spike_vector, get_spike_vector_to_indices, plus a copy from select_periods when periods are used). So peak RAM grew with the number of spikes. This PR reads spikes and extension data in time batches (max_spikes_per_batch, default 1e6) and accumulates each metric across batches. Peak RAM is now roughly flat with recording length.

  • Exact, same values as main: num_spikes, firing_rate, presence_ratio, isi/rp/sliding_rp violations, synchrony, firing_range and drift.
  • Same up to float32 vs float64 rounding: amplitude_cv and sd_ratio.
  • amplitude_cutoff, amplitude_median and noise_cutoff take a new max_spikes_per_unit (default 100_000; None uses all spikes). It subsamples each unit evenly in time.

Behavior changes

  • amplitude_cutoff, amplitude_median and noise_cutoff change for units above 100k spikes. noise_cutoff changes the most: about 0.04 on values between −0.5 and −0.2.
  • synchrony now works on lazily loaded zarr analyzers. Main returned NaN there.
  • With periods on multi-segment analyzers, metrics now return NaN when the periods leave a unit without spikes. Main raised a KeyError in that case.
  • compute_drift_metrics(..., return_positions=True) now returns the per-unit bin medians.

Benchmark vs main

Lazy zarr analyzer, 500 units, all non-PCA metrics, best of 3:

spikes main this PR
2 h 36M 3.0 GiB, 307 s 0.77 GiB, 81 s
6 h 108M 8.4 GiB, 652 s 0.81 GiB, 216 s

Metrics computed one at a time go from 2–8.7 GiB to 0.2–0.5 GiB at 6 h. Small count and window metrics get slower in absolute terms: 1.5–6 s on main, 3–17 s here.

PCA metrics are not changed. Their memory does not depend on the number of spikes, and I'll address them separately.

@alejoe91 alejoe91 added performance Performance issues/improvements metrics Related to metrics module labels Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

chronic metrics Related to metrics module performance Performance issues/improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Quality metrics memory grows with the number of spikes on long recordings

1 participant