Repository navigation
Fix phantom spikes in wobble under amplitude scaling, add an optional joint fit - #4951
Open
grahamfindlay wants to merge 2 commits into
Open
grahamfindlay wants to merge 2 commits into
grahamfindlay wants to merge 2 commits into
Conversation
With scale_amplitudes on, a detected spike was removed from the objective at its fitted amplitude but from the normalized objective at amplitude 1. For a spike larger or smaller than its template, the difference stayed in the normalized objective and could be detected again as a spike of another unit. Both objectives now use the fitted amplitude. The fitted amplitude itself was read from the wrong row. The best jitter is a position within the jitter window, but it was used to index the scalings array, which spans the whole upsampled window. It now goes through jitter_window first. Add a test with three noiseless spikes at 2, 4 and 0.6 times their templates, which must come back as exactly those three spikes.
Wobble fits each spike's amplitude once, when it is found. If a second spike overlapping it is found later, the first amplitude has already absorbed part of the second spike, which is then missed or misplaced. With the new parameter joint_fit, each new spike's start, jitter and amplitude are fitted together with the spikes it overlaps, by least squares with the amplitude prior. Those spikes are then placed and fitted again while any of them moves. A spike is fitted with at most 8 others, the ones most coupled to it, which bounds the work in dense stretches. A spike whose amplitude is refitted to zero is dropped. joint_fit is off by default. It needs amplitude scaling (amplitude_variance > 0) and numba; the kernels are in _wobble_numba.py. test_wobble_scaled_spikes now also runs with joint_fit, with two overlapping spikes added.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problems
wobble keeps two tables per chunk:
objective, from which a spike's amplitude is fitted, andobjective_normalized, from which spikes are found. With amplitude scaling on, which is the default (amplitude_variance=1),subtract_spike_traintakes a found spike out ofobjectiveat its fitted amplitude but out ofobjective_normalizedat amplitude 1.For a spike larger than its template, the difference stays in the table spikes are found from. The spike's own unit is blocked there by the refractory mask, so another unit picks the leftover up, yielding a false spike. For a spike smaller than its template, too much is removed, which can hide a real spike that overlaps it.
A second problem is with the amplitude itself. The best shift is found as a position within
jitter_window, but that position is used to indexscalings, which spans the whole upsampled window, and this wrong amplitude is taken.Fixing the 2 problems gets rid of phantom spikes, but leaves 1 residual (pun intended) problem: a spike fitted alone absorbs part of a neighbor it overlaps, and that neighbor can then be missed.
Changes
Commit 1 fixes problems 1+2 and therefore the phantom spikes:
subtract_spike_trainnow takes the spike out of both tables at its fitted amplitude, and the amplitude is read throughjitter_window. There is no downside to this commit, I think.Commit 2 fixes the third problem: It adds a
joint_fitoption, off by default. Withjoint_fit=True, each new spike's start, shift and amplitude are fitted together with the spikes it overlaps, by least squares with the amplitude prior, and those spikes are then placed and fitted again while any of them moves. A spike is fitted with at most the 8 spikes most coupled to it. A spike refitted to amplitude zero is dropped.joint_fitneeds needs numba. My initial attempt to do this in numpy was slow. Too slow to be reasonable, especially since wobble was already kinda slow. So I asked Claude to optimize a numba kernel, which makes thejoint_fitruntime tolerable, and in fact it is often faster than stock wobble was on non-pathological (most) data. But since the benefits ofjoint_fit, though very clear on tetrode data, are less clear on neuropixel data, I have left it off by default. Plus, with the joint fit, it's a little bit more of a "wobble 2.0" than "fixed wobble".Other
Sort of a separate but related discussion (does not need to be resolved here): should
amplitude_variance=1remain the default? It is kind of extreme (e.g. see dartsort's choice of 0.01).Also, not in this PR, but worth mentioning: The output's
amplitudefield is still the trace value at the spike's largest channel, not the fitted amplitude.@samuelgarcia @yger