Parallelize coherence-based bad channel detection - #4750
Conversation
alejoe91
left a comment
There was a problem hiding this comment.
Thanks @JESUSROYETH
I just have one comment but this looks great! :)
Co-authored-by: Alessio Buccino <alejoe9187@gmail.com>
for more information, see https://pre-commit.ci
Thanks for the review, @alejoe91! I applied both suggestions. |
|
Thanks @JESUSROYETH! This looks great will check it out tomorrow |
There was a problem hiding this comment.
Hi @JESUSROYETH thanks for this. The speedup is nice and the implementation is simple. I think this is a nice change.
@samuelgarcia or @alejoe91 may want to weight in as they know the parallel stuff better than me, but I cant immediately see any ways this could cause an issue.
I'm just commenting as have not played around locally or looked properly at the tests, will do this early next week!
Note to self: also check #4434 when playing around and close it.
|
sorry @JESUSROYETH I did not think, that most recent refactor makes it hard to read the diff. Please revert it while others take a look, maybe it can be refactored as a follow up PR. |
No worries, you are right — the refactor makes the diff harder to read. I reverted it, and the PR is back to the previous version. We can revisit the refactor in a follow-up PR. |
Description
The default
coherence+psdmethod first materialises 100 scaled chunks and then processes them in serial. On a 384-channel recording, profiling shows that the Welch and median computations dominate the call, while extracting the random traces takes a small part of the total time.This change sends the same random slices through
TimeSeriesChunkExecutor. The default stays atn_jobs=1, but the chunks are streamed instead of kept together in memory. A finaljob_kwargsparameter lets callers select a thread or process pool without changing positional arguments.I measured the default 100 × 0.3 s chunk path on an Intel Core i9-13900HX, with numerical thread pools pinned to one thread. The generated row uses the 30 s / 384-channel shape reported in the issue. The second row uses real 384-channel SpikeGLX traces from the public Noise4Sam fixture; the public file is short, it was repeated in memory to cover the same chunk path.
n_jobs=8, processes)The channel ids and labels matched exactly between both implementations in each run. On the serial real-trace path, three fresh-process peak RSS measurements gave a median of 1,688,604 KiB before (range 1,688,500-1,689,352) and 353,452 KiB after (range 352,984-354,896), around 79% less. This comes from processing one random slice at a time instead of retaining the scaled chunks.
Validation:
pytest -q src/spikeinterface/preprocessing/tests/test_detect_bad_channels.py: 11 passed.highpass_filterwrapperdetect_bad_channelsbuilds internally also gets exercised across a process boundary; it matches the serial result on the same data.mainrevision and pass with this patch.blackand the trailing-whitespace/end-of-file checks are clean.Fixes #2869.