Repository navigation
Speed up socketcan capture_message by ~32% - #2097
Merged
zariiii9003 merged 2 commits intoNov 10, 2026
Merged
Conversation
TheTrueAI
force-pushed
the
perf/socketcan-hoist-msg-dontroute
branch
2 times, most recently
from
September 1, 2026 01:21
90a9123 to
48bb561
Compare
TheTrueAI
marked this pull request as ready for review
September 1, 2026 01:26
Contributor
|
Tick the box to add this pull request to the merge queue (same as
|
3 of 10 tasks
`socket.MSG_DONTROUTE` is an `enum.IntFlag`, so masking it built a `MsgFlag` per frame; it is now a module-level int. `Message` is built positionally to skip keyword binding, with a new test pinning the order. Live 500 kbit/s bus (aarch64, CPython 3.13.15): 107.8 -> 73.3 us CPU per frame; output identical over 3000 real and 1995 synthetic frames. Refs hardbyte#1135
TheTrueAI
force-pushed
the
perf/socketcan-hoist-msg-dontroute
branch
from
November 9, 2026 14:00
48bb561 to
fdf9200
Compare
Contributor
Author
|
@zariiii9003 thanks for merging #2114. I updated this branch on top of it and the full matrix is green now. The change is small (socketcan.py plus a test that pins the Message argument order) and doesn't change behaviour. Happy to adjust anything. |
zariiii9003
requested changes
Nov 10, 2026
zariiii9003
left a comment
Collaborator
There was a problem hiding this comment.
The change itself looks okay, but fix the tests please.
The previous test pinned the parameter order of Message.__init__, so a swapped argument in capture_message's positional Message() call would not have been caught. Feed frames through capture_message via a mocked socket instead and compare every Message field. The cases are chosen so that every pair of boolean fields differs in at least one of them.
Contributor
Author
|
Only Python 3.10 fails. Since 3.10 is anyways EOL, is it planned to be dropped anyways soon? Then I would just park this PR till after the drop. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary of Changes
capture_messageruns once per received frame, and two lines in it cost more thanthe rest of the decode combined. This removes both. No API or behaviour change.
socket.MSG_DONTROUTEto a module-level plainint. It is anenum.IntFlagmember, somsg_flags & socket.MSG_DONTROUTEconstructed aMsgFlaginstance on every frame.Messagepositionally instead of with twelve keyword arguments;binding them per frame is measurable.
test_capture_messagetoSocketCANTest, which runs frames throughcapture_messagevia a mocked socket and checks everyMessagefield. The casesare chosen so that swapping any two of the positional arguments fails the test.
Net effect on a live 500 kbit/s bus: 107.8 -> 73.3 us of CPU per frame, a ~32%
reduction. Full A/B and correctness evidence under Additional Notes.
Related Issues / Pull Requests
Related to #1135
At the ~10k f/s of a busy 1 Mbit bus, stock
capture_messageneeds 1057 ms of CPUper second of bus time — more than a whole core just to decode, before any user code
runs. That is a concrete mechanism behind the "python can't keep up" reports in
#1135. This brings it to 733 ms/s. It does not close that issue, but it is a real
part of it.
Type of Change
hot path. Behaviour-preserving; verified field-for-field against the previous
implementation (see below).
Checklist
tox).Additional Notes
Measurements
Live 500 kbit/s bus at 1444 frames/s, aarch64 (Cortex-A53) / CPython 3.13.15 /
python-can 4.6.1. CPU time via
time.process_timeso blockingrecvmsgwaits areexcluded. 2500 frames x 3 rounds, median. Each row changes exactly one thing against
a verbatim copy of the current implementation:
capture_messageMSG_DONTROUTEhoisted to intMessage()End to end with this branch applied, same bus: 107.8 -> 73.3 us/frame.
For scale, the bare
recvmsgsyscall is 31 us/frame, so this removes ~45% of thePython-side work above the syscall. At 1444 f/s
capture_messagedrops from 15.3%to 10.4% of one core.
A tight-loop microbenchmark of the
&alone reports only 12.8 us(
13.2 usIntFlag vs0.5 usint), i.e. it understates the in-situ saving of26.3 us. A cache-thrash variant widens the microbenchmark gap (13.2 -> 15.8 us),
so the enum machinery's cold per-frame working set appears to be part of it. I could
not fully account for the remainder; the in-situ A/B is the number I would trust.
Correctness
Compared against the current implementation on identical inputs, all twelve
Messagefields plus their types:
get_channelbothFalseandTrue— 0 mismatches.MSG_DONTROUTEset (798 cases, i.e.is_rx=False)and clear, standard and extended IDs, remote and error frames, classic and FD MTU,
BRS/ESI, every valid DLC and the
len8_dlc9..15 range — 0 mismatches.The live bus only produces standard non-FD
is_rx=Trueframes, so the synthetic setis what actually exercises the changed line.
Things I measured and deliberately did not include
CAN_FD_DLCas afrozensetfor thedata_len not in can.util.CAN_FD_DLCtest in
dissect_can_frame. Looks like a 0.68 us win in isolation (1.03 -> 0.35 us)but measured -0.41 us, i.e. nothing, in situ: classic-CAN
data_lenis 0..8 andhits within the first nine list entries.
asserts on the ancillary data: worth only 1.0 us,not enough to justify the behaviour discussion.
recvmsg_intowith a preallocated buffer: 2.9 us/frame slower.bytes->bytearraycopy:4.6 us/frame slower.