Repository navigation
Fix starvation in mutually exclusive callback groups of the MultiThreadedExecutor - #2702
HarunTeper wants to merge 1 commit into
Conversation
|
The proposed changes, as well as the paper, don't make sense to me. But this might be a wording vs implementation issue. Lets wait for the actual implementation... |
|
Hi @HarunTeper, The next Client Library WG meeting will happen Friday 12/20/2024 at 8AM PST. |
I would likely join the next meeting, and work on the pull request until then. Sorry for not making it to the last one. |
|
I have now uploaded my fix for humble to a custom repository: https://github.com/HarunTeper/rclcpp_humble_multithreaded_executor You can see the changes in the branch called "fix". I am currently working on applying the same changes to the current version of the multi-threaded executor. However, since Humble, the code of the executor itself, specifically the wait set and wait_for_work functions are very different. I will try to implement the version that I propose, however, it will be different from my previous solution. For example, I think the current version has threads busy waiting for tasks to become ready if all tasks are blocked but not all threads are currently executing tasks. If there is another meeting, I will join that one to talk about the executor. In the meantime, I will work on the fix. |
|
I had a look at the 'fix' implementation. As far as I can see, this comes down to:
While I think, this fix works, the implementation is awkward, in the sense that the code flow is extremely hard to follow. To be fair, the executor code flow isn't great in the first place... I would suggest a different approach to the problem:
Poll for events:
Worker threads:
This is basically the logic I use in the EventsCBGExecutor. |
|
This pull request has been mentioned on ROS Discourse. There might be relevant details there: https://discourse.ros.org/t/multi-threaded-executor-task-starvation-fix/42190/1 |
|
Hello, I have added some changes to the structure of the executor.cpp. Mainly I have split the wait_for_work function into prepare_work and wait_for_work_simple (placeholder until wait_for_work can be fully replaced). I am now working on the following changes:
If anyone has some suggestions on how I should access wait_result_ and which data structures I should store, please let me know. |
…adedExecutor An entity that is ready while its mutually exclusive callback group is in use is skipped. The wait result is dropped before the next wait, so once the group is free again all of its ready entities are found at the same time and the one that comes first in the wait set is picked every time. With callbacks that take longer than their period the other entities of the group are never executed. Remember the skipped entities in the order in which they were skipped and let them go first when their callback group can be taken from again. Signed-off-by: HarunTeper <harun.teper@tu-dortmund.de>
|
Sorry for the long silence on this. I have rewritten this PR completely: the branch was @jmachowinski you wrote that the earlier implementation was hard to follow and suggested I did not go as far as the idle/processing lists. That would be a rewrite of how the Two things I found on the way:
The EventsCBGExecutor passes the same test scenario on jazzy and later, I have included it |
|
Update on the turn rule (alternative 1 in the description). I wrote there that it also costs time in the SingleThreadedExecutor and that I wanted to change that. I have done it now. The entity that ran last is now only remembered for a callback group after an entity of that group was blocked for the first time. Callback groups without contention do not use the rule, in any executor. It is one more commit on the turn rule (+20 -6 lines), branch
The cost with one entity per callback group and the extra allocations are gone. For the plain Waiting time of the starved entity (same setup as measurement 1, 5 runs of 30 s):
The 40 ms is the first message of each run and only that one. The rule starts with the first blocking in a group, so the first round is not in turns yet. Polling and the reference system are the same as with the first version of the turn rule. This PR still has only the fix. I can add the turn rule in this version as a second commit here or open a follow-up PR after this one, whichever you prefer. |
Fixes #2360, #2645.
Problem
If two entities share a mutually exclusive callback group and both are ready whenever the
group becomes free, the MultiThreadedExecutor always picks the one that comes first in the
wait set. The other one is never executed. The added test shows this with two 5 ms timers
whose callbacks take 20 ms: one timer gets all 20 executions, the other none. This is the
case on humble, jazzy, kilted, lyrical and rolling.
Change
The executor remembers the ready entities it had to skip because their callback group was
in use, in the order in which it skipped them. When a group becomes free, an entity from
this list is executed before any other entity of the same group.
executor.cpp. The list lives inExecutorImplementation.entity is blocked by its callback group. Fix MultiThreadedExecutor busy wait on busy callback groups #3395 does not end the starvation, the new test
fails on 917c124. The
can_be_taken_fromlambda that Fix MultiThreadedExecutor busy wait on busy callback groups #3395 added toget_next_ready_executableis the place where an entity is skipped, and it now also putsthe entity into the list. The rebuild request of Fix MultiThreadedExecutor busy wait on busy callback groups #3395 is unchanged.
get_next_ready_executable, which the MultiThreadedExecutor already calls underwait_mutex_."What it does not do" after the measurements.
wait_for_work,execute_any_executableandMultiThreadedExecutor::runare unchanged.Until now this PR contained a larger change that followed the design of our paper: a
second mutex, a split of
wait_for_workand new members inExecutor. That is not neededany more, because the executor now keeps the wait result between calls. The old design is
still measured below as alternative 2.
Tests
mutually_exclusive_timers_starvationintest_multi_threaded_executor.cpp.Each of the two timers must get at least 5 of 20 executions. Fails on rolling without the
change (20 and 0), passes with it (10 and 10). On lyrical, kilted and jazzy, which do
not have Fix MultiThreadedExecutor busy wait on busy callback groups #3395 yet, it is 20 and 0 without and 13 and 7 with the change.
every core.
test_executors_busy_waitingfrom Fix MultiThreadedExecutor busy wait on busy callback groups #3395 passed 20 of 20 runs.rclbuilt from its rolling branch, and with the same change on lyrical, kilted and jazzy.
Measurements
All numbers are from one machine, each distro in its own container with the released
binaries and Fast DDS, and
rclcppbuilt from source without and with the change (onrolling also
rcl).Rolling is measured on 917c124, which contains #3395. Lyrical, kilted, jazzy and humble
are measured on their current heads, which do not contain #3395 yet. I will repeat them
when its backports are merged.
1. Does the starved entity get executed, and how long does it wait
One mutually exclusive group, MultiThreadedExecutor with 2 threads. In the group there is
a fast timer (5 ms period, 20 ms callback, so it is always ready again when it returns) and
a victim. The victim is either a subscription that gets a message every 100 ms or a 100 ms
timer. 5 runs of 30 s per row, so 1505 messages or timer periods. "Wait" is the time from
the publication of the message (or the moment the timer is due) until the victim's callback
starts. With a 20 ms callback in the same group, 20 ms is the best possible maximum.
humble
jazzy
kilted
lyrical
rolling
Compare the rows inside one distro. The humble rows are from the humble version of the
change (#3417), and humble has no EventsCBGExecutor.
lyrical, about one second late, and nothing on humble and rolling. On rolling it got 108
of 1505 before Fix MultiThreadedExecutor busy wait on busy callback groups #3395 and gets none since.
EventsCBG row: one run had 302 timer executions.)
is the one that is picked first when the group becomes free. With the change the two take
turns (see "What it does not do"), and the victim timer waits longer than before.
distros (37 ms instead of 28 ms). The longest wait is the same.
2. Cost when nothing is blocked (
benchmark_executor)benchmark_executoris the executor benchmark that is already inrclcpp/test/benchmark. Its scenarios have no entity that is blocked by its callbackgroup, so the list stays empty. It shows what the change costs a system that does not have
the problem. Each scenario reports the time of one iteration (for example one
spin_someover 10 ready subscriptions) and the number of heap allocations in it.
The benchmark only gives absolute times, so it needs a baseline from the same machine. One
round runs rclcpp without the change and with the change directly after each other, with
10 repetitions per scenario. I ran three rounds with a different order in each, because the
machine state can differ between two runs.
How to read the tables:
rounds.
baseline of the same round: the mean of the three rounds and, in brackets, the smallest
and the largest single round. A negative value means it was faster.
The tables have the scenarios of the MultiThreadedExecutor. Humble, jazzy and kilted have
fewer scenarios in their version of the benchmark.
humble
spin_someadd_noderemove_nodespin_node_until_future_completejazzy
spin_someadd_noderemove_nodespin_node_until_future_completekilted
spin_someadd_noderemove_nodespin_node_until_future_completelyrical
spin_somespin_some, 10 callback groupswait_for_workwait_for_work, rebuildspinadd_noderemove_nodespin_node_until_future_completerolling
spin_somespin_some, 10 callback groupswait_for_workwait_for_work, rebuildspinadd_noderemove_nodespin_node_until_future_completeadd_nodedoes not go through the changed code and moves between -1.7 % and +4.0 % insingle rounds. That is the noise of this setup.
spin_node_until_future_completeonhumble is outside of it in the other direction (1 to 7 % faster), I have no explanation
for that.
spin_someis the scenario that goes through the changed code most often. It is between-0.5 % and +2.6 % in the mean, and the single rounds go from -2.3 % to +4.9 %. So a cost
of up to about 3 % per
spin_someis possible, but it is not clearly above the noise.3. Autoware reference system
reference-system (de8f3bc), the
Autoware model. It has no long callback in a shared group, so it does not show the
starvation. It shows whether the change hurts a normal system. The number is the latency of
the hot path (lidar inputs to object collision estimator). 5 runs of 10 minutes per row.
"Latency mean" is the mean of the five run means.
humble
jazzy
kilted
lyrical
rolling
No message was dropped in any run. With the change the mean latency differs by at most
0.13 ms from before, in both directions, and the runs of one row differ by more than that
among themselves. The reference system shows no cost.
4. ABI
abidiffbetweenlibrclcpp.sobuilt without and with the change:ExecutorImplementationstd::shared_ptrconstructor instantiationNo type, size, virtual table or existing function changes.
What it does not do
The entities of a group do not strictly take turns. The executor can only put an
entity into the list when it sees it ready. It does not always see it:
leaves out the groups that are in use at that moment.
running callback is done. An entity that becomes ready in this time is not noticed and
not put into the list.
empty. For this one pick the order of the wait set decides, as today. The entity that
loses is now in the list and is served next.
So an entity can be overtaken by a sibling, but only a bounded number of times, where
today the number is unbounded. In the test on lyrical, kilted and jazzy the execution order
of the two timers is
which gives 13 and 7 of 20 executions instead of 20 and 0. On rolling the two timers of the
test alternate (10 and 10), but the workload of measurement 1 shows the same effect there:
the victim waits for at most three callbacks of the other entity (60 ms with 20 ms
callbacks). One consequence is visible in the tables of measurement 1: an entity that today
is always picked first now waits longer than before (victim timer on jazzy and later, 24 ms
instead of 10 ms mean). Alternative 1 below removes this at a cost.
It does not change the polling on lyrical, kilted and jazzy. There the waiting thread
polls while a ready entity is blocked by its callback group:
rcl_waitreturns at once,the entity is skipped, and the thread waits again. #3395 fixed this on rolling, and it
stays fixed with this PR on top (see "Polling while an entity is blocked" below). On the
other distros this PR reduces the polling for a starved subscription, but it only goes away
with the backport of #3395.
Alternatives
I have two other variants of the fix ready. Both are built and measured, and I can switch
this PR to either of them if you prefer.
Alternative 1: this PR plus a turn rule. One more commit on top of this PR
(branch
mte-fix-alternation/rollingon my fork, +35 -7 lines; humble needs its ownversion, +108 -1). The executor also remembers for every
mutually exclusive group which entity ran last. That entity is not taken directly. It goes
through the list, so its ready siblings are served first. This closes the gap described in
"What it does not do": the entities of a group take turns strictly. On rolling it has one
more condition because of #3395: while a rebuild of the wait set is pending, the entity
that ran last is not taken from the list, so that the other entities of its group can show
up first.
Alternative 2: the design of the paper. This is what the PR contained before, brought
up to date. Entities of a group that is in use are taken out of the wait set and kept in a
queue. When a callback finishes, the worker wakes the waiting thread under a second mutex
and the queue is served first.
What they change in the code
wait_for_workgets a parameter, new functionsExecutorgets new membersAll three exist on all five distros and pass their starvation test and the rclcpp test
suite there.
Starvation and waiting time
The split of the 20 executions between the two timers of the new test:
More than two entities in one group, on rolling: three or four timers (5 ms period, 20 ms
callback) in one mutually exclusive group, and three such timers plus a 100 ms timer (V).
Order of the executions, the same in 3 runs of 60 executions each:
3333...321321...321321...4444...43214321...43214321...33333V3333V...321321V321V321321V...321321V321V321321V...The entities take turns in a fixed round, and V runs every time it is due, after at most
one round. I only ran this on rolling and can run it on the other distros if you want.
The workload of measurement 1 with all variants:
humble
jazzy
kilted
lyrical
rolling
with the EventsCBGExecutor (one message of 1505 on lyrical waited 29 ms).
and lyrical the subscription waits longer (up to 80 ms).
Cost when nothing is blocked
benchmark_executoras in measurement 2, three rounds with all variants in every round.Time per iteration compared with rclcpp without any change, mean of the three rounds and
smallest and largest round:
humble
spin_someadd_noderemove_nodespin_node_until_future_completejazzy
spin_someadd_noderemove_nodespin_node_until_future_completekilted
spin_someadd_noderemove_nodespin_node_until_future_completelyrical
spin_somespin_some, 10 callback groupswait_for_workwait_for_work, rebuildspinadd_noderemove_nodespin_node_until_future_completerolling
spin_somespin_some, 10 callback groupswait_for_workwait_for_work, rebuildspinadd_noderemove_nodespin_node_until_future_completeHeap allocations per iteration:
humble
spin_someadd_noderemove_nodespin_node_until_future_completejazzy
spin_someadd_noderemove_nodespin_node_until_future_completekilted
spin_someadd_noderemove_nodespin_node_until_future_completelyrical
spin_somespin_some, 10 callback groupswait_for_workwait_for_work, rebuildadd_noderemove_nodespin_node_until_future_completerolling
spin_somespin_some, 10 callback groupswait_for_workwait_for_work, rebuildadd_noderemove_nodespin_node_until_future_completeThis PR and the paper design are within the noise or a few percent above it, and neither
adds an allocation.
The turn rule costs something also when nothing is blocked. An entity that ran last in
its group is put into the list and taken from it again, and every list entry is one
allocation.
spin_somegets 4 to 5 % slower on lyrical and rolling, 28 to 30 % with 10callback groups, 22 to 23 % on jazzy and kilted and 40 % on humble. On jazzy and kilted
the benchmark has 10 nodes with one subscription each, so every subscription is alone in
its group, as in the scenario with 10 callback groups. The humble version does not
allocate, there the time goes into searching the ready lists.
The turn rule is in
get_next_ready_executable, so the SingleThreadedExecutor pays forit as well, although it cannot have the problem. On rolling:
spin_somespin_some, 10 callback groupsAutoware reference system
As in measurement 3, 5 runs of 10 minutes per row:
humble
jazzy
kilted
lyrical
rolling
No message was dropped in any run. All variants are within 0.6 ms of rclcpp without any
change, in both directions. None of the three shows a cost in this system, also not the
turn rule: its cost is a few microseconds per
spin_some, and the callbacks here takemilliseconds.
Polling while an entity is blocked
Calls of
rcl_waitin the workload of measurement 1 in 30 s, 5 runs per row. A thread thatonly wakes up for the timers and the messages of this workload needs a few thousand calls.
humble
rcl_waitcalls in 30 s, meanjazzy
rcl_waitcalls in 30 s, meankilted
rcl_waitcalls in 30 s, meanlyrical
rcl_waitcalls in 30 s, meanrolling (with #3395)
rcl_waitcalls in 30 s, meanis starved. This PR reduces that to about a quarter, the paper design a little less, and
the turn rule to about 0.2 million. In the timer case all stay at about 0.19 million
(paper design 0.14 million).
Summary
spin_somewhen nothing is blockedThis PR has the fix without the turn rule. It is the smallest change, has no clear cost,
does not affect the other executors and can go into the released distros unchanged.
I would still like to add the turn rule (alternative 1). It is the only variant where the
victim waits for at most one callback of the other entity, as with the EventsCBGExecutor,
and where no entity waits longer than before the change. As it is now, it costs one list
entry for every execution of an entity that ran last in its group, also in the
SingleThreadedExecutor (about 0.45 µs per callback if every entity is alone in its group).
My plan is to change it so that only the MultiThreadedExecutor pays for it, measure it
again and propose it as a second commit or a follow-up PR. If you prefer to have it here
as it is, I can add the commit. Alternative 2 would be for rolling only.
Backports
Branches with the same change exist for lyrical, kilted and jazzy (identical lines). They
are based on the current heads, without #3395. The change touches the same lines as #3395,
so I would rebase these branches and repeat their measurements when the backports of #3395
are merged.
Humble has no
ExecutorImplementation, there the same idea is implemented in a memorystrategy class inside the library, without changing any class layout. That is #3417.