Skip to content

Fix starvation in mutually exclusive callback groups of the MultiThreadedExecutor - #2702

Open
HarunTeper wants to merge 1 commit into
ros2:rollingfrom
HarunTeper:rolling
Open

HarunTeper wants to merge 1 commit into
ros2:rollingfrom
HarunTeper:rolling

Conversation

@HarunTeper

@HarunTeper HarunTeper commented Dec 9, 2024 •

Copy link
Copy Markdown

Fixes #2360, #2645.

Problem

If two entities share a mutually exclusive callback group and both are ready whenever the
group becomes free, the MultiThreadedExecutor always picks the one that comes first in the
wait set. The other one is never executed. The added test shows this with two 5 ms timers
whose callbacks take 20 ms: one timer gets all 20 executions, the other none. This is the
case on humble, jazzy, kilted, lyrical and rolling.

Change

The executor remembers the ready entities it had to skip because their callback group was
in use, in the order in which it skipped them. When a group becomes free, an entity from
this list is executed before any other entity of the same group.

Until now this PR contained a larger change that followed the design of our paper: a
second mutex, a split of wait_for_work and new members in Executor. That is not needed
any more, because the executor now keeps the wait result between calls. The old design is
still measured below as alternative 2.

Tests

  • New test mutually_exclusive_timers_starvation in test_multi_threaded_executor.cpp.
    Each of the two timers must get at least 5 of 20 executions. Fails on rolling without the
    change (20 and 0), passes with it (10 and 10). On lyrical, kilted and jazzy, which do
    not have Fix MultiThreadedExecutor busy wait on busy callback groups #3395 yet, it is 20 and 0 without and 13 and 7 with the change.
  • The test passed 200 of 200 repetitions on rolling, and 100 of 100 with a busy loop on
    every core. test_executors_busy_waiting from Fix MultiThreadedExecutor busy wait on busy callback groups #3395 passed 20 of 20 runs.
  • The rclcpp test suite passes locally on rolling (124 of 124 test programs), with rcl
    built from its rolling branch, and with the same change on lyrical, kilted and jazzy.

Measurements

All numbers are from one machine, each distro in its own container with the released
binaries and Fast DDS, and rclcpp built from source without and with the change (on
rolling also rcl).

Rolling is measured on 917c124, which contains #3395. Lyrical, kilted, jazzy and humble
are measured on their current heads, which do not contain #3395 yet. I will repeat them
when its backports are merged.

1. Does the starved entity get executed, and how long does it wait

One mutually exclusive group, MultiThreadedExecutor with 2 threads. In the group there is
a fast timer (5 ms period, 20 ms callback, so it is always ready again when it returns) and
a victim. The victim is either a subscription that gets a message every 100 ms or a 100 ms
timer. 5 runs of 30 s per row, so 1505 messages or timer periods. "Wait" is the time from
the publication of the message (or the moment the timer is due) until the victim's callback
starts. With a 20 ms callback in the same group, 20 ms is the best possible maximum.

humble

subscription: handled wait mean (ms) wait max (ms) timer: executed wait mean (ms) wait max (ms)
MultiThreaded, before 0 of 1505 0 of 1505
MultiThreaded, with the fix 1505 of 1505 29.9 40.1 1505 of 1505 30.2 40.1

jazzy

subscription: handled wait mean (ms) wait max (ms) timer: executed wait mean (ms) wait max (ms)
MultiThreaded, before 60 of 1505 897.4 1000.0 1505 of 1505 10.0 20.1
MultiThreaded, with the fix 1505 of 1505 28.6 58.8 1505 of 1505 24.8 39.9
EventsCBG 1505 of 1505 9.4 20.1 1505 of 1505 10.5 20.1

kilted

subscription: handled wait mean (ms) wait max (ms) timer: executed wait mean (ms) wait max (ms)
MultiThreaded, before 91 of 1505 936.1 999.1 1505 of 1505 10.1 20.1
MultiThreaded, with the fix 1505 of 1505 28.8 60.0 1505 of 1505 24.4 39.9
EventsCBG 1505 of 1505 9.8 20.2 1505 of 1505 10.0 20.1

lyrical

subscription: handled wait mean (ms) wait max (ms) timer: executed wait mean (ms) wait max (ms)
MultiThreaded, before 98 of 1505 934.1 999.6 1505 of 1505 10.3 20.1
MultiThreaded, with the fix 1505 of 1505 28.2 59.0 1505 of 1505 23.8 39.9
EventsCBG 1505 of 1505 10.1 20.1 1506 of 1505 10.3 20.1

rolling

subscription: handled wait mean (ms) wait max (ms) timer: executed wait mean (ms) wait max (ms)
MultiThreaded, before 0 of 1505 1505 of 1505 10.1 20.1
MultiThreaded, with the fix 1505 of 1505 37.3 57.5 1505 of 1505 23.6 39.9
EventsCBG 1505 of 1505 9.6 20.4 1505 of 1505 10.5 20.1

Compare the rows inside one distro. The humble rows are from the humble version of the
change (#3417), and humble has no EventsCBGExecutor.

  • Before the change the subscription gets 4 to 7 % of its messages on jazzy, kilted and
    lyrical, about one second late, and nothing on humble and rolling. On rolling it got 108
    of 1505 before Fix MultiThreadedExecutor busy wait on busy callback groups #3395 and gets none since.
  • With the change every message and every timer period is handled. (1506 in the lyrical
    EventsCBG row: one run had 302 timer executions.)
  • The victim timer is not starved on jazzy and later before the change. In this program it
    is the one that is picked first when the group becomes free. With the change the two take
    turns (see "What it does not do"), and the victim timer waits longer than before.
  • On rolling the subscription waits longer in the mean with the change than on the other
    distros (37 ms instead of 28 ms). The longest wait is the same.

2. Cost when nothing is blocked (benchmark_executor)

benchmark_executor is the executor benchmark that is already in
rclcpp/test/benchmark. Its scenarios have no entity that is blocked by its callback
group, so the list stays empty. It shows what the change costs a system that does not have
the problem. Each scenario reports the time of one iteration (for example one spin_some
over 10 ready subscriptions) and the number of heap allocations in it.

The benchmark only gives absolute times, so it needs a baseline from the same machine. One
round runs rclcpp without the change and with the change directly after each other, with
10 repetitions per scenario. I ran three rounds with a different order in each, because the
machine state can differ between two runs.

How to read the tables:

  • "before" is the time per iteration without the change in nanoseconds, mean of the three
    rounds.
  • "with the fix" says how much slower rclcpp with the change was, compared with the
    baseline of the same round: the mean of the three rounds and, in brackets, the smallest
    and the largest single round. A negative value means it was faster.

The tables have the scenarios of the MultiThreadedExecutor. Humble, jazzy and kilted have
fewer scenarios in their version of the benchmark.

humble

scenario before (ns) with the fix
spin_some 48,835 +1.5 % (+0.3 to +2.4)
add_node 1,134 -0.5 % (-1.4 to +1.2)
remove_node 959 +0.7 % (-0.1 to +1.3)
spin_node_until_future_complete 958 -4.1 % (-6.6 to -1.4)

jazzy

scenario before (ns) with the fix
spin_some 26,714 +2.0 % (+1.7 to +2.4)
add_node 7,799 +0.4 % (-0.3 to +1.6)
remove_node 4,197 -0.8 % (-1.1 to -0.2)
spin_node_until_future_complete 10,729 -1.0 % (-3.1 to +1.3)

kilted

scenario before (ns) with the fix
spin_some 23,401 -0.5 % (-2.3 to +2.7)
add_node 6,728 +0.1 % (-1.7 to +1.7)
remove_node 3,786 +1.5 % (+0.3 to +2.5)
spin_node_until_future_complete 9,246 +0.2 % (-0.7 to +0.9)

lyrical

scenario before (ns) with the fix
spin_some 16,618 +2.6 % (-0.5 to +4.9)
spin_some, 10 callback groups 16,812 +3.2 % (+0.9 to +6.5)
wait_for_work 4,486 -0.1 % (-2.3 to +2.6)
wait_for_work, rebuild 8,769 -2.7 % (-3.7 to -2.0)
cascaded spin 74,065 +0.6 % (-3.8 to +4.9)
add_node 6,171 -0.0 % (-0.8 to +1.1)
remove_node 3,487 -0.3 % (-0.8 to +0.1)
spin_node_until_future_complete 8,401 -0.0 % (-1.0 to +1.5)

rolling

scenario before (ns) with the fix
spin_some 16,020 +1.2 % (-0.1 to +2.7)
spin_some, 10 callback groups 16,312 +0.7 % (-1.9 to +3.5)
wait_for_work 3,608 -0.8 % (-3.2 to +2.8)
wait_for_work, rebuild 7,333 +0.2 % (-1.5 to +3.2)
cascaded spin 69,885 +2.5 % (+0.6 to +3.9)
add_node 6,109 +1.5 % (+0.1 to +4.0)
remove_node 3,538 -0.7 % (-1.1 to -0.4)
spin_node_until_future_complete 8,402 -0.7 % (-3.5 to +3.3)
  • The number of allocations does not change in any scenario on any distro.
  • add_node does not go through the changed code and moves between -1.7 % and +4.0 % in
    single rounds. That is the noise of this setup. spin_node_until_future_complete on
    humble is outside of it in the other direction (1 to 7 % faster), I have no explanation
    for that.
  • spin_some is the scenario that goes through the changed code most often. It is between
    -0.5 % and +2.6 % in the mean, and the single rounds go from -2.3 % to +4.9 %. So a cost
    of up to about 3 % per spin_some is possible, but it is not clearly above the noise.

3. Autoware reference system

reference-system (de8f3bc), the
Autoware model. It has no long callback in a shared group, so it does not show the
starvation. It shows whether the change hurts a normal system. The number is the latency of
the hot path (lidar inputs to object collision estimator). 5 runs of 10 minutes per row.
"Latency mean" is the mean of the five run means.

humble

runs latency mean (ms) lowest and highest run mean min (ms) max (ms) dropped messages
MultiThreaded, before 5 12.98 12.82 to 13.08 6.9 22.4 0
MultiThreaded, with the fix 5 12.96 12.88 to 13.04 6.8 25.3 0

jazzy

runs latency mean (ms) lowest and highest run mean min (ms) max (ms) dropped messages
MultiThreaded, before 5 13.38 13.25 to 13.55 6.8 23.0 0
MultiThreaded, with the fix 5 13.35 13.10 to 13.55 6.8 21.7 0
EventsCBG 5 12.43 12.31 to 12.71 6.3 25.9 0

kilted

runs latency mean (ms) lowest and highest run mean min (ms) max (ms) dropped messages
MultiThreaded, before 5 13.22 13.08 to 13.37 6.8 23.3 0
MultiThreaded, with the fix 5 13.09 12.95 to 13.21 6.8 23.1 0
EventsCBG 5 12.63 12.58 to 12.72 6.4 26.2 0

lyrical

runs latency mean (ms) lowest and highest run mean min (ms) max (ms) dropped messages
MultiThreaded, before 5 13.08 13.06 to 13.11 6.7 21.8 0
MultiThreaded, with the fix 5 13.07 12.85 to 13.28 6.8 21.6 0
EventsCBG 5 12.66 12.38 to 12.83 6.3 26.2 0

rolling

runs latency mean (ms) lowest and highest run mean min (ms) max (ms) dropped messages
MultiThreaded, before 5 13.00 12.80 to 13.16 6.8 23.0 0
MultiThreaded, with the fix 5 12.99 12.94 to 13.02 6.8 21.9 0
EventsCBG 5 13.09 12.98 to 13.27 6.3 26.1 0

No message was dropped in any run. With the change the mean latency differs by at most
0.13 ms from before, in both directions, and the runs of one row differ by more than that
among themselves. The reference system shows no cost.

4. ABI

abidiff between librclcpp.so built without and with the change:

distro result what differs
rolling compatible 3 added functions of the private ExecutorImplementation
lyrical compatible same
kilted compatible same
jazzy compatible same, and one added std::shared_ptr constructor instantiation

No type, size, virtual table or existing function changes.

What it does not do

The entities of a group do not strictly take turns. The executor can only put an
entity into the list when it sees it ready. It does not always see it:

  • After every callback of a mutually exclusive group the wait set is rebuilt, and a rebuild
    leaves out the groups that are in use at that moment.
  • If that happens, the executor does not wait on the entities of this group until the
    running callback is done. An entity that becomes ready in this time is not noticed and
    not put into the list.
  • When the group is free again, all its ready entities show up together and the list is
    empty. For this one pick the order of the wait set decides, as today. The entity that
    loses is now in the list and is served next.

So an entity can be overtaken by a sibling, but only a bounded number of times, where
today the number is unbounded. In the test on lyrical, kilted and jazzy the execution order
of the two timers is

2 1 2 | 2 1 2 | 2 1 2 | ...

which gives 13 and 7 of 20 executions instead of 20 and 0. On rolling the two timers of the
test alternate (10 and 10), but the workload of measurement 1 shows the same effect there:
the victim waits for at most three callbacks of the other entity (60 ms with 20 ms
callbacks). One consequence is visible in the tables of measurement 1: an entity that today
is always picked first now waits longer than before (victim timer on jazzy and later, 24 ms
instead of 10 ms mean). Alternative 1 below removes this at a cost.

It does not change the polling on lyrical, kilted and jazzy. There the waiting thread
polls while a ready entity is blocked by its callback group: rcl_wait returns at once,
the entity is skipped, and the thread waits again. #3395 fixed this on rolling, and it
stays fixed with this PR on top (see "Polling while an entity is blocked" below). On the
other distros this PR reduces the polling for a starved subscription, but it only goes away
with the backport of #3395.

Alternatives

I have two other variants of the fix ready. Both are built and measured, and I can switch
this PR to either of them if you prefer.

Alternative 1: this PR plus a turn rule. One more commit on top of this PR
(branch mte-fix-alternation/rolling on my fork, +35 -7 lines; humble needs its own
version, +108 -1). The executor also remembers for every
mutually exclusive group which entity ran last. That entity is not taken directly. It goes
through the list, so its ready siblings are served first. This closes the gap described in
"What it does not do": the entities of a group take turns strictly. On rolling it has one
more condition because of #3395: while a rebuild of the wait set is pending, the entity
that ran last is not taken from the list, so that the other entities of its group can show
up first.

Alternative 2: the design of the paper. This is what the PR contained before, brought
up to date. Entities of a group that is in use are taken out of the wait set and kept in a
queue. When a callback finishes, the worker wakes the waiting thread under a second mutex
and the queue is served first.

What they change in the code

before this PR this PR + turn rule paper design
lines changed without tests (rolling, on top of #3395) +113 -9 +141 -9 +346 -22
lines changed without tests (jazzy, kilted, lyrical) +118 -6 +143 -6 +344 -27
lines changed without tests (humble) +219 -2 +326 -2 +308 -40
files changed without tests (jazzy and later) 1 1 4
public or protected API unchanged unchanged wait_for_work gets a parameter, new functions
ABI compatible compatible Executor gets new members
additional mutex no no yes
can be backported yes yes no

All three exist on all five distros and pass their starvation test and the rclcpp test
suite there.

Starvation and waiting time

The split of the 20 executions between the two timers of the new test:

executions of timer 1 / timer 2 before this PR this PR + turn rule paper design
lyrical 0 / 20 7 / 13 10 / 10 10 / 10
rolling (with #3395) 0 / 20 10 / 10 10 / 10 10 / 10

More than two entities in one group, on rolling: three or four timers (5 ms period, 20 ms
callback) in one mutually exclusive group, and three such timers plus a 100 ms timer (V).
Order of the executions, the same in 3 runs of 60 executions each:

before this PR this PR + turn rule
3 timers 3333... 321321... 321321...
4 timers 4444... 43214321... 43214321...
3 timers and V 33333V3333V... 321321V321V321321V... 321321V321V321321V...

The entities take turns in a fixed round, and V runs every time it is due, after at most
one round. I only ran this on rolling and can run it on the other distros if you want.

The workload of measurement 1 with all variants:

humble

subscription: handled wait mean (ms) wait max (ms) timer: executed wait mean (ms) wait max (ms)
before 0 of 1505 0 of 1505
this PR 1505 of 1505 29.9 40.1 1505 of 1505 30.2 40.1
this PR + turn rule 1505 of 1505 9.8 20.1 1505 of 1505 10.2 20.1
paper design 1505 of 1505 29.9 40.1 1505 of 1505 30.2 40.1

jazzy

subscription: handled wait mean (ms) wait max (ms) timer: executed wait mean (ms) wait max (ms)
before 60 of 1505 897.4 1000.0 1505 of 1505 10.0 20.1
this PR 1505 of 1505 28.6 58.8 1505 of 1505 24.8 39.9
this PR + turn rule 1505 of 1505 9.5 20.2 1505 of 1505 10.0 20.1
paper design 1505 of 1505 36.0 78.9 1505 of 1505 19.6 45.5
EventsCBG 1505 of 1505 9.4 20.1 1505 of 1505 10.5 20.1

kilted

subscription: handled wait mean (ms) wait max (ms) timer: executed wait mean (ms) wait max (ms)
before 91 of 1505 936.1 999.1 1505 of 1505 10.1 20.1
this PR 1505 of 1505 28.8 60.0 1505 of 1505 24.4 39.9
this PR + turn rule 1505 of 1505 9.6 20.7 1505 of 1505 10.0 20.1
paper design 1505 of 1505 36.2 79.7 1505 of 1505 19.8 51.3
EventsCBG 1505 of 1505 9.8 20.2 1505 of 1505 10.0 20.1

lyrical

subscription: handled wait mean (ms) wait max (ms) timer: executed wait mean (ms) wait max (ms)
before 98 of 1505 934.1 999.6 1505 of 1505 10.3 20.1
this PR 1505 of 1505 28.2 59.0 1505 of 1505 23.8 39.9
this PR + turn rule 1505 of 1505 9.8 29.1 1505 of 1505 10.2 20.1
paper design 1505 of 1505 35.5 79.2 1505 of 1505 19.6 42.5
EventsCBG 1505 of 1505 10.1 20.1 1506 of 1505 10.3 20.1

rolling

subscription: handled wait mean (ms) wait max (ms) timer: executed wait mean (ms) wait max (ms)
before 0 of 1505 1505 of 1505 10.1 20.1
this PR 1505 of 1505 37.3 57.5 1505 of 1505 23.6 39.9
this PR + turn rule 1505 of 1505 9.8 20.4 1505 of 1505 10.0 20.0
paper design 1505 of 1505 39.0 57.5 1505 of 1505 23.7 39.9
EventsCBG 1505 of 1505 9.6 20.4 1505 of 1505 10.5 20.1
  • All three end the starvation on every distro.
  • With the turn rule the victim waits for at most one callback of the other entity, like
    with the EventsCBGExecutor (one message of 1505 on lyrical waited 29 ms).
  • With the paper design the victim waits about as long as with this PR. On jazzy, kilted
    and lyrical the subscription waits longer (up to 80 ms).

Cost when nothing is blocked

benchmark_executor as in measurement 2, three rounds with all variants in every round.
Time per iteration compared with rclcpp without any change, mean of the three rounds and
smallest and largest round:

humble

scenario this PR this PR + turn rule paper design
spin_some +1.5 % (+0.3 to +2.4) +39.7 % (+37.7 to +41.3) +1.3 % (+0.4 to +3.0)
add_node -0.5 % (-1.4 to +1.2) -0.1 % (-2.4 to +1.4) +0.5 % (-0.3 to +1.5)
remove_node +0.7 % (-0.1 to +1.3) -0.1 % (-0.9 to +0.5) -0.8 % (-1.4 to +0.3)
spin_node_until_future_complete -4.1 % (-6.6 to -1.4) -3.4 % (-5.0 to -2.6) -3.5 % (-5.9 to -1.6)

jazzy

scenario this PR this PR + turn rule paper design
spin_some +2.0 % (+1.7 to +2.4) +21.6 % (+19.7 to +24.7) +2.8 % (+2.5 to +3.2)
add_node +0.4 % (-0.3 to +1.6) +0.6 % (-0.7 to +1.4) +0.1 % (-1.1 to +0.6)
remove_node -0.8 % (-1.1 to -0.2) +0.8 % (-1.3 to +2.4) -0.1 % (-1.3 to +0.6)
spin_node_until_future_complete -1.0 % (-3.1 to +1.3) +0.2 % (-2.1 to +1.6) +0.3 % (-0.2 to +1.0)

kilted

scenario this PR this PR + turn rule paper design
spin_some -0.5 % (-2.3 to +2.7) +23.0 % (+21.7 to +25.3) -1.5 % (-4.2 to +2.5)
add_node +0.1 % (-1.7 to +1.7) +0.7 % (-1.7 to +4.4) +0.4 % (-2.4 to +2.0)
remove_node +1.5 % (+0.3 to +2.5) +0.6 % (-0.8 to +2.4) -0.3 % (-1.1 to +0.7)
spin_node_until_future_complete +0.2 % (-0.7 to +0.9) +0.9 % (+0.1 to +2.1) +0.3 % (-1.0 to +1.5)

lyrical

scenario this PR this PR + turn rule paper design
spin_some +2.6 % (-0.5 to +4.9) +5.2 % (+3.3 to +6.9) +1.8 % (+1.1 to +2.3)
spin_some, 10 callback groups +3.2 % (+0.9 to +6.5) +29.5 % (+28.3 to +30.8) +1.5 % (+0.9 to +2.1)
wait_for_work -0.1 % (-2.3 to +2.6) +3.5 % (+3.2 to +4.0) -1.9 % (-2.9 to -0.7)
wait_for_work, rebuild -2.7 % (-3.7 to -2.0) -1.8 % (-2.7 to -0.6) -1.7 % (-3.3 to +0.4)
cascaded spin +0.6 % (-3.8 to +4.9) +0.8 % (-1.6 to +3.1) -0.6 % (-2.1 to +2.1)
add_node -0.0 % (-0.8 to +1.1) -0.2 % (-1.1 to +0.2) -0.7 % (-2.1 to +0.2)
remove_node -0.3 % (-0.8 to +0.1) +0.0 % (-1.1 to +0.7) -0.5 % (-1.1 to -0.0)
spin_node_until_future_complete -0.0 % (-1.0 to +1.5) -0.3 % (-0.5 to -0.0) -0.5 % (-1.9 to +0.4)

rolling

scenario this PR this PR + turn rule paper design
spin_some +1.2 % (-0.1 to +2.7) +3.8 % (+1.5 to +5.8) +1.0 % (-2.5 to +5.9)
spin_some, 10 callback groups +0.7 % (-1.9 to +3.5) +27.6 % (+25.4 to +29.4) +2.1 % (-1.1 to +7.1)
wait_for_work -0.8 % (-3.2 to +2.8) +3.8 % (+1.8 to +5.9) +2.0 % (-0.9 to +4.7)
wait_for_work, rebuild +0.2 % (-1.5 to +3.2) +0.1 % (-2.2 to +1.8) -0.3 % (-3.2 to +3.3)
cascaded spin +2.5 % (+0.6 to +3.9) +0.8 % (+0.2 to +1.7) +2.7 % (-0.1 to +6.0)
add_node +1.5 % (+0.1 to +4.0) +1.7 % (+0.2 to +3.9) +1.1 % (-0.7 to +2.7)
remove_node -0.7 % (-1.1 to -0.4) +0.5 % (-0.3 to +1.7) +0.9 % (-1.0 to +2.1)
spin_node_until_future_complete -0.7 % (-3.5 to +3.3) +2.1 % (+0.0 to +4.8) +1.2 % (-1.1 to +3.7)

Heap allocations per iteration:

humble

scenario before this PR this PR + turn rule paper design
spin_some 83 83 83 83
add_node 5 5 5 5
remove_node 5 5 5 5
spin_node_until_future_complete 5 5 5 5

jazzy

scenario before this PR this PR + turn rule paper design
spin_some 47 47 57 47
add_node 57 57 57 57
remove_node 57 57 57 57
spin_node_until_future_complete 57 57 57 57

kilted

scenario before this PR this PR + turn rule paper design
spin_some 40 40 50 40
add_node 50 50 50 50
remove_node 50 50 50 50
spin_node_until_future_complete 50 50 50 50

lyrical

scenario before this PR this PR + turn rule paper design
spin_some 40 40 40 40
spin_some, 10 callback groups 40 40 50 40
wait_for_work 4 4 5 4
wait_for_work, rebuild 327 327 328 327
add_node 50 50 50 50
remove_node 50 50 50 50
spin_node_until_future_complete 50 50 50 50

rolling

scenario before this PR this PR + turn rule paper design
spin_some 40 40 40 40
spin_some, 10 callback groups 40 40 50 40
wait_for_work 4 4 5 4
wait_for_work, rebuild 327 327 328 327
add_node 50 50 50 50
remove_node 50 50 50 50
spin_node_until_future_complete 50 50 50 50
  • This PR and the paper design are within the noise or a few percent above it, and neither
    adds an allocation.

  • The turn rule costs something also when nothing is blocked. An entity that ran last in
    its group is put into the list and taken from it again, and every list entry is one
    allocation. spin_some gets 4 to 5 % slower on lyrical and rolling, 28 to 30 % with 10
    callback groups, 22 to 23 % on jazzy and kilted and 40 % on humble. On jazzy and kilted
    the benchmark has 10 nodes with one subscription each, so every subscription is alone in
    its group, as in the scenario with 10 callback groups. The humble version does not
    allocate, there the time goes into searching the ready lists.

  • The turn rule is in get_next_ready_executable, so the SingleThreadedExecutor pays for
    it as well, although it cannot have the problem. On rolling:

    scenario this PR this PR + turn rule
    SingleThreaded spin_some -0.1 % (-0.6 to +0.6) +3.4 % (+1.1 to +5.4)
    SingleThreaded spin_some, 10 callback groups -0.3 % (-3.2 to +3.7) +26.5 % (+23.8 to +31.1), 10 allocations more

Autoware reference system

As in measurement 3, 5 runs of 10 minutes per row:

humble

runs latency mean (ms) lowest and highest run mean min (ms) max (ms) dropped messages
before 5 12.98 12.82 to 13.08 6.9 22.4 0
this PR 5 12.96 12.88 to 13.04 6.8 25.3 0
this PR + turn rule 5 12.90 12.76 to 13.35 6.8 23.2 0
paper design 5 13.03 12.89 to 13.12 6.9 22.1 0

jazzy

runs latency mean (ms) lowest and highest run mean min (ms) max (ms) dropped messages
before 5 13.38 13.25 to 13.55 6.8 23.0 0
this PR 5 13.35 13.10 to 13.55 6.8 21.7 0
this PR + turn rule 5 12.83 12.21 to 13.12 6.8 22.8 0
paper design 5 13.56 13.45 to 13.72 6.9 21.7 0
EventsCBG 5 12.43 12.31 to 12.71 6.3 25.9 0

kilted

runs latency mean (ms) lowest and highest run mean min (ms) max (ms) dropped messages
before 5 13.22 13.08 to 13.37 6.8 23.3 0
this PR 5 13.09 12.95 to 13.21 6.8 23.1 0
this PR + turn rule 5 12.87 12.68 to 13.03 6.8 22.1 0
paper design 5 12.89 12.72 to 13.06 6.9 25.9 0
EventsCBG 5 12.63 12.58 to 12.72 6.4 26.2 0

lyrical

runs latency mean (ms) lowest and highest run mean min (ms) max (ms) dropped messages
before 5 13.08 13.06 to 13.11 6.7 21.8 0
this PR 5 13.07 12.85 to 13.28 6.8 21.6 0
this PR + turn rule 5 13.13 12.92 to 13.28 6.9 21.8 0
paper design 5 13.17 12.98 to 13.41 6.7 21.5 0
EventsCBG 5 12.66 12.38 to 12.83 6.3 26.2 0

rolling

runs latency mean (ms) lowest and highest run mean min (ms) max (ms) dropped messages
before 5 13.00 12.80 to 13.16 6.8 23.0 0
this PR 5 12.99 12.94 to 13.02 6.8 21.9 0
this PR + turn rule 5 12.89 12.41 to 13.16 7.2 22.1 0
paper design 5 12.98 12.84 to 13.10 6.8 22.8 0
EventsCBG 5 13.09 12.98 to 13.27 6.3 26.1 0

No message was dropped in any run. All variants are within 0.6 ms of rclcpp without any
change, in both directions. None of the three shows a cost in this system, also not the
turn rule: its cost is a few microseconds per spin_some, and the callbacks here take
milliseconds.

Polling while an entity is blocked

Calls of rcl_wait in the workload of measurement 1 in 30 s, 5 runs per row. A thread that
only wakes up for the timers and the messages of this workload needs a few thousand calls.

humble

subscription case: rcl_wait calls in 30 s, mean lowest to highest run timer case: mean lowest to highest run
before 3,633 3,633 to 3,633 3,016 3,016 to 3,016
this PR 3,913 3,904 to 3,918 3,018 3,018 to 3,018
this PR + turn rule 3,926 3,924 to 3,926 3,317 3,317 to 3,317
paper design 3,925 3,923 to 3,926 3,018 3,018 to 3,019

jazzy

subscription case: rcl_wait calls in 30 s, mean lowest to highest run timer case: mean lowest to highest run
before 11,456,479 11,132,847 to 11,717,547 190,915 188,699 to 192,261
this PR 2,682,800 2,398,856 to 2,833,664 190,551 189,100 to 191,334
this PR + turn rule 193,679 191,051 to 196,042 191,086 187,293 to 192,399
paper design 3,510,892 3,410,025 to 3,566,122 143,826 143,372 to 144,759

kilted

subscription case: rcl_wait calls in 30 s, mean lowest to highest run timer case: mean lowest to highest run
before 14,773,088 14,660,139 to 15,009,686 192,359 192,241 to 192,456
this PR 3,612,591 3,429,553 to 3,774,146 191,925 191,236 to 192,486
this PR + turn rule 235,612 193,301 to 380,793 191,915 191,502 to 192,203
paper design 4,335,801 4,042,251 to 4,468,365 143,668 142,820 to 144,713

lyrical

subscription case: rcl_wait calls in 30 s, mean lowest to highest run timer case: mean lowest to highest run
before 15,210,113 14,819,056 to 15,438,971 192,275 192,055 to 192,573
this PR 3,574,526 3,358,329 to 3,825,140 191,442 191,114 to 191,924
this PR + turn rule 213,818 192,622 to 270,453 192,072 191,957 to 192,305
paper design 4,268,703 4,008,077 to 4,513,208 144,057 142,914 to 144,722

rolling (with #3395)

subscription case: rcl_wait calls in 30 s, mean lowest to highest run timer case: mean lowest to highest run
before 3,319 3,318 to 3,320 4,515 4,513 to 4,517
this PR 3,563 3,541 to 3,582 3,314 3,313 to 3,315
this PR + turn rule 4,827 4,814 to 4,837 4,512 4,510 to 4,513
paper design 3,634 3,629 to 3,639 3,312 3,311 to 3,313
  • Humble does not poll, with or without the change.
  • Jazzy, kilted and lyrical poll today, 11 to 15 million times in 30 s when the subscription
    is starved. This PR reduces that to about a quarter, the paper design a little less, and
    the turn rule to about 0.2 million. In the timer case all stay at about 0.19 million
    (paper design 0.14 million).
  • On rolling the polling is gone since Fix MultiThreadedExecutor busy wait on busy callback groups #3395, and none of the three brings it back.

Summary

this PR this PR + turn rule paper design
ends the starvation yes yes yes
longest wait of the victim (20 ms callbacks) 40 to 60 ms 20 to 29 ms 40 to 80 ms
cost per spin_some when nothing is blocked -0.5 to +2.6 % +4 to +40 %, up to 10 allocations more -1.5 to +2.8 %
cost in the reference system none none none
size (jazzy and later) +113 to +118 lines +141 to +143 lines +344 to +346 lines
backport yes yes no
brings back the polling that #3395 removed no no no

This PR has the fix without the turn rule. It is the smallest change, has no clear cost,
does not affect the other executors and can go into the released distros unchanged.

I would still like to add the turn rule (alternative 1). It is the only variant where the
victim waits for at most one callback of the other entity, as with the EventsCBGExecutor,
and where no entity waits longer than before the change. As it is now, it costs one list
entry for every execution of an entity that ran last in its group, also in the
SingleThreadedExecutor (about 0.45 µs per callback if every entity is alone in its group).
My plan is to change it so that only the MultiThreadedExecutor pays for it, measure it
again and propose it as a second commit or a follow-up PR. If you prefer to have it here
as it is, I can add the commit. Alternative 2 would be for rolling only.

Backports

Branches with the same change exist for lyrical, kilted and jazzy (identical lines). They
are based on the current heads, without #3395. The change touches the same lines as #3395,
so I would rebase these branches and repeat their measurements when the backports of #3395
are merged.
Humble has no ExecutorImplementation, there the same idea is implemented in a memory
strategy class inside the library, without changing any class layout. That is #3417.

@HarunTeper HarunTeper changed the title Added test for starvation in the multi-threaded executor Multi-threaded Executor starvation fix Dec 9, 2024
@jmachowinski

Copy link
Copy Markdown
Collaborator

The proposed changes, as well as the paper, don't make sense to me. But this might be a wording vs implementation issue. Lets wait for the actual implementation...
@HarunTeper you might want to attend to the next client workgroup meeting, to discuss this in person.

@alsora

alsora commented Dec 12, 2024 •

Copy link
Copy Markdown
Collaborator

Hi @HarunTeper,
I agree that this topic seems complex enough that it would be better to have an in-person conversation.

The next Client Library WG meeting will happen Friday 12/20/2024 at 8AM PST.
See here some details, and I will post a reminder next week https://discourse.ros.org/t/next-client-library-wg-meeting-friday-6th-december-2024-8am-pt/40954

@sloretz sloretz added the more-information-needed Further information is required label Dec 20, 2024
@HarunTeper

Copy link
Copy Markdown
Author

Hi @HarunTeper, I agree that this topic seems complex enough that it would be better to have an in-person conversation.

The next Client Library WG meeting will happen Friday 12/20/2024 at 8AM PST. See here some details, and I will post a reminder next week https://discourse.ros.org/t/next-client-library-wg-meeting-friday-6th-december-2024-8am-pt/40954

I would likely join the next meeting, and work on the pull request until then. Sorry for not making it to the last one.

@HarunTeper

Copy link
Copy Markdown
Author

I have now uploaded my fix for humble to a custom repository:
It provides a devcontainer and VSCode tasks to build and run the test that shows starvation. (After switching branches, you should delete the build log and install folders before rebuilding)

https://github.com/HarunTeper/rclcpp_humble_multithreaded_executor

You can see the changes in the branch called "fix".

I am currently working on applying the same changes to the current version of the multi-threaded executor. However, since Humble, the code of the executor itself, specifically the wait set and wait_for_work functions are very different.

I will try to implement the version that I propose, however, it will be different from my previous solution. For example, I think the current version has threads busy waiting for tasks to become ready if all tasks are blocked but not all threads are currently executing tasks.

If there is another meeting, I will join that one to talk about the executor. In the meantime, I will work on the fix.

@jmachowinski

Copy link
Copy Markdown
Collaborator

I had a look at the 'fix' implementation. As far as I can see, this comes down to:

  • If there are unprocessed entities of a callback group, don't add any entities of the callback group to the waitset on repoll
  • Make unprocessed entities from the last poll 'persistent' / don't clear unprocessed entities of callback groups in execution on repoll
  • Use some mutexes to avoid races between unmarking of callback group execution and unprocessed entities updates

While I think, this fix works, the implementation is awkward, in the sense that the code flow is extremely hard to follow. To be fair, the executor code flow isn't great in the first place...
I would also say, that the implementations suffers from unneeded interruptions of the waitset polling and context switches. Also the be fair here, the humble and current implementation both suffer from the same problem.

I would suggest a different approach to the problem:
Create a deque of ready events per callback group.
Have two lists of callback groups

  • Idle
  • Processing

Poll for events:
Only poll for callback groups from the idle list
After each poll:

  • add the events to the corresponding deque
  • Move callback groups with ready events into the processing list

Worker threads:

  • if processing is empty
    • poll / block if someone else is already polling
    • Wakup workers corresponding to the size of the processing list
    • continue
  • remove first entry of processing
  • process and remove the first ready entity of the removed callback group
  • check if callback group has further pending events
    • if not, add to idle, wakeup poll thread
    • if there are pending events, readd to processing

This is basically the logic I use in the EventsCBGExecutor.

@ros-discourse

Copy link
Copy Markdown

This pull request has been mentioned on ROS Discourse. There might be relevant details there:

https://discourse.ros.org/t/multi-threaded-executor-task-starvation-fix/42190/1

@HarunTeper

Copy link
Copy Markdown
Author

Hello,

I have added some changes to the structure of the executor.cpp. Mainly I have split the wait_for_work function into prepare_work and wait_for_work_simple (placeholder until wait_for_work can be fully replaced).

I am now working on the following changes:

  1. In the prepare_work function, I need to determine which callbacks are still in the wait_result_ data structure. These callbacks are blocked, as their callback group flag should have can_be_taken_from set to false. There needs to be a data structure which stores them to be added to the wait_result later.

  2. After the wait_result_.emplace(wait_set_.wait()) function, the previously saved callbacks need to be added back to the wait_result_

If anyone has some suggestions on how I should access wait_result_ and which data structures I should store, please let me know.

…adedExecutor

An entity that is ready while its mutually exclusive callback group is in
use is skipped. The wait result is dropped before the next wait, so once
the group is free again all of its ready entities are found at the same
time and the one that comes first in the wait set is picked every time.
With callbacks that take longer than their period the other entities of
the group are never executed.

Remember the skipped entities in the order in which they were skipped and
let them go first when their callback group can be taken from again.

Signed-off-by: HarunTeper <harun.teper@tu-dortmund.de>
@HarunTeper HarunTeper changed the title Multi-threaded Executor starvation fix Fix starvation in mutually exclusive callback groups of the MultiThreadedExecutor Nov 16, 2026
@HarunTeper

HarunTeper commented Nov 16, 2026 •

Copy link
Copy Markdown
Author

Sorry for the long silence on this. I have rewritten this PR completely: the branch was
force-pushed with a new, much smaller change and the description is new as well. The
earlier review comments refer to the old code, which is still available as tag
archive/2026-06/rolling-pr
on my fork.

@jmachowinski you wrote that the earlier implementation was hard to follow and suggested
per-group queues as in the EventsCBGExecutor. I agree with the first point. The new version
does not touch the control flow of the executor at all: the scan in
get_next_ready_executable records the entities it has to skip, and those are served
first once their group is free. It is about 110 lines in executor.cpp, no header change
and no new lock.

I did not go as far as the idle/processing lists. That would be a rewrite of how the
MultiThreadedExecutor polls, and the EventsCBGExecutor already exists for that. My aim here
is only that the default multi-threaded executor does not starve a callback, with a change
small enough to backport.

Two things I found on the way:

  • The busy waiting I mentioned in January 2025 (the waiting thread wakes up again and again
    while a ready entity is blocked by its group) is fixed on rolling by Fix MultiThreadedExecutor busy wait on busy callback groups #3395. This PR is
    rebased on it and keeps that behaviour. The older distros still have it until Fix MultiThreadedExecutor busy wait on busy callback groups #3395 is
    backported.
  • With the change in this branch the order inside a group is not strictly first come first
    served, see "What it does not do" in the description. The description lists two
    alternatives that I have measured as well: one more commit that makes the entities of a
    group take turns, and the larger design from the paper. The branch has the fix without
    them. I would like to add the first one later, once the SingleThreadedExecutor does not
    pay for it any more, see the summary in the description.

The EventsCBGExecutor passes the same test scenario on jazzy and later, I have included it
in the measurements for comparison.

@HarunTeper

Copy link
Copy Markdown
Author

Update on the turn rule (alternative 1 in the description). I wrote there that it also costs time in the SingleThreadedExecutor and that I wanted to change that. I have done it now.

The entity that ran last is now only remembered for a callback group after an entity of that group was blocked for the first time. Callback groups without contention do not use the rule, in any executor. It is one more commit on the turn rule (+20 -6 lines), branch mte-fix-alternation/rolling-c on my fork. Tests and linters pass.

benchmark_executor on rolling, three new rounds with all builds in each round, change against rolling without this PR:

scenario this PR + turn rule + turn rule, new version
SingleThreaded spin_some, 10 callback groups +2.8 % (+2.7 to +3.0) +26.6 % (+21.8 to +31.9) +1.3 % (-1.5 to +3.3)
MultiThreaded spin_some, 10 callback groups +1.4 % (-2.9 to +4.4) +25.4 % (+22.1 to +28.2) +0.5 % (-0.8 to +1.2)
SingleThreaded spin_some +0.8 % (-0.1 to +1.9) +4.3 % (+2.5 to +7.0) +4.8 % (+2.6 to +7.2)
MultiThreaded spin_some -0.3 % (-3.9 to +2.5) +4.6 % (+1.7 to +7.9) +2.4 % (+0.3 to +5.0)
allocations per spin_some, 10 callback groups (40 without this PR) 40 50 40

The cost with one entity per callback group and the extra allocations are gone. For the plain spin_some scenario I cannot say whether a small cost is left. I repeated it four more times per build and the three builds were not distinguishable, single runs went from -3.5 % to +6.6 % for each of them.

Waiting time of the starved entity (same setup as measurement 1, 5 runs of 30 s):

subscription: wait mean (ms) wait max (ms) timer: wait mean (ms) wait max (ms)
this PR 37.3 57.5 23.6 39.9
+ turn rule 9.8 20.4 10.0 20.0
+ turn rule, new version 9.8 40.1 10.0 20.0

The 40 ms is the first message of each run and only that one. The rule starts with the first blocking in a group, so the first round is not in turns yet. Polling and the reference system are the same as with the first version of the turn rule.

This PR still has only the fix. I can add the turn rule in this version as a second commit here or open a follow-up PR after this one, whichever you prefer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

more-information-needed Further information is required

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Starvation in MultiThreadedExecutor

6 participants