Skip to content

[humble] Fix starvation in mutually exclusive callback groups of the MultiThreadedExecutor - #3417

Open
HarunTeper wants to merge 1 commit into
ros2:humblefrom
HarunTeper:mte-fix/humble
Open

HarunTeper wants to merge 1 commit into
ros2:humblefrom
HarunTeper:mte-fix/humble

Conversation

@HarunTeper

Copy link
Copy Markdown

Humble version of #2702. Fixes #2360 and #2645 on humble.

Problem

If two entities share a mutually exclusive callback group and both are ready whenever the
group becomes free, the MultiThreadedExecutor always picks the one that comes first in the
wait set. The other one is never executed. Humble has no EventsCBGExecutor, so there is no
executor to switch to.

Change

Same idea as in #2702: the ready entities that were skipped because their callback group
was in use are remembered in the order in which they were skipped, and they are executed
first when their group is free again.

Humble has no ExecutorImplementation, so this is done in the memory strategy:

  • A class DefaultMemoryStrategy in memory_strategies.cpp, derived from
    AllocatorMemoryStrategy<>. create_default_strategy() returns it. It is local to the
    library.
  • One line in allocator_memory_strategy.hpp: the handle vectors become protected
    instead of private.
  • No change to Executor, MemoryStrategy or any other class layout, no new virtual
    function, no additional mutex. Executors that are given another memory strategy behave
    as before.

Tests

Measurements

Humble container with the released binaries and Fast DDS, rclcpp built from source
without and with the change. The method is described in #2702.

A fast timer (5 ms period, 20 ms callback) shares a group with a victim, which is either a
subscription that gets a message every 100 ms or a 100 ms timer. 5 runs of 30 s, "wait" is
the time until the victim's callback starts:

subscription: handled wait mean (ms) wait max (ms) timer: executed wait mean (ms) wait max (ms)
before 0 of 1505 0 of 1505
with the fix 1505 of 1505 29.9 40.1 1505 of 1505 30.2 40.1

benchmark_executor, MultiThreadedExecutor, 3 rounds, change against before:

scenario before (ns) with the fix
spin_some 48,835 +1.5 % (+0.3 to +2.4)
add_node 1,134 -0.5 % (-1.4 to +1.2)
remove_node 959 +0.7 % (-0.1 to +1.3)
spin_node_until_future_complete 958 -4.1 % (-6.6 to -1.4)

The number of allocations is the same in all four scenarios.

Autoware reference system, 5 runs of 10 minutes:

latency mean (ms) lowest and highest run mean max (ms) dropped messages
before 12.98 12.82 to 13.08 22.4 0
with the fix 12.96 12.88 to 13.04 25.3 0

ABI

abidiff between the two builds of librclcpp.so exits with "incompatible", although no
rclcpp type, size, virtual table or function differs. It lists 5 removed and 5 added
functions. The removed ones are the std::make_shared control block for
AllocatorMemoryStrategy<>, which the library no longer instantiates because it now
creates the derived class. The added ones are std::vector<...>::_M_erase for the five
handle types. Both are weak template instantiations that every binary using them emits
for itself, so a program that creates an AllocatorMemoryStrategy<> has its own copy.

What it does not do

As in #2702 the entities of a group do not strictly take turns. An entity that is blocked
goes before the others of its group, but an entity that just ran can run once more before
a sibling that became ready in the meantime (longest wait 40 ms instead of 20 ms in the
table above). #2702 describes a turn rule that closes this gap. A humble version of it
exists, I would propose it after the discussion there.

…adedExecutor

A handle that is ready while its mutually exclusive callback group is in
use is skipped and dropped before the next wait. Its callback group is
left out of the wait set while it is in use, so once the group is free
again all of its ready handles are collected at the same time and the
first one in the list is picked every time. With callbacks that take
longer than their period the other entities of the group are never
executed.

Let the default memory strategy keep the skipped handles and execute them
first, in the order in which they were skipped, when their callback group
can be taken from again. This is done in a class that is local to the
library, so the public classes keep their layout. Executors that are
given another memory strategy behave as before.

Signed-off-by: HarunTeper <harun.teper@tu-dortmund.de>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant