Skip to content

MultiThreadedExecutor can permanently drop a MutuallyExclusive callback group from the wait set when the rebuild trigger coincides with an rmw_wait timeout (rmw_fastrtps) #3240

Closed

Description

@atsushi421

Generated by Generative AI

The investigation and this report were done with the help of Claude Code (Claude Fable 5). Every code reference below was checked by hand against the linked sources.

Operating System:

Linux 6.5.0-44-generic #44~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC x86_64 GNU/Linux
(host; the tests ran inside an Ubuntu 24.04.1 container using the ros-jazzy binary packages)

ROS version or commit hash:

jazzy (rclcpp 28.1.18, rcl 9.2.9). The same code is present on rolling, see the permalinks below.

RMW implementation (if applicable):

rmw_fastrtps_cpp (rmw_fastrtps_shared_cpp 8.4.3)

RMW Configuration (if applicable):

Default, no XML profile.

Client library (if applicable):

rclcpp

'ros2 doctor --report' output

Not collected (the container only has the packages needed to build and run the tests).

Steps to reproduce issue

The failure is a race, so it needs a setup where a thread can be preempted for a while at an unlucky moment (a hosted CI runner with a few vCPUs, or a local perturbation as described below).

  1. Create a node with a wall timer in the node's default (MutuallyExclusive) callback group whose callback takes a few milliseconds. Do not create any other MutuallyExclusive callback group in the executor (put subscriptions in a Reentrant group, or have none).
  2. Spin it with rclcpp::executors::MultiThreadedExecutor (2 or more threads) constructed with a finite next_exec_timeout (we used 25 to 400 ms). See "Preconditions" below for why the default -1 does not hit this.
  3. Run the process repeatedly on a 4 CPU cpuset while a bursty SCHED_FIFO hog steals one random CPU for 30 to 300 ms at a time (timeout 0.<n>s taskset -c <cpu> chrt -f 50 yes >/dev/null, then sleep 100 to 500 ms, in a loop). This mimics vCPU steal on hosted runners. Without the hog we could not reproduce it in 6,000+ runs; with it, 3 out of 800 runs failed.

Expected behavior

A timer (or any entity) in a MutuallyExclusive callback group keeps being serviced by MultiThreadedExecutor for as long as the node lives.

Actual behavior

Occasionally the timer fires a few times and then never again for the rest of the process lifetime. The other entities of the same callback group stop being serviced as well.

Mechanism (thread A is inside wait_for_work(), thread B executes a callback of the MutuallyExclusive group G):

  1. build_entities_collection() skips callback groups whose can_be_taken_from is false (rolling:
    if (group_ptr->can_be_taken_from().load()) {
    , jazzy:
    if (group_ptr->can_be_taken_from().load()) {
    ). While thread B executes a callback of G, can_be_taken_from of G is false, so a collect_entities() that thread A runs at that moment builds a wait set without any entity of G (for the node's default group that is every timer of the node).
  2. Those entities return only with the next rebuild. MultiThreadedExecutor::run() requests it by calling interrupt_guard_condition_->trigger() after each MutuallyExclusive callback (rolling:
    execute_any_executable(any_exec);
    if (any_exec.callback_group &&
    any_exec.callback_group->type() == CallbackGroupType::MutuallyExclusive)
    {
    try {
    interrupt_guard_condition_->trigger();
    } catch (const rclcpp::exceptions::RCLError & ex) {
    throw std::runtime_error(
    std::string(
    "Failed to trigger guard condition on callback group change: ") + ex.what());
    }
    }
    , jazzy:
    execute_any_executable(any_exec);
    if (any_exec.callback_group &&
    any_exec.callback_group->type() == CallbackGroupType::MutuallyExclusive)
    {
    try {
    interrupt_guard_condition_->trigger();
    } catch (const rclcpp::exceptions::RCLError & ex) {
    throw std::runtime_error(
    std::string(
    "Failed to trigger guard condition on callback group change: ") + ex.what());
    }
    }
    ). The trigger sets entities_need_rebuild_ only indirectly: it has to be reported as ready by rmw_wait() and then executed through ExecutorNotifyWaitable.
  3. rmw_wait() in rmw_fastrtps_shared_cpp calls set_trigger_value(false) on every guard condition after the DDS wait returns, even when it returns RMW_RET_TIMEOUT (rolling: https://github.com/ros2/rmw_fastrtps/blob/e5cc0e6422b5311691228953cea9ace87fc109e3/rmw_fastrtps_shared_cpp/src/rmw_wait.cpp#L328-L339, jazzy: https://github.com/ros2/rmw_fastrtps/blob/54d2240637f18e3ec710bb73c51c79abaf91ece4/rmw_fastrtps_shared_cpp/src/rmw_wait.cpp#L306-L317). A guard condition that was triggered keeps its slot in guard_conditions->guard_conditions[] (it is reported as ready), but the return value stays RMW_RET_TIMEOUT because wait_result is false.
  4. rcl_wait() turns RMW_RET_TIMEOUT into RCL_RET_TIMEOUT unless a timer is ready (rolling: https://github.com/ros2/rcl/blob/db3d1c8ea6eb22235494cfc03249155e35c47511/rcl/src/rcl/wait.c#L777-L779, jazzy: https://github.com/ros2/rcl/blob/22c0b957140035c675f0a061daaab5fd6b2080e7/rcl/src/rcl/wait.c#L744-L746), and rclcpp::WaitSet::wait() maps that to WaitResultKind::Timeout.
  5. Executor::wait_for_work() evaluates the notify waitable only when the result kind is Ready (rolling:
    if (this->wait_result_->kind() == WaitResultKind::Ready && current_notify_waitable_) {
    auto & rcl_wait_set = this->wait_result_->get_wait_set().get_rcl_wait_set();
    if (current_notify_waitable_->is_ready(rcl_wait_set)) {
    current_notify_waitable_->execute(current_notify_waitable_->take_data());
    }
    }
    , jazzy:
    if (this->wait_result_->kind() == WaitResultKind::Ready && current_notify_waitable_) {
    auto & rcl_wait_set = this->wait_result_->get_wait_set().get_rcl_wait_set();
    if (current_notify_waitable_->is_ready(rcl_wait_set)) {
    current_notify_waitable_->execute(current_notify_waitable_->take_data());
    }
    }
    ). On Timeout it never calls is_ready(), so the ready slot left by rcl_wait() is ignored and entities_need_rebuild_ stays false.

So if thread B's trigger() lands in the window between the DDS wait of thread A returning with a timeout and the guard condition loop at the end of rmw_wait(), the trigger value is cleared, the result is Timeout, and nothing records that a rebuild was requested. The next rmw_wait() does not see it either (has_triggered_condition() reads the already cleared value). The window is normally microseconds, but preemption of thread A inside it (vCPU steal, an RT task) stretches it to milliseconds. The state is also self reinforcing: once the timers of G are out of the wait set, rcl_wait() no longer shortens the timeout for them, so every subsequent wait ends in a genuine timeout, which is exactly the path on which the trigger can be lost.

Preconditions:

  • A finite next_exec_timeout. With the default -1, rcl_wait() derives its timeout only from the timers in the wait set, so whenever rmw_wait() returns RMW_RET_TIMEOUT a timer is due, rcl_wait() returns RCL_RET_OK, the result is Ready, and the trigger is picked up from the ready slot. Once the timers of G are gone the wait blocks indefinitely and the trigger wakes it. WaitResultKind::Timeout therefore does not occur in normal operation with -1 (remaining corner cases: a timer cancelled or reset by another thread during the wait, ROS time jumps). With a finite timeout, Timeout results are routine.
  • No other MutuallyExclusive callback group in the executor. Otherwise its callbacks keep producing triggers, one of them eventually causes a rebuild, and G recovers after a delay. With all other groups Reentrant (or with a single group), the loss is permanent.

Humble is not affected because its executor rebuilds the wait set on every wait_for_work().

Additional information

  • The behavior appeared with the wait set based executor rewrite (Utilize rclcpp::WaitSet as part of the executors #2142), which introduced both the incremental rebuild and the guard condition trigger after MutuallyExclusive callbacks.
  • Setting entities_need_rebuild_ explicitly in MultiThreadedExecutor::run() fixes it; the trigger is still needed to wake the waiting thread:
    if (any_exec.callback_group &&
      any_exec.callback_group->type() == CallbackGroupType::MutuallyExclusive)
    {
      entities_need_rebuild_.store(true);
      interrupt_guard_condition_->trigger();
    }
    With this change the failure did not reappear in 2,500+ runs under the same perturbation (0 failures, versus 3 out of 800 before). Alternatives would be to have wait_for_work() check current_notify_waitable_->is_ready() regardless of the result kind (rcl already leaves the ready slot in the wait set), or to have rmw_fastrtps not clear guard conditions on a timeout, but the first option keeps the rebuild request out of the rmw layer entirely.
  • The failure was observed and the fix verified in an executor built on rclcpp::Executor whose spin loop is the same as MultiThreadedExecutor::run(), including the trigger after MutuallyExclusive callbacks (fix(agnocastlib): request a wait set rebuild explicitly after MutuallyExclusive callbacks on Jazzy autowarefoundation/agnocast#1591). I have not yet written a standalone rclcpp-only reproducer; the mechanism above involves only rclcpp::Executor, MultiThreadedExecutor::run(), rcl_wait() and rmw_fastrtps.
  • Only rmw_fastrtps was checked. Whether other RMW implementations clear guard conditions on a timeout was not verified.
  • Related: Starvation in MultiThreadedExecutor #2645 and Multithreaded executor behavior #2360 describe starvation in MultiThreadedExecutor; this issue is a different failure (entities permanently missing from the wait set, not delayed), but PR Fix starvation in mutually exclusive callback groups of the MultiThreadedExecutor #2702 touches the same trigger.

Activity

  1. added a commit that references this issue on Aug 28, 2026
  2. fujitatomoya commented on Aug 31, 2026

    @fujitatomoya
    Collaborator

    @atsushi421 really appreciate the detailed issue report 💯 thanks!

    i suggest that having wait_for_work() check current_notify_waitable_->is_ready() regardless of result kind, this should fix it one layer down for any caller and any rmw, since rcl leaves the ready slot intact. what do you think?

    if you are willing to create PR, please open it against rolling 1st since that is our development branch. i am happy to review!

  3. atsushi421 commented on Aug 31, 2026

    @atsushi421
    Author

    @fujitatomoya Thanks for the suggestion! I agree, and I verified it works on current rolling:

    • rcl_wait() runs its array post-processing even on RMW_RET_TIMEOUT, so ready guard condition slots are left intact (wait.c#L748-L779)
    • rmw_fastrtps keeps the triggered slot non-NULL while returning RMW_RET_TIMEOUT, so checking current_notify_waitable_->is_ready() on timeout recovers exactly the trigger that was consumed

    I'll open a PR against rolling. Thanks for offering to review!

    By the way, I noticed a second, much narrower race in the rmw_fastrtps cleanup loop, which the rclcpp fix cannot cover (rmw_wait.cpp#L328-L337):

    1. get_trigger_value() returns false, so the slot is set to 0
    2. another thread calls trigger()
    3. the unconditional set_trigger_value(false) consumes the trigger

    Here the slot is already nulled, so no layer above can observe the trigger, and it is lost the same way as in this issue. A possible fix on the rmw_fastrtps side would be to reset the trigger only for conditions observed as triggered (so a late trigger survives into the next wait), and to derive the return code from the cleanup loop results (OK if any slot remains non-NULL). That would resolve this issue at its root for rmw_fastrtps, with the rclcpp-side check remaining valuable as an rmw-agnostic safety net. Do you think this is worth reporting to rmw_fastrtps?

  4. jmachowinski commented on Aug 31, 2026

    @jmachowinski
    Collaborator

    There should be an open pr about the fastrtps issue already.

  5. jmachowinski commented on Aug 31, 2026

    @jmachowinski
    Collaborator
  6. atsushi421 commented on Aug 31, 2026

    @atsushi421
    Author

    @jmachowinski Thanks for the pointer! I checked ros2/rmw_fastrtps#895 and it covers exactly both points on the rmw_fastrtps side:

    • resetting the trigger only for guard conditions observed as triggered closes the cleanup-loop race I described above
    • deriving the return code from the cleanup results fixes the case where a trigger is consumed but RMW_RET_TIMEOUT is returned, which is the rmw-side contributor to this issue

    So there is no need for a new rmw_fastrtps issue. Once #895 is merged, this issue is fixed at the root for rmw_fastrtps.

    That said, I believe #3244 is still needed independently of #895:

    • rcl_wait() deliberately runs its post-processing and leaves ready slots intact even when it returns RCL_RET_TIMEOUT, so "timed out, but an entity is ready" is a legitimate state at the rcl API level. Executor::wait_for_work() currently discards that information on a Timeout result, which is an rclcpp-layer problem regardless of which rmw is underneath.
    • Nothing in the rmw contract guarantees that an implementation never consumes a guard-condition trigger while returning timeout, so checking the notify waitable in rclcpp protects any rmw implementation, not just rmw_fastrtps. This is the "fix it one layer down for any caller and any rmw" approach @fujitatomoya suggested above.
  7. jmachowinski commented on Aug 31, 2026

    @jmachowinski
    Collaborator
    • rcl_wait() deliberately runs its post-processing and leaves ready slots intact even when it returns RCL_RET_TIMEOUT, so "timed out, but an entity is ready" is a legitimate state at the rcl API level. Executor::wait_for_work() currently discards that information on a Timeout result, which is an rclcpp-layer problem regardless of which rmw is underneath.

    • Nothing in the rmw contract guarantees that an implementation never consumes a guard-condition trigger while returning timeout, so checking the notify waitable in rclcpp protects any rmw implementation, not just rmw_fastrtps. This is the "fix it one layer down for any caller and any rmw" approach @fujitatomoya suggested above.

    That does not make any sense. If Timeout is returned nothing must be set. Anything not set must fire on the next invocation of rcl_wait.
    There must not be any races in this.

  8. atsushi421 commented on Aug 31, 2026

    @atsushi421
    Author

    @jmachowinski I agree, that is the contract we want: timeout means no entry is set, and an unreported trigger stays pending and fires on the next wait. rmw_fastrtps satisfies it with #895, and rmw_cyclonedds and rmw_zenoh already do (I checked both). But today it is neither documented nor universally implemented:

    • The rmw_wait() docs only say that entries for conditions that were not triggered are set to NULL (rmw.h#L2636-L2638); nothing says RMW_RET_TIMEOUT implies all entries are NULL, or that an unreported trigger must survive into the next wait.
    • rmw_connextdds violates it: detach() runs even on the timed-out path and consumes a guard condition trigger (trigger_value = false) while leaving its entry set (rmw_impl_waitset_std.cpp#L502-L515), then RMW_RET_TIMEOUT is returned even with active_conditions > 0 (#L585-L592). The consumed trigger never fires on the next rcl_wait(); it survives only in the entity arrays, which rcl preserves but rclcpp discards on Timeout.

    Two complementary fixes get us to "no races": enforce the contract in every rmw (merge #895, fix rmw_connextdds the same way, document the requirement in rmw_wait()), and/or the executor-side check in #3244, which costs one is_ready() call on the timeout path and covers this class of rmw bug. I am happy to file the rmw_connextdds issue and the rmw docs PR either way. Whether #3244 also lands is up to the maintainers; I would keep it as defense in depth, given two of four in-tree rmws have broken this contract and the failure mode (an executor that silently stops waking up) is very hard to debug.

  9. jmachowinski commented on Aug 31, 2026

    @jmachowinski
    Collaborator

    This feels like a copy and paste session with a chat bot.
    I will close this ticket now and urge you to read the AI policy.

  10. atsushi421 commented on Aug 31, 2026

    @atsushi421
    Author

    While it's true that I used AI, I did check the post myself before publishing it. However, I respect your opinion as the maintainer.

  11. fujitatomoya commented on Sep 1, 2026

    @fujitatomoya
    Collaborator

    @jmachowinski ah i lost #3240, thanks for pointing that out.

    btw, @atsushi421 is a human developer from https://github.com/autowarefoundation, and he's got nothing to do with #3244 (review)

  12. jmachowinski commented on Sep 18, 2026

    @jmachowinski
    Collaborator

    RMW fixes for fastDDS have been merge upstream, the latest source builds should not suffer from this problem any more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions