Skip to content

Make AgentManagerImpl.handleCommands resilient to listener exceptions - #13788

Open
Damans227 wants to merge 1 commit into
apache:mainfrom
Damans227:power-state-sync-npe-resilience
Open

Damans227 wants to merge 1 commit into
apache:mainfrom
Damans227:power-state-sync-npe-resilience

Conversation

@Damans227

Copy link
Copy Markdown
Collaborator

Description

If one listener in handleCommands throws, the whole loop stops and every listener after it never runs. This can starve pingBy() and cause fake ping timeouts.

This wraps each listener call in a try/catch, so one bad listener just logs a warning and the rest still run. Also fixed a trace log nearby that was logging the wrong class (the Pair wrapper instead of the listener).

Also added the missing hostId field to IdsPowerStateSelectSearch. Without it, VMs loaded through that path always showed a null host, even when they had one. Not a crash, just a wrong log message.

Types of changes

  • Bug fix

Feature/Enhancement Scale or Bug Severity

Bug Severity

  • Major

How Has This Been Tested?

Added tests: a throwing listener doesn't block the next one, and existing behavior with only successful listeners is unchanged. Also added tests for the null-hostId case, which had no coverage before.

@codecov

codecov Bot commented Aug 4, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 87.50000% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 19.91%. Comparing base (7ed0cec) to head (57a5ff8).

Files with missing lines Patch % Lines
.../main/java/com/cloud/vm/dao/VMInstanceDaoImpl.java 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff            @@
##               main   #13788   +/-   ##
=========================================
  Coverage     19.91%   19.91%           
- Complexity    20198    20203    +5     
=========================================
  Files          6373     6373           
  Lines        577230   577236    +6     
  Branches      70696    70696           
=========================================
+ Hits         114936   114967   +31     
+ Misses       449736   449703   -33     
- Partials      12558    12566    +8     
Flag Coverage Δ
uitests 3.71% <ø> (ø)
unittests 21.18% <87.50%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@DaanHoogland DaanHoogland added this to the 4.24.0 milestone Aug 5, 2026
@DaanHoogland DaanHoogland moved this from Backlog to Ready in CloudStack Testing Sep 1, 2026
@github-actions

Copy link
Copy Markdown

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

…steners and ensuring downstream listeners are invoked. Add tests to verify behavior with null hostId and exception handling.
@Damans227
Damans227 force-pushed the power-state-sync-npe-resilience branch from 90ce7bd to 57a5ff8 Compare October 5, 2026 13:40
@Damans227

Copy link
Copy Markdown
Collaborator Author

@nvazquez can you review this one? its rebased on main and the conflicts are fixed

@nvazquez
nvazquez requested a balanced review from Copilot November 9, 2026 17:17

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

3 open issues
What changed in this PR

Improves resilience in AgentManagerImpl.handleCommands so a single misbehaving listener can’t prevent subsequent listeners (e.g., ping) from running, and fixes a DAO partial-select projection that omitted hostId, leading to incorrect VM host reporting.

Changes:

  • Wrap each Listener.processCommands invocation in handleCommands with try/catch and log-and-continue on failures.
  • Fix trace logging to report the listener type rather than the Pair wrapper.
  • Include hostId in VMInstanceDaoImpl.IdsPowerStateSelectSearch and add tests covering the null-hostId projection case.
File Description
engine/​orchestration/​src/​main/​java/​com/​cloud/​agent/​manager/​AgentManagerImpl.java Adds exception isolation per listener and corrects trace logging to avoid starving downstream listeners.
engine/​orchestration/​src/​test/​java/​com/​cloud/​agent/​manager/​AgentManagerImplTest.java Adds tests ensuring throwing listeners don’t abort the listener loop and downstream listeners still run.
engine/​schema/​src/​main/​java/​com/​cloud/​vm/​dao/​VMInstanceDaoImpl.java Fixes partial-select projection by adding hostId to selected fields.
engine/​schema/​src/​test/​java/​com/​cloud/​vm/​dao/​VMInstanceDaoImplTest.java Adds regression tests for update-power-state behavior when hostId is null due to partial-select projection.

Review effort: Lite

Comment on lines +463 to +469
try {
final boolean processed = listener.second().processCommands(attache.getId(), sequence, cmds);
logger.trace("SeqA {}-{}: {} by {}", attache.getId(), sequence, (processed ? "processed" : "not processed"), listener.second().getClass());
} catch (final Exception e) {
logger.warn("Listener {} threw an exception processing commands for agent {} seq {}; continuing to next listener",
listener.second().getClass().getName(), attache.getId(), sequence, e);
}
Comment on lines +215 to +217
* hostId (the default from setUp), which is what happens when the row was loaded through
* a partial-select projection that doesn't include hostId. isPowerStateInSyncWithInstanceState
* must not NPE on the null hostId, and must still detect the state is out-of-sync.
Comment on lines +185 to +189
Mockito.when(throwingListener.processCommands(Mockito.anyLong(), Mockito.anyLong(), Mockito.any()))
.thenThrow(new RuntimeException("simulated NPE from power-state sync"));

Listener goodListener = Mockito.mock(Listener.class);
Mockito.when(goodListener.processCommands(Mockito.anyLong(), Mockito.anyLong(), Mockito.any()))

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: Ready

Development

Successfully merging this pull request may close these issues.

3 participants