Skip to content

fix(net): stop BackupServer.close() hanging while bind is in flight - #150

Open
halibobo1205 wants to merge 1 commit into
developfrom
fix/backup-server-close-race
Open

fix(net): stop BackupServer.close() hanging while bind is in flight#150
halibobo1205 wants to merge 1 commit into
developfrom
fix/backup-server-close-race

Conversation

@halibobo1205

@halibobo1205 halibobo1205 commented Sep 9, 2026

Copy link
Copy Markdown
Owner

User description

What does this PR do?

Fix a BackupServer shutdown race that can cause BackupServerTest to time out.

Create the event loop group in initServer() so close() can shut it down even before the channel is published, then interrupt pending waits and await server-thread termination.

Why are these changes required?

Previously, close() only closed an already-published channel. If bind() completed afterward, the channel could remain open and leave the server thread blocked in closeFuture().sync().

Related CI failure: GitHub Actions job due to

org.tron.common.backup.BackupServerTest > test FAILED
    org.junit.runners.model.TestTimedOutException: test timed out after 60 seconds
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1475)
        at java.util.concurrent.Executors$DelegatedExecutorService.awaitTermination(Executors.java:675)
        at org.tron.common.es.ExecutorServiceManager.shutdownAndAwaitTermination(ExecutorServiceManager.java:90)
        at org.tron.common.backup.socket.BackupServer.close(BackupServer.java:106)
        at org.tron.common.backup.BackupServerTest.tearDown(BackupServerTest.java:42)

org.tron.common.backup.BackupServerTest > test FAILED
    org.junit.runners.model.TestTimedOutException: test timed out after 60 seconds
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1475)
        at java.util.concurrent.Executors$DelegatedExecutorService.awaitTermination(Executors.java:675)
        at org.tron.common.es.ExecutorServiceManager.shutdownAndAwaitTermination(ExecutorServiceManager.java:90)
        at org.tron.common.backup.socket.BackupServer.close(BackupServer.java:106)
        at org.tron.common.backup.BackupServerTest.tearDown(BackupServerTest.java:42)

org.tron.common.backup.BackupServerTest > test FAILED
    org.junit.runners.model.TestTimedOutException: test timed out after 60 seconds
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1475)
        at java.util.concurrent.Executors$DelegatedExecutorService.awaitTermination(Executors.java:675)
        at org.tron.common.es.ExecutorServiceManager.shutdownAndAwaitTermination(ExecutorServiceManager.java:90)
        at org.tron.common.backup.socket.BackupServer.close(BackupServer.java:106)
        at org.tron.common.backup.BackupServerTest.tearDown(BackupServerTest.java:42)

org.tron.common.backup.BackupServerTest > test FAILED
    org.junit.runners.model.TestTimedOutException: test timed out after 60 seconds
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1475)
        at java.util.concurrent.Executors$DelegatedExecutorService.awaitTermination(Executors.java:675)
        at org.tron.common.es.ExecutorServiceManager.shutdownAndAwaitTermination(ExecutorServiceManager.java:90)
        at org.tron.common.backup.socket.BackupServer.close(BackupServer.java:106)
        at org.tron.common.backup.BackupServerTest.tearDown(BackupServerTest.java:42)

org.tron.common.backup.BackupServerTest > test FAILED
    org.junit.runners.model.TestTimedOutException: test timed out after 60 seconds
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1475)
        at java.util.concurrent.Executors$DelegatedExecutorService.awaitTermination(Executors.java:675)
        at org.tron.common.es.ExecutorServiceManager.shutdownAndAwaitTermination(ExecutorServiceManager.java:90)
        at org.tron.common.backup.socket.BackupServer.close(BackupServer.java:106)
        at org.tron.common.backup.BackupServerTest.tearDown(BackupServerTest.java:42)

The provided test log records executor shutdown before bind success is logged:

06:27:24.034 INFO [backup] Closing backup server...
06:27:24.352 INFO [common-executor] Pool BackupServer shutdown...
06:27:24.366 INFO [backup] Backup server started, bind port 11729
...
06:28:15.616 INFO [common-executor] Pool BackupServer shutdown done
06:28:15.616 INFO [backup] Backup server closed.
06:28:15.617 ERROR [backup] Start backup server with port 11729 failed.
java.lang.InterruptedException: null
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:502)
    at io.netty.util.concurrent.DefaultPromise.await(DefaultPromise.java:268)
    ...
    at org.tron.common.backup.socket.BackupServer.start(BackupServer.java:80)

In the source, BackupServer.java:80 is channel.closeFuture().sync().

This PR has been tested by:

  • Regression coverage includes shutdown after startup, during startup, and with bind deterministically paused.
  • BackupManagerTest now uses normal server shutdown for cleanup.

CodeAnt-AI Description

Prevent backup server shutdown from hanging during startup

What Changed

  • Closing the backup server now stops pending startup and bind operations, even when shutdown begins before the server channel is ready
  • Backup server shutdown waits for its worker processes to finish and no longer leaves background work running
  • Added a regression test covering close during a pending bind
  • Backup manager tests now use the server shutdown path to stop keep-alive work

Impact

✅ Backup shutdown completes during startup
✅ Fewer test and application shutdown timeouts
✅ No lingering backup server background tasks

💡 Usage Guide

Checking Your Pull Request

Every time you make a pull request, our system automatically looks through it. We check for security issues, mistakes in how you're setting up your infrastructure, and common code problems. We do this to make sure your changes are solid and won't cause any trouble later.

Talking to CodeAnt AI

Got a question or need a hand with something in your pull request? You can easily get in touch with CodeAnt AI right here. Just type the following in a comment on your pull request, and replace "Your question here" with whatever you want to ask:

@codeant-ai ask: Your question here

This lets you have a chat with CodeAnt AI about your pull request, making it easier to understand and improve your code.

Example

@codeant-ai ask: Can you suggest a safer alternative to storing this secret?

Preserve Org Learnings with CodeAnt

You can record team preferences so CodeAnt AI applies them in future reviews. Reply directly to the specific CodeAnt AI suggestion (in the same thread) and replace "Your feedback here" with your input:

@codeant-ai: Your feedback here

This helps CodeAnt AI learn and adapt to your team's coding style and standards.

Example

@codeant-ai: Do not flag unused imports.

Retrigger review

Ask CodeAnt AI to review the PR again, by typing:

@codeant-ai: review

Check Your Repository Health

To analyze the health of your code repository, visit our dashboard at https://app.codeant.ai. This tool helps you identify potential issues and areas for improvement in your codebase, ensuring your repository maintains high standards of code health.

@codeant-ai

codeant-ai Bot commented Sep 9, 2026

Copy link
Copy Markdown

🤖 CodeAnt AI — Review Status

Status Commit Started (UTC) Finished (UTC)
✅ Incremental review completed a24a765 Sep 09, 2026 · 12:07 12:07
✅ Reviewed your PR f2cc454 Sep 09, 2026 · 10:19 10:22

@codeant-ai

codeant-ai Bot commented Sep 9, 2026

Copy link
Copy Markdown

Thanks for using CodeAnt! 🎉

We're free for open-source projects. if you're enjoying it, help us grow by sharing.

Share on X ·
Reddit ·
LinkedIn

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 9, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-09T12:48:16.254923Z 21076b1 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@codeant-ai codeant-ai Bot added the size:L This PR changes 100-499 lines, ignoring generated files label Sep 9, 2026
Comment thread framework/src/main/java/org/tron/common/backup/socket/BackupServer.java Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f2cc4548a5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread framework/src/main/java/org/tron/common/backup/socket/BackupServer.java Outdated
@halibobo1205
halibobo1205 force-pushed the fix/backup-server-close-race branch from f2cc454 to a24a765 Compare September 9, 2026 12:07

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a24a765e00

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

if (port > 0 && commonParameter.getBackupMembers().size() > 0) {
try {
// Let close() reach the group before bind() completes.
group = new NioEventLoopGroup(1);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Prevent startup from publishing resources after close

When close() races with this constructor before group is assigned, it observes both group and executor as null and returns; initServer() then publishes the group and submits start() to a new non-daemon executor. Since shutdown is already true, start() skips binding and returns after shutting down the group, but nothing ever shuts down the executor, leaving its worker alive and potentially preventing JVM termination. Coordinate initialization with close() or recheck shutdown before creating and submitting the executor.

Useful? React with 👍 / 👎.

Fix a shutdown race where close() runs before bind() publishes the
channel, leaving the server thread blocked until the 60-second executor
timeout.

Create the event loop group in initServer() so close() can shut it down
and interrupt pending waits.
@halibobo1205
halibobo1205 force-pushed the fix/backup-server-close-race branch from a24a765 to 21076b1 Compare September 9, 2026 12:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:L This PR changes 100-499 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant