Skip to content

Fix DataNode status handling for Pipe receiver disk failures - #18484

Open
Caideyipi wants to merge 2 commits into
apache:masterfrom
Caideyipi:fix/datanode-multi-disk-status-recovery
Open

Fix DataNode status handling for Pipe receiver disk failures#18484
Caideyipi wants to merge 2 commits into
apache:masterfrom
Caideyipi:fix/datanode-multi-disk-status-recovery

Conversation

@Caideyipi

@Caideyipi Caideyipi commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator

Description

Avoid duplicate node status transitions

CommonConfig#setNodeStatus now returns immediately when the requested status is already active. This preserves the existing status reason and prevents repeated ReadOnly -> ReadOnly transition logs from causing a log storm.

Preserve the existing storage-disk recovery semantics

DataNode heartbeat entry and recovery decisions continue to use the original aggregate available/total disk ratio. Recovery does not require every FileStore to exceed the warning threshold, so the conditions for restoring Running are not made stricter.

Pipe receiver-only directories are not added to the system/storage disk aggregate. If Pipe and the storage engine use separate disks, a full Pipe disk therefore does not affect the DataNode status. If they share a disk, the storage-engine directory still makes that disk part of the normal aggregate.

Report Pipe disk failures without setting global ReadOnly

FolderManager keeps its existing behavior by default, while allowing Pipe receiver managers to disable the global ReadOnly side effect. Both the regular DataNode Pipe receiver and the IoTConsensusV2 receiver use this mode. Disk-space exceptions still propagate through the existing receiver response path, so the sender receives an error and can report/retry it.

Tests

  • CommonConfigTest: verifies that a repeated ReadOnly update preserves DISK_FULL.
  • DataNodeInternalRPCServiceImplDiskTest: verifies aggregate-ratio Running recovery remains unchanged and a below-threshold storage aggregate still enters ReadOnly.
  • FolderManagerTest: verifies all directory strategies throw DiskSpaceInsufficientException for a full Pipe receiver without changing the global node status.

Executed:

mvn -o -pl iotdb-core/node-commons -am -Dtest=CommonConfigTest -Dsurefire.failIfNoSpecifiedTests=false test
mvn -o -pl iotdb-core/datanode -am -Dcheckstyle.skip=true -Dspotless.skip=true -Dtest=DataNodeInternalRPCServiceImplDiskTest,FolderManagerTest -Dsurefire.failIfNoSpecifiedTests=false test
mvn -o -pl iotdb-core/node-commons,iotdb-core/datanode,iotdb-core/metrics/interface -DskipTests validate

This PR has:

  • been self-reviewed.
  • added unit tests to cover the new code paths.

Key changed/added classes (or packages if there are too many classes) in this PR
  • CommonConfig
  • FolderManager
  • DirectoryStrategy
  • DataNodeInternalRPCServiceImpl
  • IoTDBDataNodeReceiver
  • IoTConsensusV2Receiver

@Caideyipi Caideyipi changed the title Fix DataNode disk status recovery across multiple disks Fix DataNode status handling for Pipe receiver disk failures Aug 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant