DAOS-17281 rebuild: handle admin stop during op:Fail_reclaim by kccain · Pull Request #16813 · daos-stack/daos

kccain · 2025-09-04T18:36:16Z

Before this change, upon a failed rebuild, both an op:Fail_reclaim and a retry of the original op:Rebuild were scheduled. With this change, the scheduling of the retry of the original op:Rebuild is deferred until the subsequent completion of op:Fail_reclaim. This is done so that if a rebuild stop command happens to arrive during Fail_reclaim, the administrator can stop an endless retry loop of op:Rebuild / Fail_reclaim from otherwise occurring.

ds_rebuild_scheudle() now accepts additional arguments specifically interpreted when scheduling op:Fail_reclaim. So that when Fail_reclaim later completes and might schedule a retry of the original op:Rebuild, it has the appropriate pool map version and rebuild opcode to supply to that later ds_rebuild_schedule() call.

rebuild_task_complete_schedule() is updated to schedule Fail_reclaim, and detect on its completion if a stop command was received, and decide whether to schedule a retry of original op:Rebuild.

Exercise the new stop handling during Fail_reclaim from daos_test -v (simple rebuild, rebuild_many_objects_with_failure test) when run with --rebuild_interactive. This is done via a new fault injection, DAOS_REBUILD_ADMIN_STOP_RECLAIM.

Features: rebuild
Allow-unstable-test: true

Steps for the author:

Commit message follows the guidelines.
Appropriate Features or Test-tag pragmas were used.
Appropriate Functional Test Stages were run.
At least two positive code reviews including at least one code owner from each category referenced in the PR.
Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

Gatekeeper requested (daos-gatekeeper added as a reviewer).

Before this change, upon a failed rebuild, both an op:Fail_reclaim and a retry of the original op:Rebuild were scheduled. With this change, the scheduling of the retry of the original op:Rebuild is deferred until the subsequent completion of op:Fail_reclaim. This is done so that if a rebuild stop command happens to arrive during Fail_reclaim, the administrator can stop an endless retry loop of op:Rebuild / Fail_reclaim from otherwise occurring. ds_rebuild_scheudle() now accepts additional arguments specifically interpreted when scheduling op:Fail_reclaim. So that when Fail_reclaim later completes and might schedule a retry of the original op:Rebuild, it has the appropriate pool map version and rebuild opcode to supply to that later ds_rebuild_schedule() call. rebuild_task_complete_schedule() is updated to schedule Fail_reclaim, and detect on its completion if a stop command was received, and decide whether to schedule a retry of original op:Rebuild. Exercise the new stop handling during Fail_reclaim from daos_test -v (simple rebuild, rebuild_many_objects_with_failure test) when run with --rebuild_interactive. This is done via a new fault injection, DAOS_REBUILD_ADMIN_STOP_RECLAIM. Features: rebuild Allow-unstable-test: true Signed-off-by: Kenneth Cain <kenneth.cain@hpe.com>

github-actions · 2025-09-04T18:36:31Z

Ticket title is 'Interactive mode rebuild'
Status is 'In Progress'
https://daosio.atlassian.net/browse/DAOS-17281

daosbuild3 · 2025-09-05T21:10:19Z

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16813/1/execution/node/1464/log

daosbuild3 · 2025-09-06T01:19:10Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16813/1/execution/node/1509/log

daosbuild3 · 2025-09-06T01:25:19Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-16813/1/testReport/

kccain · 2025-09-06T02:24:30Z

I will put this PR up for review now.

Build 1 with Features: rebuild performed well. Two tests got a container destroy DER_IO issue (known problems), and one rebuild/mdtest.py hung waiting for a rebuild, and it looks like that one got a -1007 (-DER_NOSPACE) failure in the rebuild, so it is unlikely related to this change.

wangshilong · 2025-09-10T13:30:34Z

I did not see rebuild extend16 failure before, it could be related to this PR?

liuxuezhao

hmm, I need to think a little bit more tomorrow.

liuxuezhao · 2025-09-10T14:40:03Z

+	 * retrying the original op:Rebuild.
+	 */
+	if (rgt->rgt_abort || (rgt->rgt_opc == RB_OP_FAIL_RECLAIM))
+		rgt->rgt_stop_admin = 1;


this setting seems duplicate with above L901, is it really needed? or can it be combined to above L901 section?

L901 branch is when the opcode is RB_OP_FAIL_RECLAIM and the force argument is true.
Here, for RB_OP_FAIL_RECLAIM and force==false, we still want to set rgt_stop_admin=1 but not rgt_abort.

Maybe I can just make the rgt_stop_admin = 1 assignment here only, rather than in the above branch.

One other thing about the branch around L901 above

@kjacque asked a question in a different PR what happens when the rebuild stop --force command occurs while op:Fail_reclaim is running for the first time? We may want to prevent that Fail_reclaim from being aborted - I can work on this in a subsequent patch I'm developing that integrates with the dmg commands from that PR.

liuxuezhao · 2025-09-10T14:41:34Z

-	bool                  fi_admin_stop    = false;
+	unsigned int          nsleeps           = 0;
+	bool                  fi_admin_stop     = false;
+	bool                  fi_admin_stop_frc = false;


the "frc" means force?

oh, frc in this case meant "fail_reclaim"

liuxuezhao · 2025-09-10T14:43:47Z

+		if (do_log)
+			DL_CDEBUG(error, DLOG_ERR, DLOG_INFO, error, DF_UUID " opc %u/%u retry",
+				  DP_UUID(task->dst_pool_uuid), task->dst_rebuild_op,
+				  task->dst_map_ver);


I don't get the point why the log depend on do_map_revert flag?

In my patch I have 2 calls to this function, the first one not updating the pool map. So I was trying to make it so that the logging would only happen on the second call where the pool map gets updated.

But @wangshilong in his question about the daos_test -B (extend simple) test failure made me take a look at this again, the caller and the function. I will be making a simplification after determining that my patch caused the EXTEND16 failure.

kccain · 2025-09-10T22:20:36Z

I did not see rebuild extend16 failure before, it could be related to this PR?

Yes, I looked at this in detail today and I think my patch failed to revert the pool map. Also the execution flow in rebuild_task_complete_schedule() with its calls to retry_rebuild_task() can be simplified to help address the issue (and hopefully still work for all other rebuild cases). I'm working on an updated patch.

daos_test -B --subtests="15" EXTEND16 failure Features: rebuild Allow-unstable-test: true Signed-off-by: Kenneth Cain <kenneth.cain@hpe.com>

daosbuild3 · 2025-09-11T22:32:29Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16813/3/execution/node/930/log

This is due to a problem running tests on the hdr-130 cluster, refer to SRE-3342 Build 4 was launched and these tests ran without errors on a different cluster.

daosbuild3 · 2025-09-12T01:19:19Z

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16813/3/execution/node/975/log

erasurecode/online_rebuild.py and erasurecode/multiple_target_failure.py - both are container destroy errors with DER_IO, likely instances of https://daosio.atlassian.net/browse/DAOS-17725

kccain · 2025-09-12T13:21:01Z

@liuxuezhao and @wangshilong this PR is ready for review. The latest push had some issue not running unit tests stage becasue it is based on master commit 3323a5d that caused some issues with that. I manually advanced the CI build to run through functional / HW test stages.

A fix for that has been landed, so I'll merge latest master and kick off a PR CI test run with these changes while awaiting reviews.

daosbuild3 · 2025-09-12T14:39:27Z

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16813/4/execution/node/476/log

Test-tag: pr test_daos_rebuild_simple_interactive Allow-unstable-test: true

daosbuild3 · 2025-09-12T15:58:14Z

Test stage Unit Test on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-16813/5/testReport/

daosbuild3 · 2025-09-12T16:01:17Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-16813/5/testReport/

daosbuild3 · 2025-09-12T16:13:57Z

Test stage Unit Test with memcheck on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-16813/5/testReport/

daosbuild3 · 2025-09-12T17:35:42Z

Test stage Unit Test bdev with memcheck on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-16813/5/testReport/

daosbuild3 · 2025-09-12T18:40:55Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-16813/6/testReport/

daosbuild3 · 2025-09-12T18:51:06Z

Test stage Unit Test on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-16813/6/testReport/

daosbuild3 · 2025-09-12T19:28:46Z

Test stage Unit Test with memcheck on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-16813/6/testReport/

daosbuild3 · 2025-09-12T20:20:34Z

Test stage Unit Test bdev with memcheck on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-16813/6/testReport/

Test-tag: pr test_daos_rebuild_simple_interactive Allow-unstable-test: true

daosbuild3 · 2025-09-20T03:22:01Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16813/7/execution/node/1423/log

Overall test timeout hit within 1 second of starting the final rebuild_ec_combined_ops test case of daos_rebuild_ec.c - likely an instance of (reopened) ticket https://daosio.atlassian.net/browse/DAOS-17673

I will kick off a build starting from Functional HW stages to get another trial.

kccain · 2025-09-21T12:38:00Z

@liuxuezhao @gnailzenh this one has passed Jenkins testing. Some unit test issues seen in GHA as a result of the landing of commit https://github.com/daos-stack/daos/commit/3323a5d6fb2ea6b0181c75cacfa9528d17450e01 including after some repair commits. Most recent merge of master for this PR was Sep 16 and took many days to get through CI.

I think we have a path to get most of this feature landed by reviewing & landing this PR, then taking a look at #16890 that is in CI testing now.

kccain mentioned this pull request Sep 4, 2025

DAOS-17281 rebuild: handle admin stop during op:Fail_reclaim #16731

Closed

6 tasks

kccain marked this pull request as ready for review September 6, 2025 02:24

kccain requested review from a team as code owners September 6, 2025 02:24

kccain requested review from a team, liuxuezhao, liw and wangshilong September 6, 2025 02:24

kccain mentioned this pull request Sep 8, 2025

DAOS-17831 control: Add dmg pool rebuild start|stop cmds #16686

Merged

4 of 6 tasks

liw previously approved these changes Sep 10, 2025

View reviewed changes

wangshilong previously approved these changes Sep 10, 2025

View reviewed changes

liuxuezhao reviewed Sep 10, 2025

View reviewed changes

Merge branch 'master' into kccain/daos_17281_stop_in_fail_reclaim

1fd502d

Address review comments and candidate test fix

10a2d3f

daos_test -B --subtests="15" EXTEND16 failure Features: rebuild Allow-unstable-test: true Signed-off-by: Kenneth Cain <kenneth.cain@hpe.com>

kccain dismissed stale reviews from wangshilong and liw via 10a2d3f September 11, 2025 02:46

kccain requested review from liuxuezhao and wangshilong September 12, 2025 13:21

wangshilong approved these changes Sep 12, 2025

View reviewed changes

Merge branch 'master' into kccain/daos_17281_stop_in_fail_reclaim

3f1762d

Test-tag: pr test_daos_rebuild_simple_interactive Allow-unstable-test: true

Merge branch 'master' into kccain/daos_17281_stop_in_fail_reclaim

cdd38b9

Test-tag: pr test_daos_rebuild_simple_interactive Allow-unstable-test: true

gnailzenh approved these changes Sep 22, 2025

View reviewed changes

gnailzenh merged commit 3787112 into master Sep 22, 2025
41 of 45 checks passed

gnailzenh deleted the kccain/daos_17281_stop_in_fail_reclaim branch September 22, 2025 15:42

Conversation

kccain commented Sep 4, 2025

Steps for the author:

After all prior steps are complete:

Uh oh!

github-actions Bot commented Sep 4, 2025

Uh oh!

daosbuild3 commented Sep 5, 2025

Uh oh!

daosbuild3 commented Sep 6, 2025

Uh oh!

daosbuild3 commented Sep 6, 2025

Uh oh!

kccain commented Sep 6, 2025

Uh oh!

wangshilong commented Sep 10, 2025

Uh oh!

liuxuezhao left a comment

Choose a reason for hiding this comment

Uh oh!

liuxuezhao Sep 10, 2025

Choose a reason for hiding this comment

Uh oh!

kccain Sep 10, 2025

Choose a reason for hiding this comment

Uh oh!

kccain Sep 10, 2025

Choose a reason for hiding this comment

Uh oh!

liuxuezhao Sep 10, 2025

Choose a reason for hiding this comment

Uh oh!

kccain Sep 10, 2025

Choose a reason for hiding this comment

Uh oh!

liuxuezhao Sep 10, 2025

Choose a reason for hiding this comment

Uh oh!

kccain Sep 10, 2025

Choose a reason for hiding this comment

Uh oh!

kccain commented Sep 10, 2025

Uh oh!

daosbuild3 commented Sep 11, 2025 • edited by kccain Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

daosbuild3 commented Sep 12, 2025 • edited by kccain Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kccain commented Sep 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

daosbuild3 commented Sep 12, 2025

Uh oh!

daosbuild3 commented Sep 12, 2025

Uh oh!

daosbuild3 commented Sep 12, 2025

Uh oh!

daosbuild3 commented Sep 12, 2025

Uh oh!

daosbuild3 commented Sep 12, 2025

Uh oh!

daosbuild3 commented Sep 12, 2025

Uh oh!

daosbuild3 commented Sep 12, 2025

Uh oh!

daosbuild3 commented Sep 12, 2025

Uh oh!

daosbuild3 commented Sep 12, 2025

Uh oh!

daosbuild3 commented Sep 20, 2025 • edited by kccain Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kccain commented Sep 21, 2025

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

daosbuild3 commented Sep 11, 2025 •

edited by kccain

Loading

daosbuild3 commented Sep 12, 2025 •

edited by kccain

Loading

kccain commented Sep 12, 2025 •

edited

Loading

daosbuild3 commented Sep 20, 2025 •

edited by kccain

Loading