DAOS-17831 control: Add dmg pool rebuild start|stop cmds by tanabarr · Pull Request #16686 · daos-stack/daos

tanabarr · 2025-07-30T13:45:52Z

Add start and stop interactive rebuild commands for a pool to dmg.
Includes dmg syntax updates, dRPC protobuf definitions and call
handling which in turn executes existing ds_mgmt_pool_rebuild_start
and stop API.

Steps for the author:

Commit message follows the guidelines.
Appropriate Features or Test-tag pragmas were used.
Appropriate Functional Test Stages were run.
At least two positive code reviews including at least one code owner from each category referenced in the PR.
Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

Gatekeeper requested (daos-gatekeeper added as a reviewer).

github-actions · 2025-07-30T13:46:10Z

Ticket title is 'Add dmg pool rebuild start|stop cmds'
Status is 'In Review'
Labels: 'Rebuild,control-plane'
https://daosio.atlassian.net/browse/DAOS-17831

daosbuild3 · 2025-07-30T13:58:48Z

Test stage Build on Leap 15.5 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16686/1/execution/node/415/log

daosbuild3 · 2025-07-30T13:59:02Z

Test stage Build on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16686/1/execution/node/407/log

daosbuild3 · 2025-07-30T13:59:04Z

Test stage Build on Leap 15.5 with Intel-C and TARGET_PREFIX completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16686/1/execution/node/418/log

daosbuild3 · 2025-07-30T13:59:05Z

Test stage Build on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16686/1/execution/node/420/log

daosbuild3 · 2025-07-30T14:01:42Z

Test stage Build RPM on EL 8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16686/1/execution/node/338/log

daosbuild3 · 2025-07-30T14:01:51Z

Test stage Build RPM on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16686/1/execution/node/360/log

daosbuild3 · 2025-07-30T14:04:35Z

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16686/1/execution/node/335/log

daosbuild3 · 2025-08-28T22:39:53Z

Test stage Functional on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-16686/2/testReport/

daosbuild3 · 2025-08-29T15:01:06Z

Test stage Functional on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-16686/3/testReport/

daosbuild3 · 2025-09-02T14:05:33Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-16686/4/testReport/

daosbuild3 · 2025-09-02T17:31:52Z

Test stage Functional on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-16686/4/display/redirect

tanabarr · 2025-09-02T17:39:12Z

@kccain can you suggest some suitable steps to run some basic manual sanity/smoke testing on this PR? e.g. create pool, exclude ranks, start interactive rebuild, see something happened in the logs etc.

kccain · 2025-09-02T21:11:06Z

@kccain can you suggest some suitable steps to run some basic manual sanity/smoke testing on this PR? e.g. create pool, exclude ranks, start interactive rebuild, see something happened in the logs etc.

@tanabarr sure. The timing of the rebuild execution and the stop command can be tricky to get right. If the op:Rebuild finishes before the stop command is received, then the rebuild will have gone to its final step (op:Reclaim) and will result the rebuild completing successfully (Rebuild done, status=0). You may have more luck with stopping a reintegration rebuild than the original exclude. That's what happened for me when I built and tried this branch on a developer system. Missed the timing window to stop/restart the exclude rebuild. But caught it on the subsequent reintegration. See below.

As one option (exlcude-based rebuild), you'll want to run dmg pool exclude --ranks= <pool_label> (or kill a rank and wait for the exclusion to occur), wait for the rebuild to get started (according to dmg pool query rebuild state becoming Busy, status=0), then (quickly) running dmg pool rebuild stop <pool_label>. Then, monitoring with dmg pool query <pool_label> should (assuming stop command timing was fortunate) eventually report "Rebuild failed, status=-2027 with -2027 being the value -DER_OP_CANCELED). The pool will stay in this state until the start command is run next. Run dmg pool rebuild start <pool_label> and monitor the rebuild progress with dmg pool query <pool_label> until the state is Done, status=0.

Alternatively, for reintegration: After the original exclude rebuild ran (successfully, without being stopped), run dmg pool reintegrate --ranks=, wait for rebuild to start, run the rebuild stop command, confirm rebuild status=-2027, then dmg pool rebuild start again and confirm with pool query that the rebuild state becomes Done, status=0.

In the PS leader engine logs, here is an example of what would be seen with a (stopped) rebuild (the op:Rebuild is aborted with status=-2027, and an op:Fail_reclaim is run for this "failure"):

hsw-202: /tmp/kccain/server1.log.343704.rank=2:09/02-20:57:06.46 hsw-202 DAOS[343704/0/170] rebuild INFO src/rebuild/srv.c:1042 rebuild_leader_status_check() rb=ae660164/25/3/Rebuild [aborted] (leader 2 dtx_gl 25 toberb_obj=9, rb_obj=3, rec=178258112, size=178258168 done 0 status -2027/0 stable 20ba013f95580000 reclaim 20ba012d2fe40002 duration=6 secs)
hsw-202: /tmp/kccain/server1.log.343704.rank=2:09/02-20:57:17.46 hsw-202 DAOS[343704/0/176] rebuild INFO src/rebuild/srv.c:1042 rebuild_leader_status_check() rb=ae660164/17/4/Reclaim fail [completed] (leader 2 dtx_gl 25 toberb_obj=0, rb_obj=0, rec=0, size=0 done 2 status 0/0 stable 20ba01688ff00000 reclaim 20ba013f95580000 duration=6 secs)

And then when the pool rebuild start command is run, eventually the rebuild op:Rebuild should complete successfully, and its corresponding op:Reclaim

hsw-202: /tmp/kccain/server1.log.343704.rank=2:09/02-21:04:58.46 hsw-202 DAOS[343704/0/182] rebuild INFO src/rebuild/srv.c:1042 rebuild_leader_status_check() rb=ae660164/25/5/Rebuild [completed] (leader 2 dtx_gl 25 toberb_obj=9, rb_obj=3, rec=263192768, size=263192824 done 2 status 0/0 stable 20ba0816795c0000 reclaim 20ba080478940002 duration=8 secs)
hsw-202: /tmp/kccain/server1.log.343704.rank=2:09/02-21:05:11.46 hsw-202 DAOS[343704/0/195] rebuild INFO src/rebuild/srv.c:1042 rebuild_leader_status_check() rb=ae660164/25/6/Reclaim [completed] (leader 2 dtx_gl 33 toberb_obj=0, rb_obj=3, rec=0, size=0 done 2 status 0/0 stable 20ba0846e6e80000 reclaim 20ba080478940002 duration=8 secs)

tanabarr · 2025-09-10T01:06:19Z

had to resolve conflicts

daltonbohning · 2025-09-10T14:15:00Z

Copyright check is failing because the latest message was signed off with an intel email

Signed-off-by: Tom Nabarro <tom.nabarro@intel.com>

…olrbld-startstop Features: control Allow-unstable-test: true Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>

daosbuild3 · 2025-09-12T15:55:30Z

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16686/10/execution/node/734/log

daosbuild3 · 2025-09-12T21:24:34Z

Test stage Unit Test on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-16686/11/testReport/

daosbuild3 · 2025-09-12T21:34:40Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-16686/11/testReport/

daosbuild3 · 2025-09-12T22:04:19Z

Test stage Unit Test with memcheck on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-16686/11/testReport/

daosbuild3 · 2025-09-12T23:16:21Z

Test stage Unit Test bdev with memcheck on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-16686/11/testReport/

daosbuild3 · 2025-09-14T10:31:47Z

Test stage Unit Test on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-16686/12/testReport/

daosbuild3 · 2025-09-14T10:43:32Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-16686/12/testReport/

daosbuild3 · 2025-09-14T11:01:40Z

Test stage Unit Test with memcheck on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-16686/12/testReport/

daosbuild3 · 2025-09-14T12:19:41Z

Test stage Unit Test bdev with memcheck on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-16686/12/testReport/

knard38

LGTM for what I understand

tanabarr self-assigned this Jul 30, 2025

tanabarr added control-plane work on the management infrastructure of the DAOS Control Plane CR Catastrophic Recovery Feature labels Jul 30, 2025

tanabarr force-pushed the tanabarr/control-poolrbld-startstop branch from 8365f7d to 651dc81 Compare August 28, 2025 17:09

tanabarr force-pushed the tanabarr/control-poolrbld-startstop branch from 651dc81 to adb6313 Compare August 29, 2025 09:48

tanabarr force-pushed the tanabarr/control-poolrbld-startstop branch from adb6313 to 2acc178 Compare September 2, 2025 13:14

tanabarr force-pushed the tanabarr/control-poolrbld-startstop branch from 2acc178 to 696c466 Compare September 2, 2025 17:30

tanabarr marked this pull request as ready for review September 2, 2025 17:30

tanabarr requested review from a team as code owners September 2, 2025 17:30

tanabarr requested review from kccain, kjacque, knard38 and liw September 2, 2025 17:36

tanabarr dismissed stale reviews from kjacque and kccain via 854e060 September 10, 2025 01:04

tanabarr removed the request for review from a team September 10, 2025 01:05

tanabarr removed the forced-landing The PR has known failures or has intentionally reduced testing, but should still be landed. label Sep 10, 2025

tanabarr requested review from kccain and kjacque September 10, 2025 01:05

kccain previously approved these changes Sep 10, 2025

View reviewed changes

kjacque previously approved these changes Sep 10, 2025

View reviewed changes

kccain mentioned this pull request Sep 10, 2025

DAOS-17281 rebuild: handle admin stop during op:Fail_reclaim #16813

Merged

6 tasks

Merge remote-tracking branch 'origin/master' into tanabarr/control-po…

becfcc0

…olrbld-startstop Features: control Allow-unstable-test: true Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>

tanabarr added the forced-landing The PR has known failures or has intentionally reduced testing, but should still be landed. label Sep 12, 2025

tanabarr dismissed stale reviews from kjacque and kccain via becfcc0 September 12, 2025 20:33

kjacque approved these changes Sep 12, 2025

View reviewed changes

kccain approved these changes Sep 15, 2025

View reviewed changes

knard38 approved these changes Sep 16, 2025

View reviewed changes

johannlombardi merged commit 6225f35 into master Sep 16, 2025
28 of 31 checks passed

johannlombardi deleted the tanabarr/control-poolrbld-startstop branch September 16, 2025 23:25

johannlombardi mentioned this pull request Sep 16, 2025

DAOS-17357 rebuild: dmg helpers for rebuild stop/start commands #16819

Closed

6 tasks

Conversation

tanabarr commented Jul 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Steps for the author:

After all prior steps are complete:

Uh oh!

github-actions Bot commented Jul 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

daosbuild3 commented Jul 30, 2025

Uh oh!

daosbuild3 commented Jul 30, 2025

Uh oh!

daosbuild3 commented Jul 30, 2025

Uh oh!

daosbuild3 commented Jul 30, 2025

Uh oh!

daosbuild3 commented Jul 30, 2025

Uh oh!

daosbuild3 commented Jul 30, 2025

Uh oh!

daosbuild3 commented Jul 30, 2025

Uh oh!

daosbuild3 commented Aug 28, 2025

Uh oh!

daosbuild3 commented Aug 29, 2025

Uh oh!

daosbuild3 commented Sep 2, 2025

Uh oh!

daosbuild3 commented Sep 2, 2025

Uh oh!

tanabarr commented Sep 2, 2025

Uh oh!

kccain commented Sep 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tanabarr commented Sep 10, 2025

Uh oh!

daltonbohning commented Sep 10, 2025

Uh oh!

daosbuild3 commented Sep 12, 2025

Uh oh!

daosbuild3 commented Sep 12, 2025

Uh oh!

daosbuild3 commented Sep 12, 2025

Uh oh!

daosbuild3 commented Sep 12, 2025

Uh oh!

daosbuild3 commented Sep 12, 2025

Uh oh!

daosbuild3 commented Sep 14, 2025

Uh oh!

daosbuild3 commented Sep 14, 2025

Uh oh!

daosbuild3 commented Sep 14, 2025

Uh oh!

daosbuild3 commented Sep 14, 2025

Uh oh!

knard38 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Milestone

Development

Uh oh!

7 participants

tanabarr commented Jul 30, 2025 •

edited

Loading

github-actions Bot commented Jul 30, 2025 •

edited

Loading

kccain commented Sep 2, 2025 •

edited

Loading