Repository navigation
DAOS-17357 rebuild: use dmg for stop/start, force-stop fix - #16890
Merged
Merged
Conversation
Before this change, upon a failed rebuild, both an op:Fail_reclaim and a retry of the original op:Rebuild were scheduled. With this change, the scheduling of the retry of the original op:Rebuild is deferred until the subsequent completion of op:Fail_reclaim. This is done so that if a rebuild stop command happens to arrive during Fail_reclaim, the administrator can stop an endless retry loop of op:Rebuild / Fail_reclaim from otherwise occurring. ds_rebuild_scheudle() now accepts additional arguments specifically interpreted when scheduling op:Fail_reclaim. So that when Fail_reclaim later completes and might schedule a retry of the original op:Rebuild, it has the appropriate pool map version and rebuild opcode to supply to that later ds_rebuild_schedule() call. rebuild_task_complete_schedule() is updated to schedule Fail_reclaim, and detect on its completion if a stop command was received, and decide whether to schedule a retry of original op:Rebuild. Exercise the new stop handling during Fail_reclaim from daos_test -v (simple rebuild, rebuild_many_objects_with_failure test) when run with --rebuild_interactive. This is done via a new fault injection, DAOS_REBUILD_ADMIN_STOP_RECLAIM. Features: rebuild Allow-unstable-test: true Signed-off-by: Kenneth Cain <kenneth.cain@hpe.com>
daos_test -B --subtests="15" EXTEND16 failure Features: rebuild Allow-unstable-test: true Signed-off-by: Kenneth Cain <kenneth.cain@hpe.com>
Test-tag: pr test_daos_rebuild_simple_interactive Allow-unstable-test: true
Test-tag: pr test_daos_rebuild_simple_interactive Allow-unstable-test: true
Remove use of fault injection for interactive rebuild testing in daos_test -v (rebuild simple tests). Replace with new dmg command helper functions to invoke the control plane commands dmg pool rebuild stop and dmg pool rebuild start. Some common test functions are added and modified to support this. Also, some rebuild engine logic is updated to remove the associated fault injection handling. Restore the engine rebuild_ults() logic so that it does not loop forever (a change made in initial implementation for fault injection). Also, update the engine rebuild stop logic so it does not allow "dmg pool rebuild stop --force" to stop a rebuild in op:Fail_reclaim, unless it has been failing repeatedly. For this, new counts are added in struct rebuild_task and struct rebulid_global_pool_tracker to track, for a high level rebuild operation, how many times op:Rebuild, Reclaim, Fail_reclaim, etc. have been run (and how many times op:Rebuild and op:Fail_reclaim have failed). daos_test -v rebuild_many_objects_with_failure() is updated to specifically use the --force option to exercise arrival of command during Fail_reclaim, and exercise that the rebuild will not be stopped during Fail_reclaim in this instance. i.e., it will work the same as without the --force option, preventing retry of original op:Rebuild after the one/only Fail_reclaim is done. Test-tag: pr test_daos_rebuild_simple_interactive Allow-unstable-test: true Signed-off-by: Kenneth Cain <kenneth.cain@hpe.com>
|
Ticket title is 'interactive rebuild stop/start: engine integration with MS control plane' |
6 tasks
Base automatically changed from
kccain/daos_17281_stop_in_fail_reclaim
to
master
September 22, 2025 15:42
6 tasks
Test-tag: pr test_daos_rebuild_simple_interactive Allow-unstable-test: true Signed-off-by: Kenneth Cain <kenneth.cain@hpe.com>
Collaborator
|
Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16890/3/execution/node/1294/log erasurecode/online_rebuild_mdtest.py failed due to existing issue https://daosio.atlassian.net/browse/DAOS-17751 |
Collaborator
|
Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-16890/3/testReport/ Failures were introduced by the PR. Pushing a fix for these.
|
Increase sleep time by one more second in rebuild test functions rebuild_stop_with_dmg() and rebuild_force_stop_with_dmg(), to better ensure stop arrives during Fail_reclaim in the test (rebuild_many_objects_with_failure). Test-tag: pr test_daos_rebuild_simple_interactive Allow-unstable-test: true Signed-off-by: Kenneth Cain <kenneth.cain@hpe.com>
liw
approved these changes
Sep 30, 2025
wangshilong
approved these changes
Sep 30, 2025
osalyk
pushed a commit
that referenced
this pull request
Oct 1, 2025
Remove use of fault injection for interactive rebuild testing in daos_test -v (rebuild simple tests). Replace with new dmg command helper functions to invoke the control plane commands dmg pool rebuild stop and dmg pool rebuild start. Some common test functions are added and modified to support this. Also, some rebuild engine logic is updated to remove the associated fault injection handling. Restore the engine rebuild_ults() logic so that it does not loop forever (a change made in initial implementation for fault injection). Also, update the engine rebuild stop logic so it does not allow "dmg pool rebuild stop --force" to stop a rebuild in op:Fail_reclaim, unless it has been failing repeatedly. For this, new counts are added in struct rebuild_task and struct rebulid_global_pool_tracker to track, for a high level rebuild operation, how many times op:Rebuild, Reclaim, Fail_reclaim, etc. have been run (and how many times op:Rebuild and op:Fail_reclaim have failed). daos_test -v rebuild_many_objects_with_failure() is updated to specifically use the --force option to exercise arrival of command during Fail_reclaim, and exercise that the rebuild will not be stopped during Fail_reclaim in this instance. i.e., it will work the same as without the --force option, preventing retry of original op:Rebuild after the one/only Fail_reclaim is done. Signed-off-by: Kenneth Cain <kenneth.cain@hpe.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Remove use of fault injection for interactive rebuild testing in daos_test -v (rebuild simple tests). Replace with new dmg command helper functions to invoke the control plane commands dmg pool rebuild stop and dmg pool rebuild start. Some common test functions are added and modified to support this. Also, some rebuild engine logic is updated to remove the associated fault injection handling.
Restore the engine rebuild_ults() logic so that it does not loop forever (a change made in initial implementation for fault injection).
Also, update the engine rebuild stop logic so it does not allow "dmg pool rebuild stop --force" to stop a rebuild in op:Fail_reclaim, unless it has been failing repeatedly. For this, new counts are added in struct rebuild_task and struct rebulid_global_pool_tracker to track, for a high level rebuild operation, how many times op:Rebuild, Reclaim, Fail_reclaim, etc. have been run (and how many times op:Rebuild and op:Fail_reclaim have failed).
daos_test -v rebuild_many_objects_with_failure() is updated to specifically use the --force option to exercise arrival of command during Fail_reclaim, and exercise that the rebuild will not be stopped during Fail_reclaim in this instance. i.e., it will work the same as without the --force option, preventing retry of original op:Rebuild after the one/only Fail_reclaim is done.
Test-tag: pr test_daos_rebuild_simple_interactive
Allow-unstable-test: true
Steps for the author:
After all prior steps are complete: