[kccain]

Also, remove the corresponding original fault injection controls (DAOS_REBUILD_ADMIN_STOP and DAOS_REBUILD_ADMIN_START) from the rebuild engine implementation and tests. The simple rebuild test (daos_test -v) uses the helper functions/commands, and no longer the fault injections.

Restore the rebuild_ults() logic so that it does not live / loop forever (a change that was done earlier to accommodate the faults).

Features: rebuild
Allow-unstable-test: true

Steps for the author:

• Commit message follows the guidelines.
• Appropriate Features or Test-tag pragmas were used.
• Appropriate Functional Test Stages were run.
• At least two positive code reviews including at least one code owner from each category referenced in the PR.
• Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

• Gatekeeper requested (daos-gatekeeper added as a reviewer).

[github-actions]

Ticket title is 'interactive rebuild stop/start: engine integration with MS control plane'
Status is 'In Progress'
https://daosio.atlassian.net/browse/DAOS-17357

[daosbuild3]

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-16819/3/testReport/

An instance of rare/intermittent known failure https://daosio.atlassian.net/browse/DAOS-17416

[daosbuild3]

Test stage Functional on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-16819/3/testReport/

daos_test/dfuse.py failures are an instance of known issue https://daosio.atlassian.net/browse/DAOS-17758

[daosbuild3]

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-16819/3/testReport/

[daosbuild3]

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16819/3/execution/node/1648/log

The base branch was changed.

[johannlombardi]

PR #16686 is now landed. Please rebase. Thanks.

[daosbuild3]

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16819/7/execution/node/1294/log

[daosbuild3]

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-16819/7/testReport/

[kccain]

I'm working on #16890 instead of this one for DAOS-17357