DAOS-17281 rebuild: handle admin stop during op:Fail_reclaim by kccain · Pull Request #16731 · daos-stack/daos

kccain · 2025-08-12T01:06:03Z

Features: rebuild
Allow-unstable-test: true

Steps for the author:

Commit message follows the guidelines.
Appropriate Features or Test-tag pragmas were used.
Appropriate Functional Test Stages were run.
At least two positive code reviews including at least one code owner from each category referenced in the PR.
Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

Gatekeeper requested (daos-gatekeeper added as a reviewer).

github-actions · 2025-08-12T01:06:19Z

Ticket title is 'Interactive mode rebuild'
Status is 'In Progress'
https://daosio.atlassian.net/browse/DAOS-17281

daosbuild3 · 2025-08-12T09:30:09Z

Test stage Functional on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16731/1/execution/node/1247/log

daosbuild3 · 2025-08-12T15:13:37Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16731/1/execution/node/1518/log

daosbuild3 · 2025-08-12T16:12:14Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16731/1/execution/node/1507/log

daosbuild3 · 2025-08-13T02:38:19Z

Test stage Functional on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-16731/2/display/redirect

Features: rebuild Allow-unstable-test: true Signed-off-by: Kenneth Cain <kenneth.cain@hpe.com>

daosbuild3 · 2025-08-23T04:07:23Z

Test stage Functional on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16731/6/execution/node/478/log

kccain · 2025-08-23T11:37:06Z

Test stage Functional on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16731/6/execution/node/478/log

control/dmg_pool_query_ranks.py failure is an instance of known issue DAOS-17756

(to be determined still) if control/log_entry.py failure is an instance of https://daosio.atlassian.net/browse/DAOS-17854

daosbuild3 · 2025-08-23T23:20:21Z

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16731/6/execution/node/735/log

daosbuild3 · 2025-08-23T23:21:54Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16731/6/execution/node/825/log

daosbuild3 · 2025-08-24T03:07:26Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16731/6/execution/node/780/log

kccain · 2025-08-25T13:16:28Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16731/6/execution/node/780/log

daos_test/rebuild.py failures are an instance of known bug https://daosio.atlassian.net/browse/DAOS-17907

kccain · 2025-08-25T13:17:44Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16731/6/execution/node/825/log

daos_test/suite.py failures may be due to some ib1 fabric interface intermittent problem on machine hdr-222 as noted in https://daosio.atlassian.net/browse/DAOS-17885

kccain · 2025-08-25T13:20:59Z

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16731/6/execution/node/735/log

erasurecode/mdtest_smoke.py, erasurecode/offline_rebuild_aggregation.py, rebuild/widely_striped.py failures all due to https://daosio.atlassian.net/browse/SRE-3150

ior/hard_rebuild.py failure due to https://daosio.atlassian.net/browse/DAOS-17747 (and may also be NVMe device related)

kccain · 2025-08-25T13:23:39Z

Due to the many machine or known bug failures in build 6, I've restarted testing from the "Test Hardware" stage. Build 7 in progress now.

daosbuild3 · 2025-08-25T20:57:48Z

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16731/7/execution/node/453/log

daosbuild3 · 2025-08-25T22:08:38Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-16731/7/testReport/

daosbuild3 · 2025-08-25T23:40:25Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16731/7/execution/node/543/log

kccain · 2025-08-26T15:36:13Z

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16731/7/execution/node/453/log

more SRE-3150 issues

kccain · 2025-08-26T15:55:28Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16731/7/execution/node/543/log

daos_test/rebuild.py rebuild33 and rebuild34 again seemed to hit https://daosio.atlassian.net/browse/DAOS-17907

kccain · 2025-08-26T15:57:25Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-16731/7/testReport/

daos_test extend simple test failure, created new ticket https://daosio.atlassian.net/browse/DAOS-17930

daosbuild3 · 2025-08-26T22:50:12Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16731/8/execution/node/490/log

daosbuild3 · 2025-08-26T23:12:21Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-16731/8/testReport/

daosbuild3 · 2025-08-27T01:09:38Z

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16731/8/execution/node/509/log

daosbuild3 · 2025-08-27T22:24:31Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16731/9/execution/node/524/log

daosbuild3 · 2025-08-27T22:39:12Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16731/9/execution/node/543/log

daosbuild3 · 2025-08-27T23:36:18Z

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16731/9/execution/node/453/log

kccain · 2025-09-04T18:42:10Z

closing this initial implementation attempt PR - it's testing history is getting too long, and was affected by a bunch of unrelated test / infrastructure issues. A new PR is in process now, #16813

kccain force-pushed the kccain/daos_17281_stop_during_fail_reclaim branch from 253fd6e to d938ed6 Compare August 12, 2025 22:05

kccain force-pushed the kccain/daos_17281_stop_during_fail_reclaim branch from d938ed6 to 99a37f8 Compare August 22, 2025 20:03

DAOS-17281 rebuild: handle admin stop during op:Fail_reclaim

3109e38

Features: rebuild Allow-unstable-test: true Signed-off-by: Kenneth Cain <kenneth.cain@hpe.com>

kccain force-pushed the kccain/daos_17281_stop_during_fail_reclaim branch from 99a37f8 to 3109e38 Compare August 22, 2025 20:33

kccain closed this Sep 4, 2025

Conversation

kccain commented Aug 12, 2025

Steps for the author:

After all prior steps are complete:

Uh oh!

github-actions Bot commented Aug 12, 2025

Uh oh!

daosbuild3 commented Aug 12, 2025

Uh oh!

daosbuild3 commented Aug 12, 2025

Uh oh!

daosbuild3 commented Aug 12, 2025

Uh oh!

daosbuild3 commented Aug 13, 2025

Uh oh!

daosbuild3 commented Aug 23, 2025

Uh oh!

kccain commented Aug 23, 2025

Uh oh!

daosbuild3 commented Aug 23, 2025

Uh oh!

daosbuild3 commented Aug 23, 2025

Uh oh!

daosbuild3 commented Aug 24, 2025

Uh oh!

kccain commented Aug 25, 2025

Uh oh!

kccain commented Aug 25, 2025

Uh oh!

kccain commented Aug 25, 2025

Uh oh!

kccain commented Aug 25, 2025

Uh oh!

daosbuild3 commented Aug 25, 2025

Uh oh!

daosbuild3 commented Aug 25, 2025

Uh oh!

daosbuild3 commented Aug 25, 2025

Uh oh!

kccain commented Aug 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kccain commented Aug 26, 2025

Uh oh!

kccain commented Aug 26, 2025

Uh oh!

daosbuild3 commented Aug 26, 2025

Uh oh!

daosbuild3 commented Aug 26, 2025

Uh oh!

daosbuild3 commented Aug 27, 2025

Uh oh!

daosbuild3 commented Aug 27, 2025

Uh oh!

daosbuild3 commented Aug 27, 2025

Uh oh!

daosbuild3 commented Aug 27, 2025

Uh oh!

kccain commented Sep 4, 2025

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Milestone

Development

Uh oh!

2 participants

kccain commented Aug 26, 2025 •

edited

Loading