Skip to content

HDDS-16281. Reduce per-key overhead in Recon reprocess tasks - #11123

Open
rich7420 wants to merge 1 commit into
apache:masterfrom
rich7420:HDDS-16281
Open

HDDS-16281. Reduce per-key overhead in Recon reprocess tasks#11123
rich7420 wants to merge 1 commit into
apache:masterfrom
rich7420:HDDS-16281

Conversation

@rich7420

@rich7420 rich7420 commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Recon full reprocess tasks repeatedly resolve worker-local accumulation maps through a shared concurrent map for every key. The OBS namespace summary task also performs RocksDB bucket-table point reads for every key, even when consecutive keys belong to the same bucket.

This change:

  • caches OBS bucket metadata per worker and reuses the cached bucket object ID;
  • uses ThreadLocal worker accumulation maps while retaining a concurrent registry for final merging and flushing;
  • applies the worker-map optimization to namespace summary, container-key mapping, and file-size count reprocess paths;
  • adds regression coverage for the OBS bucket lookup count.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-16281

How was this patch tested?

https://github.com/rich7420/ozone/actions/runs/32956525485

  • 10 x 10 flaky-test-check: 100/100 passed after retrying one Maven Central HTTP 403 infrastructure failure.
  • A local before/after task-level benchmark used 100,000 OBS keys across two buckets, two warmups, and five measured runs, repeated in reverse order. Median reprocess time decreased from 2098/2109 ms on upstream/master to 1598/1548 ms with this patch (24-27% lower), while throughput improved by 31-36%.

Generated-by: Codex (GPT-5)

Copilot AI lite review requested due to automatic review settings August 26, 2026 13:58

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants