HDDS-15879. [Ozone Versioning] [T1] Metadata foundation by symious · Pull Request #10781 · apache/ozone

symious · 2026-07-16T06:25:27Z

What changes were proposed in this pull request?

This is the first basic task of Ozone versioning, including the following points

bucket proto three-state enum
set-property state machine
OmKeyInfo proto extension
versionedKeyTableColumnFamily
layout feature gate

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-15879

How was this patch tested?

unit test.

chihsuan

Thanks for the clean foundation work! @symious I verified that the five commits map nicely to the design doc. Left a few small comments below.

chihsuan · 2026-07-16T13:30:36Z

+    }
+
+    /** No-op when status is null (e.g. records without the new field). */
+    public Builder setVersioningStatus(BucketVersioningStatus status) {


Nice work keeping the two fields in sync here! One spot that looks left behind:

OzoneManager#getBucketInfo copies only the legacy flag when resolving a link bucket:

ozone/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java

Lines 3059 to 3072 in b981afb

return bucketInfo.toBuilder()

.setDefaultReplicationConfig(

realBucket.getDefaultReplicationConfig())

.setIsVersionEnabled(realBucket.getIsVersionEnabled())

.setStorageType(realBucket.getStorageType())

.setQuotaInBytes(realBucket.getQuotaInBytes())

.setQuotaInNamespace(realBucket.getQuotaInNamespace())

.setUsedBytes(realBucket.getUsedBytes())

.setSnapshotUsedBytes(realBucket.getSnapshotUsedBytes())

.setSnapshotUsedNamespace(realBucket.getSnapshotUsedNamespace())

.setUsedNamespace(realBucket.getUsedNamespace())

.addAllMetadata(realBucket.getMetadata())

.setBucketLayout(realBucket.getBucketLayout())

.build();

Since SUSPENDED maps to flag=false, a suspended bucket read through a link would come back as UNVERSIONED. Should we also add .setVersioningStatus(realBucket.getVersioningStatus()) there?

Nice catch, Updated, PTAL.

chihsuan · 2026-07-16T13:35:11Z

+   * Returns the requested versioning status, or null if not being changed.
+   * @return BucketVersioningStatus
+   */
+  public BucketVersioningStatus getVersioningStatus() {


Tiny one: toAuditMap() still only records IS_VERSION_ENABLED, so a status change via the new field would show up as isVersionEnabled=null in the audit log. Worth adding a versioningStatus entry while we're here?

ozone/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmBucketArgs.java

Lines 199 to 206 in f74c331

public Map<String, String> toAuditMap() {

Map<String, String> auditMap = new LinkedHashMap<>();

auditMap.put(OzoneConsts.VOLUME, this.volumeName);

auditMap.put(OzoneConsts.BUCKET, this.bucketName);

auditMap.put(OzoneConsts.GDPR_FLAG,

getMetadata().get(OzoneConsts.GDPR_FLAG));

auditMap.put(OzoneConsts.IS_VERSION_ENABLED,

String.valueOf(this.isVersionEnabled));

Updated, PTAL.

chihsuan · 2026-07-16T13:37:07Z

+  // absent on records that predate versioning support (treated as the null
+  // version). A delete marker has isDeleteMarker set and no data blocks.
+  // isNullVersion marks the single overwritable "null version" slot per key.
+  private final Long versionId;


Could we also add these three fields to isKeyInfoSame() (and toString())? Right now two records differing only in versionId, or a delete marker vs. a regular key, compare as equal. Looks like the follow-up tasks will rely on OmKeyInfo equality in tests; having this in the foundation would prevent false-positive assertions later. OmBucketInfo already includes the new status in both, so this would keep the two helpers consistent.

ozone/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmKeyInfo.java

Lines 1004 to 1017 in f74c331

public boolean isKeyInfoSame(OmKeyInfo omKeyInfo, boolean checkPath,

boolean checkKeyLocationVersions,

boolean checkModificationTime,

boolean checkUpdateID,

boolean checkOwnerName) {

boolean isEqual = dataSize == omKeyInfo.dataSize &&

creationTime == omKeyInfo.creationTime &&

volumeName.equals(omKeyInfo.volumeName) &&

bucketName.equals(omKeyInfo.bucketName) &&

replicationConfig.equals(omKeyInfo.replicationConfig) &&

Objects.equals(getMetadata(), omKeyInfo.getMetadata()) &&

Objects.equals(acls, omKeyInfo.acls) &&

Objects.equals(getTags(), omKeyInfo.getTags()) &&

getObjectID() == omKeyInfo.getObjectID();

Nice catch, Updated, PTAL.

jojochuang · 2026-07-20T16:30:48Z

@yandrey321

chungen0126 · 2026-07-21T02:12:03Z

Thanks @symious for working on this! However, based on a recent community discussion, we’ve decided to deprecate Google Docs in favor of maintaining Markdown files via PRs (see #8284 ). Could you please convert the Google Doc into a Markdown file and submit it as a PR instead? Thanks!

symious · 2026-07-21T02:26:44Z

@chungen0126 OEP file here: #10822

rich7420 · 2026-08-23T15:05:29Z

+      processingPhase = RequestProcessingPhase.PRE_PROCESS,
+      requestType = Type.SetBucketProperty
+  )
+  public static OMRequest disallowSetBucketPropertyWithVersioningStatus(


This gate covers SetBucketProperty, but CreateBucket carries the same BucketInfo.versioningStatus and persists it (OmBucketInfo.getFromProtobuf) with no matching OBJECT_VERSIONING validator — so a pre-finalized cluster would accept a CreateBucket that sets it, bypassing this contract (and the create path runs no canTransitionTo). Not triggerable yet since no client sends the field. Worth mirroring this validator on Type.CreateBucket, or a TODO + follow-up Jira.

Good catch. Following S3 here: CreateBucket has no versioning parameter, versioning is only set on an existing bucket via PutBucketVersioning. So instead of the layout gate, T1.2 now rejects any CreateBucket carrying versioningStatus outright with INVALID_REQUEST. That covers post-finalization too, so I dropped the mirrored validator from T1.5.

S3 gives a bucket three versioning states, while Ozone has a single isVersionEnabled boolean. This adds the three-state status alongside the flag rather than in place of it, so existing buckets and older clients go on working unchanged. BucketVersioningStatus holds the three states and the state machine that governs them: UNVERSIONED may move anywhere, but once versioning has been enabled or suspended a bucket can never return to UNVERSIONED. The proto gains a matching enum and an optional versioningStatus on both BucketInfo and BucketArgs. OmBucketInfo keeps the two representations in sync in both directions: a status derives the flag (ENABLED -> true), and a record carrying only the flag derives a status, so a bucket written before this change still answers getVersioningStatus(). Disabling the flag is the asymmetric case - it leaves an explicitly SUSPENDED status alone, since the state machine has no way back to UNVERSIONED. Nothing enforces the state machine yet; this commit only defines it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

A bucket's versioning status may only change along the state machine the previous commit defined. OMBucketSetPropertyRequest now checks the requested status against the one the bucket already holds and rejects with INVALID_REQUEST what the state machine forbids - the return to UNVERSIONED once versioning has been enabled or suspended. A request carrying only the legacy flag is mapped onto the same machine before that check: enabling always means ENABLED, while disabling means SUSPENDED, except on a bucket that is still UNVERSIONED, where it stays UNVERSIONED. The status is refused outright at bucket creation. S3 has no way to create a bucket already in a versioning state: CreateBucket carries no such parameter, and the state is set afterwards through PutBucketVersioning. versioningStatus sits on BucketInfo because that message is the bucket's on-disk record and the shape InfoBucket and ListBuckets return, not because CreateBucket needs it; honouring it there would let a caller land on any status in one step, with none of the above applied. Nothing populates the field on a create today, so the request is rejected rather than quietly ignored - ignoring it would leave a future caller believing it had created a versioned bucket. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Each section below was a separate commit; they are folded here so the task's review fixes land as one change. * Address comments * Use a 0x00 separator in versionedKeyTable dbKeys Key names in OBJECT_STORE buckets contain '/' verbatim, so a '/' separator interleaves a key's versions with those of keys nested under it, breaking the single-seek promotion and the merged ListObjectVersions order. * Do not derive a versioning status from the legacy flag Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

chihsuan reviewed Jul 16, 2026

View reviewed changes

jojochuang added the design label Jul 20, 2026

symious force-pushed the HDDS-15879 branch 2 times, most recently from 09d92b3 to e89e8be Compare July 31, 2026 02:28

symious changed the title ~~HDDS-15879. Ozone Versioning Metadata foundation~~ HDDS-15879. [Ozone Versioning] [T1] Metadata foundation Jul 31, 2026

symious force-pushed the HDDS-15879 branch from 3be7956 to f26c7c6 Compare August 18, 2026 06:22

rich7420 reviewed Aug 23, 2026

View reviewed changes

symious force-pushed the HDDS-15879 branch 3 times, most recently from 1128ce3 to e79471c Compare August 24, 2026 07:35

symious and others added 6 commits August 28, 2026 11:07

T1.3. Add versionId, isDeleteMarker and isNullVersion to OmKeyInfo

ebc2e92

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

T1.4. Add versionedKeyTable column family for noncurrent object versions

2b6bfa0

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

T1.5. Gate object versioning behind OMLayoutFeature.OBJECT_VERSIONING

640ddc2

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

symious force-pushed the HDDS-15879 branch from e79471c to b723516 Compare August 28, 2026 03:16

Navigation Menu

Uh oh!

HDDS-15879. [Ozone Versioning] [T1] Metadata foundation - #10781

HDDS-15879. [Ozone Versioning] [T1] Metadata foundation#10781
symious wants to merge 6 commits into
apache:HDDS-15728from
symious:HDDS-15879

symious commented Jul 16, 2026

Uh oh!

chihsuan left a comment

Uh oh!

chihsuan Jul 16, 2026

Uh oh!

symious Jul 17, 2026

Uh oh!

chihsuan Jul 16, 2026

Uh oh!

symious Jul 17, 2026

Uh oh!

chihsuan Jul 16, 2026

Uh oh!

symious Jul 17, 2026

Uh oh!

jojochuang commented Jul 20, 2026

Uh oh!

chungen0126 commented Jul 21, 2026

Uh oh!

symious commented Jul 21, 2026

Uh oh!

rich7420 Aug 23, 2026

Uh oh!

symious Aug 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

	return bucketInfo.toBuilder()
	.setDefaultReplicationConfig(
	realBucket.getDefaultReplicationConfig())
	.setIsVersionEnabled(realBucket.getIsVersionEnabled())
	.setStorageType(realBucket.getStorageType())
	.setQuotaInBytes(realBucket.getQuotaInBytes())
	.setQuotaInNamespace(realBucket.getQuotaInNamespace())
	.setUsedBytes(realBucket.getUsedBytes())
	.setSnapshotUsedBytes(realBucket.getSnapshotUsedBytes())
	.setSnapshotUsedNamespace(realBucket.getSnapshotUsedNamespace())
	.setUsedNamespace(realBucket.getUsedNamespace())
	.addAllMetadata(realBucket.getMetadata())
	.setBucketLayout(realBucket.getBucketLayout())
	.build();

	public Map<String, String> toAuditMap() {
	Map<String, String> auditMap = new LinkedHashMap<>();
	auditMap.put(OzoneConsts.VOLUME, this.volumeName);
	auditMap.put(OzoneConsts.BUCKET, this.bucketName);
	auditMap.put(OzoneConsts.GDPR_FLAG,
	getMetadata().get(OzoneConsts.GDPR_FLAG));
	auditMap.put(OzoneConsts.IS_VERSION_ENABLED,
	String.valueOf(this.isVersionEnabled));

	public boolean isKeyInfoSame(OmKeyInfo omKeyInfo, boolean checkPath,
	boolean checkKeyLocationVersions,
	boolean checkModificationTime,
	boolean checkUpdateID,
	boolean checkOwnerName) {
	boolean isEqual = dataSize == omKeyInfo.dataSize &&
	creationTime == omKeyInfo.creationTime &&
	volumeName.equals(omKeyInfo.volumeName) &&
	bucketName.equals(omKeyInfo.bucketName) &&
	replicationConfig.equals(omKeyInfo.replicationConfig) &&
	Objects.equals(getMetadata(), omKeyInfo.getMetadata()) &&
	Objects.equals(acls, omKeyInfo.acls) &&
	Objects.equals(getTags(), omKeyInfo.getTags()) &&
	getObjectID() == omKeyInfo.getObjectID();

Uh oh!

Conversation

symious commented Jul 16, 2026

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

chihsuan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jojochuang commented Jul 20, 2026

Uh oh!

chungen0126 commented Jul 21, 2026

Uh oh!

symious commented Jul 21, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants