HDDS-16093. [Ozone versioning] [T6] Reclamation - #10964
Open
symious wants to merge 28 commits into
Open
Conversation
symious
force-pushed
the
HDDS-16093
branch
6 times, most recently
from
August 24, 2026 07:51
154c8b8 to
cc3dd37
Compare
S3 gives a bucket three versioning states, while Ozone has a single isVersionEnabled boolean. This adds the three-state status alongside the flag rather than in place of it, so existing buckets and older clients go on working unchanged. BucketVersioningStatus holds the three states and the state machine that governs them: UNVERSIONED may move anywhere, but once versioning has been enabled or suspended a bucket can never return to UNVERSIONED. The proto gains a matching enum and an optional versioningStatus on both BucketInfo and BucketArgs. OmBucketInfo keeps the two representations in sync in both directions: a status derives the flag (ENABLED -> true), and a record carrying only the flag derives a status, so a bucket written before this change still answers getVersioningStatus(). Disabling the flag is the asymmetric case - it leaves an explicitly SUSPENDED status alone, since the state machine has no way back to UNVERSIONED. Nothing enforces the state machine yet; this commit only defines it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A bucket's versioning status may only change along the state machine the previous commit defined. OMBucketSetPropertyRequest now checks the requested status against the one the bucket already holds and rejects with INVALID_REQUEST what the state machine forbids - the return to UNVERSIONED once versioning has been enabled or suspended. A request carrying only the legacy flag is mapped onto the same machine before that check: enabling always means ENABLED, while disabling means SUSPENDED, except on a bucket that is still UNVERSIONED, where it stays UNVERSIONED. The status is refused outright at bucket creation. S3 has no way to create a bucket already in a versioning state: CreateBucket carries no such parameter, and the state is set afterwards through PutBucketVersioning. versioningStatus sits on BucketInfo because that message is the bucket's on-disk record and the shape InfoBucket and ListBuckets return, not because CreateBucket needs it; honouring it there would let a caller land on any status in one step, with none of the above applied. Nothing populates the field on a create today, so the request is rejected rather than quietly ignored - ignoring it would leave a future caller believing it had created a versioned bucket. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Each section below was a separate commit; they are folded here so the task's review fixes land as one change. * Address comments * Use a 0x00 separator in versionedKeyTable dbKeys Key names in OBJECT_STORE buckets contain '/' verbatim, so a '/' separator interleaves a key's versions with those of keys nested under it, breaking the single-seek promotion and the merged ListObjectVersions order. * Do not derive a versioning status from the legacy flag Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds VersionIdGenerator, the pluggable source of the id an object version is numbered with, and UniqueIdVersionIdGenerator as the cluster default. The id is proposed on the OM that received the request, in preExecute, so it travels in the replicated request and every OM applies a version that is already numbered. Nothing about it depends on the transaction carrying the write, or on OM being replicated by Ratis. The default numbers a version with the time it was written, through the scheme Ozone already uses for block local IDs: currentTimeMillis << 16 with a 16-bit counter separating ids proposed inside one millisecond. It needs no allocator state and no coordination, which is what makes it safe to read on any OM. The interface has one abstract method, generateVersionId(), plus a default versionIdFor(proposed, hasCurrentVersion) that lets a generator number some versions specially at apply time. The implementation is selected cluster-wide by ozone.om.versioning.version-id-generator. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds VersionIdAllocator, which turns the id proposed for a version into the id it is applied with. versionedKeyTable orders a key's versions by Long.MAX_VALUE - versionId, so the ids of one key have to increase in the order the versions were written. A proposal is a clock reading and cannot promise that: ids proposed inside one millisecond can exhaust the counter separating them, and a leader change onto a lagging clock proposes a lower value. So a proposal is a floor. The applied id is the later of it and the id after the key's current version, which the write path already holds - no read of its own, no global state, and identical on every OM. Under a clock regression an affected key's ids climb by one until proposals overtake them again: the versions stay ordered and only the id's reading as a time degrades. propose() runs in preExecute on the OM that received the request; allocate() runs under the write's lock on every OM. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds PinnedFirstVersionIdGenerator, which numbers versions like the default except that a key's first version takes FIRST_VERSION_ID, so it can be referenced without listing the key's versions first. Whether the key already has a version is not known when the id is proposed, so the generator decides it in versionIdFor, under the write's lock, from the current version the allocator was handed. The sentinel is 1: below every proposed id, so a pinned version sorts at the old end of the key in versionedKeyTable, and above the unset value a pre-versioning record carries. It says nothing about the null version, which carries a proposed id like any other and is marked by isNullVersion. Known trade-off: once every version of a key has been permanently deleted, a recreated key takes the sentinel again, so an external reference to the first version resolves to the new content. The generator is off by default and selected by ozone.om.versioning.version-id-generator. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
On a versioning-enabled bucket a commit no longer reclaims the version it overwrites: the previous current version moves to the versionedKeyTable and the new record becomes current, in one WriteBatch. A record written before versioning was enabled carries no versionId and becomes the key's null version. The new version is numbered from KeyArgs.proposedVersionId, which preExecute stamps into the request on the OM that received it, so every OM applies the same id and nothing is generated during apply. The allocator raises the proposal to come after the key's current version when it does not already. An hsync re-commit keeps updating the version it opened, so it keeps its versionId and moves nothing. Both reclaim branches now depend on the versioning status rather than on the legacy isVersionEnabled flag being kept in sync with it, so dropping that sync cannot strand a version record by reclaiming the blocks it still refers to. S3MultipartUploadCompleteRequest is guarded the same way; it does not yet record a version for the key it supersedes, so an MPU overwrite on a versioned bucket leaks those blocks until T5 lands. OMKeyCommitRequestWithFSO is left alone: isS3VersioningEnabled() requires the OBJECT_STORE layout, so the check is structurally false there. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A delete without a versionId on a versioning-enabled bucket removes no data: a delete marker becomes the key's current version and the version it supersedes moves to the versionedKeyTable. As in S3, the marker is inserted even when the key does not exist. The marker is a version, so its id comes from KeyArgs.proposedVersionId the same way a commit's does, proposed in preExecute and raised to come after the key's current version at apply time. The failure response declares the same tables as the successful one: the double buffer cleans the table cache from the response's CleanupTableInfo, so a marker request that failed after touching the versionedKeyTable cache would otherwise leave an entry behind that is in no DB and never cleaned. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A batch delete is the same operation over more keys, but it goes through a request of its own and that one still hard-deleted on a versioned bucket: the current version's blocks were reclaimed and the versions underneath were left in the versionedKeyTable with nothing in the keyTable above them. Those versions then read as absent, hold quota, and cannot be promoted, because promotion only runs when a version is deleted by id. The gap opens with T3.1, where the two tables first diverge, so it is closed here rather than later. It is reachable today: the S3 gateway wires DeleteObjects straight to this request, and so does the client's deleteKeys API. Rather than write a second marker implementation, the one T3.2 added moves to OMKeyRequest and returns what it changed, so the single-key and batch requests build their own responses from the same insertion. One proposed versionId covers a batch: an id only has to increase within one key, and the applying OM raises it per key against that key's own current version. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
GET/HEAD/lookup can address one version of a key instead of its current one: KeyArgs carries either a versionId or nullVersion, at most one. A versionId naming the current version is answered from the keyTable with no versionedKeyTable read; anything else is a point lookup there. The null version is found by its isNullVersion attribute rather than by id, because it carries a normally proposed id like any other version, so no dbKey addresses it. That search reads the table cache as well as the DB: Table.iterator() goes straight to RocksDB, and a version demoted by a transaction the double buffer has not flushed yet lives only in the cache, so an iterator-only scan would report a version that exists as missing while the point lookups beside it, which do consult the cache, resolve it. A read that lands on a current delete marker is a plain not-found, while one that names a marker by id - current or not - is KEY_IS_DELETE_MARKER. S3 answers such a read with 405 rather than 404, which is why the two conditions stay distinct; the gateway mapping arrives with the endpoints. Only OBJECT_STORE buckets can hold versions, so addressing a version on any other layout is NOT_SUPPORTED_OPERATION. The check is on the layout rather than on isFileSystemOptimized(), because LEGACY reaches the same lookup path as OBJECT_STORE and would otherwise scan the versionedKeyTable and report the key not found. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
DELETE ?versionId= is the only delete that destroys data on a versioned bucket: the version leaves the versionedKeyTable and its blocks go to the deletedTable, which stays the single path through which version blocks are reclaimed. The null slot is addressed by attribute, so it is found by the same bounded prefix scan the read path uses. Addressing the current version is rejected for now: removing it has to promote the next-newest version to keep the keyTable authoritative, which T4.3 adds. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
keyTable holds the current version of every key that still has one, so removing the current version has to hand the place over: one seek on the key's version prefix yields the newest remaining version, which moves back into the keyTable in the same WriteBatch as the delete. The record travels unchanged - promotion is positional, and a version keeps the identity it was created with. When no version survives, the key disappears entirely. Deleting a current delete marker this way is exactly S3's restore-an-object flow: the version the marker superseded becomes current again. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Each section below was a separate commit; they are folded here so the task's review fixes land as one change. * Send the addressed version with the request OmKeyArgs.toProtobuf() left versionId and nullVersion out, so a client that addressed a specific version had the request silently downgraded to a read of the current one. The OM already reads both fields off KeyArgs in the lookupKey and getKeyInfo handlers, and toBuilder() already carries them, so only the proto conversion was missing. Nothing hits this yet because the client and S3 gateway do not address versions themselves, but it would be an awkward bug to find once they do. * Refuse a request that names a version twice S3 names the null version with the literal versionId "null", so exactly one of versionId and nullVersion is set on a well-formed request. Both being set was not refused: the OmKeyArgs builder keeps the two mutually exclusive by dropping one when the other is set, so a read silently resolved to the null version, and the delete path picked whichever field it looked at first. The check now runs where the proto arrives - the lookupKey and getKeyInfo handlers, and the delete request's preExecute - and returns INVALID_REQUEST rather than answering an ambiguous request. * Name the version delete by the key it deletes The dbKey the version is removed from was called versionedKey and left null when the current version was the one addressed, so the response was handed deletingCurrent ? objectKey : versionedKey - the request working out which table the key belongs to, which the response already decides from deletedCurrent when it picks the table to delete from. The variable now holds the key in both cases and is named for what it is. No behaviour change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…nded Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Suspending versioning stops new versions from being created; it does not stop the ones already there from being protected. So a delete still writes a marker, but the marker is the key's null version: it replaces whatever held that slot rather than superseding it, and only that record is destroyed. Versions written while versioning was enabled stay readable and deletable by versionId. Both delete paths take this route. The batch request branched on ENABLED alone, which would have hard-deleted on a suspended bucket and stranded the versions underneath - the same data loss the single-key path was changed to avoid, just in a different bucket state. Its response now also writes out the null slot the marker replaced: dropping the versionedKeyTable record and queueing its blocks, which the quota accounting has already been charged for. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Enabling versioning does not rewrite the objects a bucket already holds, and S3 reports the version of such an object as "null". A record written before versioning carries no versionId at all, while one written while versioning was suspended carries the isNullVersion flag; a request naming version "null" addresses either, so the sites that resolve it match on isNullVersionRecord() rather than on the flag alone - the read path, the search of a key's noncurrent versions, and deleteVersion's check for whether the addressed version is the current one. A key still has at most one null version either way, since a suspended write replaces whichever one the key already has. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Completing a multipart upload creates a version like any other write: the version it supersedes is kept in the versionedKeyTable instead of being reclaimed, except for the null version, which a suspended write replaces outright. This closes the block leak an MPU overwrite left on a versioned bucket, where the previous version's blocks were neither reclaimed nor recorded. The new version is numbered from KeyArgs.proposedVersionId, stamped in preExecute like a commit's, and is marked as the null version while versioning is suspended. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Each section below was a separate commit; they are folded here so the task's review fixes land as one change. * Check the quota against what a suspended write nets A suspended write that replaces the key's null version removes a record as it adds one, but the quota was checked before the removal was accounted for: the namespace check asked for one record and the byte check for the new record's full size, and only afterwards were the replaced version's usage and slot given back. A bucket sitting exactly at its namespace quota therefore rejected a write that leaves its record count unchanged, and one near its space quota rejected a write that leaves it smaller. The replaced version is now resolved before the checks and folded into both deltas, so what is checked is what the transaction nets. The lookup moving ahead of the retention below is safe: a null version record is never retained, so the record retained there always carries a real versionId and cannot be what the lookup finds. * Delete the null version of a never-versioned bucket S3 calls the object itself the null version wherever a bucket has no version history, so DELETE ?versionId=null works on a bucket that was never versioned and removes the object. The read path already answers it that way - it gates on the bucket layout, not on the versioning status - while the delete refused it as unsupported, so the same request could be read but not deleted. It now falls through to the plain delete. An id still names nothing on such a bucket and stays unsupported, and the layout gate the read path applies is now explicit here rather than implied by the versioning status never being set outside OBJECT_STORE. * Check the marker's quota against what it nets A delete marker that takes the key's null version slot replaces a record rather than adding one, but the namespace check asked for a record before the replaced version's slot was given back, so a bucket sitting exactly at its namespace quota rejected a delete that leaves its record count unchanged. This is the same ordering as on the commit path, in the delete marker path that OMKeyDeleteRequest and OMKeysDeleteRequest share. It covers the marker replacing a noncurrent null version too, not only the current one: the current version is demoted in that case, so the marker still nets no record. * Build the delete marker from a stated set of fields The marker was a copy of the version it supersedes with the content cleared out afterwards, so it inherited everything OmKeyInfo happens to carry - encryption info, storage policy, parent and file name - and a field added later would land on markers by default. It is now built once from the fields a marker is meant to have: the key's identity - objectID, owner, replication and ACLs - and nothing describing content it does not hold. * Note why a marker can have no version to inherit from * Take the replaced null version in the constructor The delete marker response was completed by a call after construction, so a caller that forgot it got a response that silently skipped removing the replaced null version and queueing its blocks. There is one call site and no subclass, so the two fields are constructor arguments now. OMKeyCommitResponse keeps its setter: OMKeyCommitResponseWithFSO passes the constructor through, and an FSO bucket can never be versioned, so the field would only ever be null there. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Version reclamation is expressed as rules on the lifecycle engine already in master rather than as a service of its own, so the rule model has to carry the two S3 actions that act on versions. NoncurrentVersionExpiration reclaims a key's noncurrent versions by age (NoncurrentDays), by count (NewerNoncurrentVersions, bounded at 100 as in S3), or by both; a version goes once either limit says so. The current version is never a candidate, which is what separates it from Expiration. Expiration gains ExpiredObjectDeleteMarker, which removes a delete marker once it is the key's only remaining version. It removes a marker rather than expiring an object by age, so it is exclusive with Days and Date - the three-way check replaces the previous either-days-or-date one. Both travel through the proto, the client model and the S3 gateway's XML binding, so a rule carrying them survives Put, Get and Delete like any other. Nothing acts on them yet: T6.2 adds the scan that selects noncurrent versions, and T6.3 through T6.5 the rest. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds the second scan the versioning actions need. The versionedKeyTable holds a key's versions adjacent and newest first, so one pass over a bucket's prefix visits them in the order the rules reason about: position N is the Nth newest noncurrent version, and the moment a version stopped being current is the moment the version above it was written. For the newest noncurrent version that is the key's current version, which the keyTable holds - so NoncurrentDays needs no new field on the record. Reclamation goes through a request of its own rather than DeleteKeys, which addresses keys by name and cannot name a version. The request re-reads each version under the bucket lock and skips the ones that were permanently deleted or promoted since the scan selected them, deducting quota for those it does remove and queueing their blocks in the deletedTable; a version holding no blocks releases namespace only. A pass is bounded by the versions it reads rather than the ones it selects, so a bucket whose versions all survive still yields between passes instead of being walked to the end in one, and the scan reports the work it did whether or not anything expired. It may only stop at a key boundary: resuming mid-key would restart the per-key count and keep more versions than the rule allows. Scan state gets a field of its own, lastScannedVersionKey, because this scan walks a different table than lastScannedKey's, and it is saved on the same throttle the keyTable scan uses - resuming from an older boundary only re-selects versions the reclaim request then skips as already gone. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
On a bucket that has ever been versioned, Expiration.Days and .Date hide the object behind a delete marker instead of deleting it, as S3 does. The scan already reaches that behaviour through the batch delete request, which inserts markers on such a bucket, so what is left here is the part the scan itself decides. A key whose current version is already a delete marker is skipped. The marker carries the time it was written, so it ages into the same rule that created it; expiring it would demote it under a new marker and the chain would grow for as long as the rule exists. The object is already gone from an unversioned read - removing the marker that hides it is ExpiredObjectDeleteMarker's decision, in T6.4. The run accounting follows. Inserting a marker deletes nothing and frees no space: the version that was current is demoted, not reclaimed. Counting it as a key deleted with its bytes released would report space that is still held, so those runs report markers inserted instead. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ExpiredObjectDeleteMarker is the action that ends a key's life on a versioned bucket. A marker that is the key's only remaining version hides nothing, and no versionId addresses it, so without this nothing would ever remove it and the key would sit in the keyTable holding a namespace slot forever. "Expired" here is not about the marker's own age, unlike every other Expiration condition: a marker is expired exactly when it is the last version left. While a noncurrent version survives, removing the marker would promote that version and bring back an object the user deleted, so the marker has to stay. That condition is checked twice. The scan checks it so it does not submit markers that will be refused; the request checks it again under the bucket lock, which is what decides, since a version can be written between the two. The request also refuses a marker a write has superseded - the key's current version is then a real object. Removal goes through the reclaim request rather than DeleteKeys, which on a versioned bucket answers a delete by inserting another marker. The marker leaves the keyTable and the key disappears with it; it holds no blocks, so nothing is queued in the deletedTable and only namespace is released. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ning A lifecycle rule that never runs is worse than no rule, and versioning is the first feature that depends on reclamation for correct operation rather than for tidiness: it is the only thing that ever removes a version. ozone.lifecycle.service.enabled flips to true. OM refuses to enable versioning on a bucket while the service is off, on both the create and the set-property path. This does not promise that versions are bounded - a bucket whose owner writes no rule accumulates them either way, exactly as on S3 - but it rules out the state where a rule exists and nothing will ever act on it. The check runs in preExecute. Whether the service runs is a property of the OM that received the request rather than of the replicated state - the flag comes from that OM's configuration and is never written to the DB - so only that OM may decide it, and what travels to the others is its decision. A replica re-deciding in validateAndUpdateCache could reach a different answer and diverge. Suspending is not refused: it stops new versions from being created, so it can only reduce what has to be reclaimed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Each section below was a separate commit; they are folded here so the task's review fixes land as one change. * Find the versionId suffix from the end of the DB key A versionedKeyTable DB key is the key name, the separator, and a fixed-width hex versionId. Searching forward for the separator assumes the key name holds none itself, and OBJECT_STORE key names are not validated against that. A key whose name contains a NUL would be cut short, and the truncated name would then read as a key boundary, so the scan would count versions and pick an expiry deadline per fragment rather than per key. The separator is the last one in the DB key: what follows it is hex only. Searching backwards costs nothing here, since it stops at the sixteenth character. * Stamp when a version stopped being current A NoncurrentVersionExpiration rule counts from the moment a version became noncurrent, which the selector derived from the neighbouring records: walking a key newest first, each version was dated by the modification time of the version above it, and the newest noncurrent version by the current version in the keyTable. That derivation moves. Removing the version that superseded a record - by a version delete, or by the reclaim pass itself - leaves the record dated against whatever is above it next, restarting its clock. S3 fixes the moment when it happens, so a version that has been noncurrent for 90 days stays 90 days noncurrent whatever happens to the versions above it. Store it on the record instead. Every path that moves a version out of the keyTable stamps it with the write that superseded it, and promoting a version back to current clears it. The keyTable read the derivation needed per key goes away with it. * Name the record the scan selected, not its dbKey The reclaim request carried only dbKeys, and the checks it re-ran on apply asked whether the record there could be reclaimed, never whether it was the record the rules expired. A dbKey names a position: scan and apply are separated by replication, and the key can be rewritten in between. For a delete marker the position is the plain key name, so it holds whichever version of the key is current. Deleting a key, writing it and deleting it again leaves another marker there - a marker, with nothing surviving under it, and written before this transaction - which every check accepted. The reclaim then removed a marker the user had just created and that no rule had ever expired. A version's dbKey does carry its versionId, but a versionId is allocated against the key's current version alone, so one that was permanently deleted can be handed out again and put a different version where the scan looked. Carry the updateID of the record the scan read and require it unchanged on apply. It subsumes the previous check that the record was not written by a later transaction, which was the same guard stated as an inequality. * Look for a surviving version in the cache as well Before removing a key's expired delete marker the reclaim re-checks that no version survives under it, because removing it while one does would promote that version back to current and resurrect an object the user deleted. That check iterated the versionedKeyTable, and an iterator reads RocksDB alone. So a version demoted by a transaction the double buffer has not flushed yet was invisible to it, and the check answered that nothing survived. The keyTable read a few lines above does consult the cache, so the two reads were taken on different views of the same moment. NoncurrentVersions searches the cache and the DB together for exactly this reason; the check now goes through it. The scan that selects the marker keeps its plain iteration: both ways it can be wrong there are safe. A stale version it still sees keeps the marker, and one it misses is caught by this check before anything is removed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Please only review commits start with "T6".
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-16093
How was this patch tested?
unit test