Repository navigation
Conversation
…fines from one normal form Each of the six models writes its document in its simplest equivalent spelling in one function. The key that == and hash compare is that document as JSON text with the claims the scope made, canonicalize reads it, and refines reads the other model's document in this model's scope and compares. The per-model key functions, the per-model refines traversals and the declaration-level refines are removed. A model no longer refines one whose document its scope reads with a problem, such as a configuration key that the definition does not declare under a name the other scope left unclaimed. Assisted-by: ClaudeCode:claude-fable-5-2
…for every generated document Carries the tests of zarr-developers#4648 that do not name codec_refines, which this branch does not have. Assisted-by: ClaudeCode:claude-fable-5-2
Assisted-by: ClaudeCode:claude-fable-5-2
Owner
Author
|
🤖 AI text below 🤖 Closed by the agent in favour of zarr-developers#4648, which now holds this design on its own branch. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
refineson every model now reads the other model's document in this model's scope and compares the two, so a document read in a richer scope refines the same document read in a poorer one, whatever the richer scope learns of the chunk each codec receives. A model no longer refines one whose document its own scope reads with a problem: agzipcodec with a configuration key its definition does not declare, read in a scope that claims nogzip, is not refined by the model of the same codec without that key.Why this exists: a prototype of one normal form per model
This is a prototype for a decision, not a PR meant to merge as it is. It answers whether the defects fixed in zarr-developers#4641, zarr-developers#4647 and zarr-developers#4648 share a cause that a smaller design removes.
Before this branch, what a document means was decided in several separately written walks over the same tree, kept in agreement by hand: the key that
==andhashcompare (array_key_of, each model's_key_of,extra_fields_key), the canonical document (canonical_array_document_v3and five others), and refinement (sixrefinesmethods, the declaration-levelrefines,_gained,own_key, and in zarr-developers#4648codec_refines). Each of the three defects was one walk missing something another walk had.On this branch each model writes its document in its simplest equivalent spelling in one function,
_canonical_document, and everything else is derived inmodel/_keyed.py:==andhashcompare the claims the scope made and the normal form as JSON text.canonicalize()reads the normal form.refines(other)holds when this scope read every name the other scope read, with the same definition, and the other document read in this scope equals this model.Member order means nothing by construction, because JSON text sorts the members of an object. The cross-scope case of zarr-developers#4648 holds by construction, because one reader, with one scope's knowledge of the chunks, reads both documents.
What changed in the source
Against the base (the branches of zarr-developers#4642 and zarr-developers#4647 merged), the source is 201 lines added and 354 removed. Against the three drafts merged (zarr-developers#4642, zarr-developers#4647, zarr-developers#4648) it is 211 added and 452 removed.
Removed:
ZarrV3ArrayMetadata._plain_key,ZarrV2ArrayMetadata._plain_keyand_fill_value_key,extra_fields_key,ZarrV2ConsolidatedMetadata._other_entries_text,_node_refines,_v2_node_refines, the body of each model's_key_ofandrefines, and fromdefinition/_scope.pythe declaration-levelrefines,_gainedand_Claimed, withown_keyfromdefinition/_definition.py. None of these was exported from a public module.Not added:
codec_refinesandread_as_claimedfrom zarr-developers#4648. The publiccodec_refinesis not needed on this design.One change outside the models:
_canonical_declarationno longer reads a configuration again when the definition'scanonicalreturned what it was given. The key now goes through that function, and without this a listing called each definition's rules more than once per declaration.Behaviour difference found
refinesused to judge a gain by reading the unclaimed declaration again and dropping the problems of that read. So the model of{"name": "gzip", "configuration": {"level": 5}}refined a model, read in a scope without a gzip, of the same codec with an extra configuration key.refined_inon that second model raisesMetadataValidationError, so the two methods disagreed. On this branchrefinesreturnsFalsethere.tests/v3/test_scope.pyhas the case asno-gain-over-a-problem.Tests
tests/v3/test_scope.pythat called the removed declaration-levelrefines. Those are ported to models with the same cases. Two cases have no model equivalent and were dropped, because they compare refused declarations and no model holds one.codec_refines: the sixrefined_incases,tests/model/test_invariants.py, and the additions totests/v3/test_definitions.py.==andrefinesanswers as the fix(zarr-metadata): derive each model's key, canonicalize and refines from one normal form zarr-developers/zarr-python#4648 branch: 2,810 equal pairs and 6,187 refinements.==of two sharded arrays 103 µs against 101 µs, a firstrefinesacross scopes 215 µs against 195 µs, a first==of two listings of 200 arrays 15.3 ms against 15.2 ms.What this prototype does not do
declaration_key, next to_canonical_declaration. Models no longer use it, butAcceptedDeclaration.__eq__does. Deriving it from the same walk needs a normal form for refused declarations.canonicalis still called from three places (declaration_key,_canonical_declaration, and_asked_for_chunkin the pipeline). Folding the per-chunk step into_canonical_declarationwould removerespelledand the checks fix(zarr-metadata): simplify a codec for its chunk in its canonical spelling, and check the result zarr-developers/zarr-python#4647 added around it, and would change whatcodec_for_chunkreturns, which is unreleased.Stack
Base branch
integration/zarr-metadata-4642-4647is the branch of zarr-developers#4642 with the branch of zarr-developers#4647 merged into it, so this diff shows only the prototype. If this design is adopted, it replaces zarr-developers#4648, and it would be reopened upstream after zarr-developers#4642 and zarr-developers#4647 merge.🤖 Generated with Claude Code