Skip to content

refactor(zarr-metadata): derive each model's key, canonicalize and refines from one normal form - #465

Closed
d-v-b wants to merge 3 commits into
integration/zarr-metadata-4642-4647from
refactor/zarr-metadata-one-normal-form
Closed

d-v-b wants to merge 3 commits into
integration/zarr-metadata-4642-4647from
refactor/zarr-metadata-one-normal-form

Conversation

@d-v-b

@d-v-b d-v-b commented Nov 18, 2026

Copy link
Copy Markdown
Owner

🤖 AI text below 🤖

refines on every model now reads the other model's document in this model's scope and compares the two, so a document read in a richer scope refines the same document read in a poorer one, whatever the richer scope learns of the chunk each codec receives. A model no longer refines one whose document its own scope reads with a problem: a gzip codec with a configuration key its definition does not declare, read in a scope that claims no gzip, is not refined by the model of the same codec without that key.

Why this exists: a prototype of one normal form per model

This is a prototype for a decision, not a PR meant to merge as it is. It answers whether the defects fixed in zarr-developers#4641, zarr-developers#4647 and zarr-developers#4648 share a cause that a smaller design removes.

Before this branch, what a document means was decided in several separately written walks over the same tree, kept in agreement by hand: the key that == and hash compare (array_key_of, each model's _key_of, extra_fields_key), the canonical document (canonical_array_document_v3 and five others), and refinement (six refines methods, the declaration-level refines, _gained, own_key, and in zarr-developers#4648 codec_refines). Each of the three defects was one walk missing something another walk had.

On this branch each model writes its document in its simplest equivalent spelling in one function, _canonical_document, and everything else is derived in model/_keyed.py:

  • == and hash compare the claims the scope made and the normal form as JSON text.
  • canonicalize() reads the normal form.
  • refines(other) holds when this scope read every name the other scope read, with the same definition, and the other document read in this scope equals this model.

Member order means nothing by construction, because JSON text sorts the members of an object. The cross-scope case of zarr-developers#4648 holds by construction, because one reader, with one scope's knowledge of the chunks, reads both documents.

What changed in the source

Against the base (the branches of zarr-developers#4642 and zarr-developers#4647 merged), the source is 201 lines added and 354 removed. Against the three drafts merged (zarr-developers#4642, zarr-developers#4647, zarr-developers#4648) it is 211 added and 452 removed.

Removed: ZarrV3ArrayMetadata._plain_key, ZarrV2ArrayMetadata._plain_key and _fill_value_key, extra_fields_key, ZarrV2ConsolidatedMetadata._other_entries_text, _node_refines, _v2_node_refines, the body of each model's _key_of and refines, and from definition/_scope.py the declaration-level refines, _gained and _Claimed, with own_key from definition/_definition.py. None of these was exported from a public module.

Not added: codec_refines and read_as_claimed from zarr-developers#4648. The public codec_refines is not needed on this design.

One change outside the models: _canonical_declaration no longer reads a configuration again when the definition's canonical returned what it was given. The key now goes through that function, and without this a listing called each definition's rules more than once per declaration.

Behaviour difference found

refines used to judge a gain by reading the unclaimed declaration again and dropping the problems of that read. So the model of {"name": "gzip", "configuration": {"level": 5}} refined a model, read in a scope without a gzip, of the same codec with an extra configuration key. refined_in on that second model raises MetadataValidationError, so the two methods disagreed. On this branch refines returns False there. tests/v3/test_scope.py has the case as no-gain-over-a-problem.

Tests
What this prototype does not do
Stack

Base branch integration/zarr-metadata-4642-4647 is the branch of zarr-developers#4642 with the branch of zarr-developers#4647 merged into it, so this diff shows only the prototype. If this design is adopted, it replaces zarr-developers#4648, and it would be reopened upstream after zarr-developers#4642 and zarr-developers#4647 merge.

🤖 Generated with Claude Code

…fines from one normal form

Each of the six models writes its document in its simplest equivalent
spelling in one function. The key that == and hash compare is that
document as JSON text with the claims the scope made, canonicalize reads
it, and refines reads the other model's document in this model's scope
and compares. The per-model key functions, the per-model refines
traversals and the declaration-level refines are removed.

A model no longer refines one whose document its scope reads with a
problem, such as a configuration key that the definition does not
declare under a name the other scope left unclaimed.

Assisted-by: ClaudeCode:claude-fable-5-2
…for every generated document

Carries the tests of zarr-developers#4648 that do not name
codec_refines, which this branch does not have.

Assisted-by: ClaudeCode:claude-fable-5-2
Assisted-by: ClaudeCode:claude-fable-5-2
@d-v-b

d-v-b commented Nov 18, 2026

Copy link
Copy Markdown
Owner Author

🤖 AI text below 🤖

Closed by the agent in favour of zarr-developers#4648, which now holds this design on its own branch.

@d-v-b d-v-b closed this Nov 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant