Skip to content

About

Zarr convention for writing bytes in attributes as base64 strings

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Bytes Attributes Convention

Description

This convention defines how a sequence of bytes is written in the attributes of a Zarr v3 array or group. It is written as a string: the prefix base64: followed by the bytes in base64. A string that has the same form is written with the prefix _str_, so that every value can be read back as it was written.

It is a companion to the Non-Finite Attributes convention, and uses the same escape. Like that convention, it does not define a set of attributes. It defines how the values of a node's attributes are encoded, and so it has no properties of its own and no namespace.

Motivation

Zarr metadata is JSON as defined by RFC 8259. A JSON string is a sequence of Unicode characters, so JSON has no representation for bytes, and no way to hold bytes that are not valid text. Formats that Zarr is used to store or reference do have them. An HDF5 attribute can hold fixed-length or variable-length byte strings in any encoding, or opaque binary data such as a digest, a header, or a serialized structure. To write such an attribute in Zarr, a converter has to decode it as text, which fails or changes the value when the bytes are not valid UTF-8, or drop it.

This convention gives writers one encoding to agree on:

  • Valid JSON: every implementation can open the node, whether or not it knows the convention.
  • Exact: every byte sequence and every string is read back as it was written, including strings that look like the encoded form.
  • Familiar: base64: is the prefix kerchunk uses for inline binary data, and base64 is decoded by the standard library of most languages.
  • Explicit: only a node that registers the convention is decoded, so the meaning of existing data does not change.

Convention Registration

The convention must be registered in zarr_conventions on every array or group whose attributes are encoded with it:

{
  "zarr_conventions": [
    {
      "schema_url": "https://raw.githubusercontent.com/hdmf-dev/zarr-bytes-attributes/refs/tags/v1/schema.json",
      "spec_url": "https://github.com/hdmf-dev/zarr-bytes-attributes/blob/v1/README.md",
      "uuid": "2de6770a-9c00-4432-9e84-dd9e1dfed441",
      "name": "bytes-attributes",
      "description": "Bytes in attributes are written as base64 strings"
    }
  ]
}

Registration applies to the node that holds it. It does not extend to the members of a group, so each node that needs the convention registers it.

Applicable To

This convention can be used with these parts of the Zarr hierarchy:

  • Group
  • Array

Encoding

The tag is base64:. An affected string is a string that consists of zero or more repetitions of the prefix _str_, followed by the tag, followed by any characters. The comparison is case sensitive. Every other string is unaffected.

The payload of a sequence of bytes is its base64 encoding in the standard alphabet, with padding and no other characters (RFC 4648, section 4). Every sequence of bytes has exactly one payload, so the encoding is canonical. An empty sequence has an empty payload.

When writing the attributes of a node that registers the convention, an implementation MUST apply these rules to every value:

  1. A sequence of bytes is written as the tag followed by its payload.
  2. An affected string is written with one additional _str_ prefix.
  3. Any other value is written unchanged.

When reading the attributes of a node that registers the convention, an implementation MUST apply the inverse rules to every value:

  1. A string that begins with the tag is read as the sequence of bytes whose payload follows the tag. If what follows is not the payload of any sequence of bytes, the value is invalid, and the reader MUST report an error for it. It MUST NOT guess at a value.
  2. An affected string that has one or more _str_ prefixes is read as a string, with one prefix removed.
  3. Any other value is read unchanged.
Attribute value Written in zarr.json
the bytes 00 FF "base64:AP8="
the bytes 61 62 63 "base64:YWJj"
the empty sequence of bytes "base64:"
the string base64:AP8= "_str_base64:AP8="
the string base64: "_str_base64:"
the string _str_base64:AP8= "_str__str_base64:AP8="
the strings Base64:AP8=, AP8= "Base64:AP8=" and "AP8="
the string _str_hello "_str_hello"
the string NaN "NaN" (see below)

Which values count as a sequence of bytes depends on the language. In Python they are bytes, bytearray and memoryview, which includes numpy.bytes_. In JavaScript they are Uint8Array.

Which Values the Rules Apply To

The rules apply to every value within attributes, including the members of objects and the elements of arrays at any depth, with these exceptions:

  • The value of zarr_conventions is written and read as it is.
  • The names of the members of an object are never changed.

The rules do not apply to any other part of zarr.json.

Use with Other Conventions

Conventions like this one, which encode values, are applied when attributes are read and written. Conventions that interpret attributes are applied to the decoded values. A reader therefore decodes with every registered value-encoding convention before applying the others, and a writer encodes after.

This convention and the Non-Finite Attributes convention can be registered on the same node, and can be applied in either order:

  • No string is affected by both. An affected string of the other convention ends in NaN, Infinity or -Infinity after its _str_ prefixes, and one of this convention begins with base64: after them.
  • Both add and remove the same _str_ prefix, and each only on its own affected strings.

So each convention changes only its own values and its own affected strings, and leaves the other's unchanged. A reader that knows only one of the two decodes that convention's values and returns the other's as they are written, so a NaN comes back as the string "NaN" and an escaped string keeps its prefix. In this example, a NaN, a sequence of bytes and the string NaN are written as "NaN", "base64:ECAw" and "_str_NaN".

Use for HDF5 Attributes (Informative)

h5py reads a fixed-length string attribute as numpy.bytes_, even when it is declared as UTF-8 and holds text. It reads a variable-length string attribute as str. A converter should decode a string attribute that is declared as text in HDF5 (ASCII or UTF-8) and decodes cleanly, and write it as a JSON string. It should use this convention for attributes whose bytes are not valid text in their declared encoding, and for opaque binary data, which HDF5 stores as byte strings or as uint8 arrays.

Reference Implementations and Test Vectors

This repository has reference implementations in Python and JavaScript, and test_vectors.json lists values with their encodings, and encodings a reader must reject, for checking other implementations. Both reference implementations are tested against the vectors and the examples. The Python test also checks, on random nested values, that this convention and the Non-Finite Attributes convention give the same result in either order.

npm install
npm test
python reference/test_reference.py

Limitations

  • A reader that does not know the convention gets strings. The node opens and the other attributes are unaffected, but a sequence of bytes is returned as a string that begins with base64:.
  • The size grows by a third. Base64 writes four characters for every three bytes. Large binary data belongs in an array, not an attribute.
  • The source data type is not kept. A fixed-length and a variable-length byte string are written the same way, and a uint8 array written as bytes is read back as bytes, not as an array of integers.
  • Registration is repeated on each node. A hierarchy with many nodes that hold bytes attributes carries the convention metadata object on each of them.
  • Existing data is not covered. A node that does not register the convention is read as it always was, so data written with base64: strings and no registration stays ambiguous.

Related Work

About

Zarr convention for writing bytes in attributes as base64 strings

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages