Skip to content

fix: return the caller's out from decode(buf, out=out) in every codec - #907

Merged
d-v-b merged 3 commits into
zarr-developers:mainfrom
d-v-b:typing/02-out-contract
Nov 21, 2026
Merged

d-v-b merged 3 commits into
zarr-developers:mainfrom
d-v-b:typing/02-out-contract

Conversation

@d-v-b

@d-v-b d-v-b commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

🤖 AI text below 🤖

decode(buf, out=out) now returns out itself for every codec, as the Codec docstring describes. Zlib, BZ2, LZMA, Base64, Shuffle, ZFPY, PCodec, the three VLen codecs, and every codec built on numcodecs.compat.ndarray_copy returned a flattened view of out instead, or a new ndarray when out was a bytearray. The decoded bytes were already written into out; only the returned object changes.

Summary
  • The fix: ndarray_copy and the ten codecs listed above rebound out to a flattened view and returned that view. They now return the caller's object. When the agent probed every codec, the bytes always landed in out; only the identity of the return value was wrong.
  • JenkinsLookup3: decode used to call out.view(...), so only an ndarray worked as out. It now accepts any contiguous writable buffer and raises ValueError when out has the wrong size.
  • Fletcher32: decode without out returns a uint8 ndarray, like the other checksum codecs. It used to return a Cython _memoryviewslice, which is not a memoryview subclass, so no annotation could name it.
  • Tests: all strict xfails in tests/test_codec_contract.py are removed, and there is a new test for the JenkinsLookup3 size check.

Overlap: #891 also validates JenkinsLookup3's out. Whichever lands second needs a small rebase.

Tests

Full suite passes locally (982 passed); the contract tests pass with no xfails.

Stack and merge order

The agent split honest type annotations for numcodecs into a stack of 10 PRs, following an audit of the existing annotations and of the earlier attempts (#443, #698, #701, #702). Each PR is opened from the fork against main, so until the PRs before it merge, its diff also contains their commits. The change specific to this PR starts at commit 145994e; later commits on its branch are follow-up fixes and merges of the PR below it. Merge in this order:

  1. test: check the Codec contract for every codec and add static type tests #906 typing/01-harness: test: check the Codec contract for every codec and add static type tests
  2. fix: return the caller's out from decode(buf, out=out) in every codec #907 typing/02-out-contract: fix: return the caller's out from decode(buf, out=out) in every codec ← this PR
  3. fix(typing): correct registry, compat and NDArrayLike annotations and add __all__ #908 typing/03-fix-annotations: fix(typing): correct registry, compat and NDArrayLike annotations and add all
  4. feat(abc): annotate the Codec base class and add structural codec protocols #909 typing/04-base-class: feat(abc): annotate the Codec base class and add structural codec protocols
  5. feat(typing): ship .pyi stubs for the Cython extension modules #910 typing/05-cython-stubs: feat(typing): ship .pyi stubs for the Cython extension modules
  6. feat(typing): annotate the compressor codecs with per-codec config TypedDicts #911 typing/06a-compressors: feat(typing): annotate the compressor codecs with per-codec config TypedDicts
  7. feat(typing): annotate the checksum codecs and make JenkinsLookup3 configs JSON #912 typing/06b-checksums: feat(typing): annotate the checksum codecs and make JenkinsLookup3 configs JSON
  8. feat(typing): annotate the filter codecs #913 typing/06c-filters: feat(typing): annotate the filter codecs
  9. feat(typing): annotate the object and array-to-bytes codecs #914 typing/06d-object-codecs: feat(typing): annotate the object and array-to-bytes codecs
  10. feat(typing): ship py.typed and check numcodecs with strict mypy #915 typing/07-py-typed: feat(typing): ship py.typed and check numcodecs with strict mypy

🤖 Generated with Claude Code

Add tests/test_codec_contract.py, which runs every registered codec through
the behavior the Codec base class promises: encode and decode return buffers,
decode(out=...) returns the caller's out, and get_config round-trips through
from_config. Known violations are strict xfails, so fixing one forces its
entry to be removed.

Add tests/typing, where assert_type checks (verified by mypy and pyright in
strict mode) sit next to runtime assertions (verified by pytest), and a
`typing` CI job that runs both checkers plus the repository's own mypy
configuration. The type checkers are pinned in a new `typing` dependency
group.

Assisted-by: ClaudeCode:claude-opus-5-5
The Codec docstring says decode writes into `out`, and callers expect `out`
back. Ten codecs (Zlib, BZ2, LZMA, Base64, Shuffle, ZFPY, PCodec and the
three VLen codecs) rebound `out` to a flattened view and returned that, and
numcodecs.compat.ndarray_copy did the same for any `out` that was not an
ndarray, which every filter and checksum codec inherited. The decoded bytes
were already in the caller's buffer; only the returned object changes.

JenkinsLookup3.decode now accepts any contiguous writable buffer as `out`
and checks its size, and Fletcher32.decode returns a uint8 ndarray instead of
a Cython memoryview slice that no type annotation could name.

The strict xfails in tests/test_codec_contract.py are removed.

Assisted-by: ClaudeCode:claude-opus-5-5
@codecov

codecov Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (28983da) to head (cffb3e4).

Additional details and impacted files
@@            Coverage Diff            @@
##              main      #907   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           27        27           
  Lines          921       922    +1     
=========================================
+ Hits           921       922    +1     
Files with missing lines Coverage Δ
src/numcodecs/base64.py 100.00% <100.00%> (ø)
src/numcodecs/bz2.py 100.00% <100.00%> (ø)
src/numcodecs/checksum32.py 100.00% <100.00%> (ø)
src/numcodecs/compat.py 100.00% <100.00%> (ø)
src/numcodecs/lzma.py 100.00% <100.00%> (ø)
src/numcodecs/pcodec.py 100.00% <100.00%> (ø)
src/numcodecs/shuffle.py 100.00% <100.00%> (ø)
src/numcodecs/zfpy.py 100.00% <100.00%> (ø)
src/numcodecs/zlib.py 100.00% <100.00%> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@d-v-b
d-v-b merged commit 1d2229f into zarr-developers:main Nov 21, 2026
36 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant