Skip to content

test: ignore the numcodecs Pickle deprecation in pickle opt-in tests - #468

Open
d-v-b wants to merge 4 commits into
mainfrom
test/filter-pickle-deprecation
Open

d-v-b wants to merge 4 commits into
mainfrom
test/filter-pickle-deprecation

Conversation

@d-v-b

@d-v-b d-v-b commented Nov 21, 2026

Copy link
Copy Markdown
Owner

🤖 AI text below 🤖

Ignore numcodecs' Pickle deprecation warning in the tests that construct the pickle codec on purpose. Without this, those tests fail under filterwarnings = "error" once numcodecs ships the deprecation (zarr-developers/numcodecs#905).

What changes
  • feat(pickle): deprecate the Pickle codec, warning on construction zarr-developers/numcodecs#905 makes numcodecs.Pickle.__init__ emit DeprecationWarning("The Pickle codec is deprecated and will be removed in numcodecs 0.19.0. ..."). That covers direct construction and get_codec when array metadata names pickle.
  • Six test cases in tests/test_v2.py construct the codec to exercise the array.allow_unsafe_pickle_decode opt-in from feat: require opt-in to read data encoded with the pickle codec zarr-developers/zarr-python#4465: the five that call _pickle_compressed_array() (one is parametrized twice), plus test_pickle_codec_read_does_not_unpickle, which opens metadata with compressor: {"id": "pickle"}.
  • The agent added one module-level mark, ignore_pickle_deprecation = pytest.mark.filterwarnings("ignore:The Pickle codec is deprecated:DeprecationWarning"), and applied it to those tests only. It did not add a global pyproject.toml entry, so a new Pickle use elsewhere still fails loudly.
  • The filter matches nothing on numcodecs releases that do not warn, so it is safe across the supported range (numcodecs>=0.16.6,!=0.17.0).
  • No source code changes. zarr itself never constructs Pickle outside metadata-driven get_codec.
Why now

The "upstream" test environment installs numcodecs from git main, so it starts failing as soon as zarr-developers/numcodecs#905 merges, before any numcodecs release.

Verification

The agent tested against numcodecs built from the zarr-developers/numcodecs#905 branch:

  • Without this change, pytest tests/test_v2.py -k pickle gives 6 failed, 1 passed. The failures are exactly the six cases listed above, all on the deprecation warning.
  • With it, tests/test_v2.py passes (35 tests).
  • The full suite (pytest tests -n 8, Python 3.13) gives 13209 passed, 1306 skipped, 9 xfailed, 0 failed.
  • ruff check, ruff format, codespell and numpydoc-validation (via pre-commit) pass. mypy tests/test_v2.py passes.
Notes for upstreaming

🤖 Generated with Claude Code

bendichter and others added 4 commits November 18, 2026 09:46
…zarr-developers#4611)

* Narrow partial reads of uncompressed chunks within a single row

When a selection touches one index along the first axis, apply the same
row window along the next axis, and so on. arr[t, y0:y1, x0:x1] then
fetches rows y0 to y1 of frame t instead of the whole frame, and a
single item is fetched for a point.

The fetched block keeps the narrowed axes with length 1, so the shifted
selection applies to it unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Add release note for zarr-developers#4611

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
…duplicating writes (zarr-developers#4649)

* perf(zarr-indexing): group chunk points with an unstable sort and stop deduplicating writes

Planning grouped each chunk's points with a stable sort so that every
`IndexedSet` and `JointSet` row kept request order, and the writer relied
on that order to make the last repeated write win. Both promises are
dropped: the order of points within a row is unspecified, and a source
cell a write addresses more than once receives one of its values, whole,
without saying which. Callers who need a particular value remove repeats
from the selection first.

Grouping now uses the default sort behind `_group_by_key`, which also
skips the sort for already-grouped joint keys, and the writer no longer
deduplicates targets before its single fancy assignment or `np.put`.

Tests that assumed last-wins now accept any written value, and a new test
runs orthogonal, vectorized and mixed repeated writes with the grouping
replaced by one that reverses each row.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(zarr-indexing): add changelog fragment for unspecified repeated-write order

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(zarr-indexing): number the changelog fragment for upstream PR 4649

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(zarr-indexing): group chunk keys with a 16-bit radix sort

NumPy's stable sort is a linear-time radix sort for integers of at most
16 bits. `_group_by_key` shifts keys to start at zero and sorts them one
16-bit digit at a time: one pass when their range fits 16 bits, two
least-significant-first passes when it fits 32 bits, and the default
sort beyond. Grouping 1M random keys over 65,536 chunks drops from about
46 ms to 2 ms. The order within a group stays unspecified.

The approach follows zarr-python zarr-developers#4643's `argsort_chunk_indices`, with a
shift by the minimum key because grids may have negative chunk IDs.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(zarr-indexing): radix-sort chunk keys of any width

`_group_by_key` fell back to the default sort above a 2**32 key range,
which was slower than radix passes and, as zarr-python zarr-developers#4643 measured,
slower at every key width. It now takes one 16-bit pass per 16 bits of
the shifted key range, up to four, reading the shifted keys as uint64 so
a range above 2**63 stays exact. Inputs under 128 keys per pass take one
stable comparison sort, which beats the passes' fixed overhead. This
mirrors zarr-developers#4643's `argsort_chunk_indices`; the order within a group stays
unspecified.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(zarr-indexing): record why repeated write targets get no promised value

Add a "Repeated write targets" design note: the library promises no
value for a cell a write addresses more than once, chooses its internal
order by performance (today a stable radix sort that happens to keep the
last write), carries no order within a planning-table row, and does not
deduplicate, because the pass would cost more than the assignment.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…developers#4650)

Bumps the python-dependencies group with 5 updates:

| Package | From | To |
| --- | --- | --- |
| [typer](https://github.com/fastapi/typer) | `0.27.3` | `0.29.1` |
| [hypothesis](https://github.com/HypothesisWorks/hypothesis) | `6.170.14` | `6.170.15` |
| [mkdocs-material[imaging]](https://github.com/squidfunk/mkdocs-material) | `9.8.0` | `9.8.1` |
| [markdown-exec[ansi]](https://github.com/pawamoy/markdown-exec) | `1.12.6` | `1.13.0` |
| [ruff](https://github.com/astral-sh/ruff) | `0.17.3` | `0.17.4` |


Updates `typer` from 0.27.3 to 0.29.1
- [Release notes](https://github.com/fastapi/typer/releases)
- [Changelog](https://github.com/fastapi/typer/blob/master/docs/release-notes.md)
- [Commits](fastapi/typer@0.27.3...0.29.1)

Updates `hypothesis` from 6.170.14 to 6.170.15
- [Release notes](https://github.com/HypothesisWorks/hypothesis/releases)
- [Commits](HypothesisWorks/hypothesis@v6.170.14...v6.170.15)

Updates `mkdocs-material[imaging]` from 9.8.0 to 9.8.1
- [Release notes](https://github.com/squidfunk/mkdocs-material/releases)
- [Changelog](https://github.com/squidfunk/mkdocs-material/blob/master/CHANGELOG)
- [Commits](squidfunk/mkdocs-material@9.8.0...9.8.1)

Updates `markdown-exec[ansi]` from 1.12.6 to 1.13.0
- [Release notes](https://github.com/pawamoy/markdown-exec/releases)
- [Changelog](https://github.com/pawamoy/markdown-exec/blob/main/CHANGELOG.md)
- [Commits](pawamoy/markdown-exec@1.12.6...1.13.0)

Updates `ruff` from 0.17.3 to 0.17.4
- [Release notes](https://github.com/astral-sh/ruff/releases)
- [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md)
- [Commits](astral-sh/ruff@0.17.3...0.17.4)

---
updated-dependencies:
- dependency-name: typer
  dependency-version: 0.29.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: python-dependencies
- dependency-name: hypothesis
  dependency-version: 6.170.15
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: mkdocs-material[imaging]
  dependency-version: 9.8.1
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: markdown-exec[ansi]
  dependency-version: 1.13.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: python-dependencies
- dependency-name: ruff
  dependency-version: 0.17.4
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
numcodecs deprecates the Pickle codec and warns whenever one is
constructed (zarr-developers/numcodecs#905). The tests for
array.allow_unsafe_pickle_decode construct the codec on purpose, and
pytest turns warnings into errors, so they would fail once numcodecs
ships the deprecation. Apply a filterwarnings mark scoped to the
deprecation message to those tests only. The filter matches nothing on
numcodecs releases that do not warn.

Assisted-by: ClaudeCode:claude-opus-5.5

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants