diff --git a/AGENTS.md b/AGENTS.md index 1d07f4f..077abba 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -10,7 +10,9 @@ point of the tool is that memory is set by `--chunk`, not by the size of the input, so code that loads a whole matrix or column is a regression unless it is behind an explicit `--in-memory`. pandas and scipy are not dependencies either: they come only with the `pickle` extra, for pycisTopic pickles -(`conversion/cistopic.py`), and are imported there alone. +(`conversion/cistopic.py`), and are imported there alone. pyarrow likewise +comes only with the `spatial` extra, for Visium HD's Parquet positions, and +is imported only in `formats/parquet.py`. Naming: - The PyPI distribution is `pyadata-cli`. diff --git a/Dockerfile b/Dockerfile index 1e70cf6..b774ebd 100644 --- a/Dockerfile +++ b/Dockerfile @@ -61,9 +61,10 @@ COPY . . # them; that reclaims about 9 MB, the cache being mostly hardlinks into the # venv rather than separate copies. # -# The `pickle` extra (pandas, scipy) is in: a pipeline cannot add it to the -# image later, and without it pycisTopic pickles do not convert. -RUN uv sync --locked --extra pickle \ +# The `pickle` (pandas, scipy) and `spatial` (pyarrow) extras are in: a +# pipeline cannot add them to the image later, and without them pycisTopic +# pickles and Visium HD do not convert. +RUN uv sync --locked --extra pickle --extra spatial \ && rm -rf /tmp/.cache /tmp/uv-*.lock \ && mkdir -p /tmp/.cache \ && chmod 1777 /tmp/.cache diff --git a/README.md b/README.md index e6273a1..894baac 100644 --- a/README.md +++ b/README.md @@ -21,8 +21,9 @@ adata convert filtered_feature_bc_matrix.h5 -o counts.h5ad ``` The command is `adata`; the distribution is `pyadata-cli`, because `adata-cli` was -taken on PyPI. `pip install 'pyadata-cli[pickle]'` adds pycisTopic pickles. The -container image is `quay.io/cellgeni/adata-cli`: +taken on PyPI. `pip install 'pyadata-cli[pickle]'` adds pycisTopic pickles, and +`'pyadata-cli[spatial]'` Visium HD. The container image, with both, is +`quay.io/cellgeni/adata-cli`: ```bash docker run --rm -v "$PWD":/data quay.io/cellgeni/adata-cli:latest adata view /data/data.h5ad diff --git a/STATUS.md b/STATUS.md index ed5e471..4544224 100644 --- a/STATUS.md +++ b/STATUS.md @@ -20,13 +20,29 @@ The conversion epic's sub-issues (#21–#30) are all done and ticked in #20. ## In progress -Nothing. Spatial (standard Visium and Xenium) is out in 0.17.0; Visium HD -is first among the follow-ups below. +### Visium HD (`feat/visium-hd`, for 0.17.1) + +Decisions: +- One bin size per output, `--bin-size` (8 µm by default, as 10x + recommends); a bin folder given directly reads as it is. +- pyarrow comes with a new `spatial` extra, in the image too, and is + imported only by `formats/parquet.py`. +- The Parquet positions are read as a buffered stream, a set of columns per + pass; the join holds barcodes as bytes; images are written in strips as + they are read. 2 µm on 10x's tiny sample: 3.5 s, 470 MB without images. + +Steps: +- [x] `formats/parquet.py`, HD in `visium.py`, `--bin-size` +- [x] Tests: HD fixture, flags and refusals, the missing extra, a performance + guard, a scanpy and pandas comparison +- [x] Real data: 10x's SR 4.0.1 tiny HD sample at 2, 8 and 16 µm +- [x] Docs +- [ ] PR, review, merge; release 0.17.1 ## Follow-ups noticed -- Visium HD bins and Space Ranger 4's cell segmentation, behind a `spatial` - extra (pyarrow) for their Parquet positions. +- Space Ranger 4's cell segmentation (`segmented_outputs/`): cells as + GeoJSON polygons, mapped to bins by `barcode_mappings.parquet`. - Spatial bundles whose matrix is a MEX folder only (refused for now, with a pointer to `--from 10x-mtx`). - Space Ranger's own `analysis/` (clusters, UMAP) for Visium, as Xenium's. diff --git a/docs/developers.md b/docs/developers.md index 9587174..317fe38 100644 --- a/docs/developers.md +++ b/docs/developers.md @@ -19,9 +19,10 @@ uv run adata --help uv run pytest -m "not integration" # the unit suite CI gates on, a few minutes ``` -The tool itself depends on h5py, zarr, numpy, Pillow, Typer and Rich. -anndata, pandas and scipy come with the `dev` extra, as the reference the -tests check against; scanpy, loompy, mudata and older anndata releases are +The tool itself depends on h5py, zarr, numpy, Pillow, Typer and Rich; the +`pickle` extra adds pandas and scipy, and the `spatial` extra pyarrow. +anndata, pandas, scipy and pyarrow come with the `dev` extra, as the +reference the tests check against; scanpy, loompy, mudata and older anndata releases are installed by the `integration` tests in environments of their own. ## Repository layout diff --git a/docs/guides/convert.md b/docs/guides/convert.md index 41c1153..a85a863 100644 --- a/docs/guides/convert.md +++ b/docs/guides/convert.md @@ -41,6 +41,7 @@ no extension. |---|---|---| | Cell Ranger's `*_feature_bc_matrix.h5` | `adata convert in.h5 -o out.h5ad` | [10x HDF5](../reference/formats/10x-h5.md) | | A Space Ranger `outs/` folder (standard Visium) | `adata convert outs -o out.h5ad` | [Visium](../reference/formats/visium.md) | +| A Visium HD `outs/` folder | `adata convert outs -o out.h5ad --bin-size 8` (needs `pyadata-cli[spatial]`) | [Visium HD](../reference/formats/visium.md#visium-hd) | | A Xenium output bundle | `adata convert bundle -o out.h5ad` | [Xenium](../reference/formats/xenium.md) | | CellBender `output.h5` | `adata convert output.h5 -o out.h5ad` | [CellBender](../reference/formats/cellbender.md) | | A `matrix.mtx` folder from Cell Ranger, STARsolo or GEO | `adata convert folder -o out.h5ad` | [MTX folders](../reference/formats/10x-mtx.md) | diff --git a/docs/guides/pipelines.md b/docs/guides/pipelines.md index 4d0ecdb..4edee48 100644 --- a/docs/guides/pipelines.md +++ b/docs/guides/pipelines.md @@ -13,7 +13,7 @@ nav_order: 6 ## The container image Every release is published as `quay.io/cellgeni/adata-cli:`, and the -latest as `:latest`. The image has the `pickle` extra and +latest as `:latest`. The image has the `pickle` and `spatial` extras and [duckdb](https://duckdb.org) in it, and sets no entrypoint, as Nextflow needs, so name the command: diff --git a/docs/index.md b/docs/index.md index f1932a0..c2eed4c 100644 --- a/docs/index.md +++ b/docs/index.md @@ -30,13 +30,14 @@ container or a pipeline step where the scientific stack would be overkill. ```bash pip install pyadata-cli pip install 'pyadata-cli[pickle]' # also read pycisTopic pickles (adds pandas, scipy) +pip install 'pyadata-cli[spatial]' # also read Visium HD (adds pyarrow) ``` The command is `adata`. The distribution is named `pyadata-cli` because `adata-cli` was already taken on PyPI by an unrelated project. Python 3.12 or later. -Or use the container, which has the `pickle` extra and duckdb in it: +Or use the container, which has both extras and duckdb in it: ```bash docker run --rm -v "$PWD":/data quay.io/cellgeni/adata-cli:latest \ diff --git a/docs/reference/commands.md b/docs/reference/commands.md index 830a2d3..a27a968 100644 --- a/docs/reference/commands.md +++ b/docs/reference/commands.md @@ -120,6 +120,7 @@ adata convert counts.h5ad -o for_cellbender.h5 --layer counts adata convert Solo.out/Gene/filtered -o counts.h5ad adata convert counts.h5ad -o counts_mtx --to 10x-mtx adata convert spaceranger_run/outs -o section.h5ad +adata convert visium_hd_run/outs -o bins.h5ad --bin-size 8 adata convert xenium_bundle -o cells.h5ad adata convert splitpipe_out -o parse.h5ad adata convert velocyto.loom -o velocyto.h5ad @@ -146,7 +147,7 @@ adata convert multiome.h5ad -o multiome.h5mu | `h5ad` | `.h5ad` | yes | yes | | `zarr` | `.zarr` (v2 or v3) | yes | yes | | [`10x-h5`](formats/10x-h5.md) | `.h5` | Cell Ranger 2 and 3+; the matrix alone of Space Ranger and Xenium | Cell Ranger 3+ layout | -| [`visium`](formats/visium.md) | a folder | Space Ranger `outs/`: counts, spot positions, images (standard Visium; not HD yet) | no | +| [`visium`](formats/visium.md) | a folder | Space Ranger `outs/`: counts, spot or bin positions, images (standard Visium; Visium HD with `pyadata-cli[spatial]`) | no | | [`xenium`](formats/xenium.md) | a folder | a Xenium bundle: counts, cells and centroids, Xenium's analysis | no | | [`cellbender`](formats/cellbender.md) | `.h5` | CellBender `remove-background` 0.2 and later | no | | [`10x-mtx`](formats/10x-mtx.md) | a folder (`--to 10x-mtx`) | Cell Ranger 2 and 3+, STARsolo (velocyto too) | Cell Ranger 3+ layout, gzipped | diff --git a/docs/reference/formats/index.md b/docs/reference/formats/index.md index 8abc228..36813a8 100644 --- a/docs/reference/formats/index.md +++ b/docs/reference/formats/index.md @@ -18,7 +18,7 @@ flags every conversion takes (`--output`, `--from`, `--to`, `--compression`, | [10x Genomics HDF5](10x-h5.md) | `10x-h5` | Cell Ranger 2 and 3+; the matrix alone of Space Ranger and Xenium | Cell Ranger 3+ layout | | [CellBender](cellbender.md) | `cellbender` | `remove-background` 0.2 and later, with its latents | no | | [10x and STARsolo MTX folders](10x-mtx.md) | `10x-mtx` | Cell Ranger 2 and 3+, STARsolo (velocyto too), GEO prefixes | Cell Ranger 3+ layout, gzipped | -| [Visium (Space Ranger)](visium.md) | `visium` | an `outs/` folder: counts, spot positions, tissue images, scale factors (Space Ranger 1–4; not HD yet) | no | +| [Visium (Space Ranger)](visium.md) | `visium` | an `outs/` folder: counts, spot or bin positions, tissue images, scale factors (Space Ranger 1–4; Visium HD needs `pyadata-cli[spatial]`) | no | | [Xenium](xenium.md) | `xenium` | an output bundle: counts, cells and centroids, Xenium's clusters, UMAP and PCA | no | | [Parse Biosciences](parse.md) | `parse` | split-pipe: a run, a sample or a `DGE_*` folder | no | | [loom and SCope](loom.md) | `loom` | loompy, velocyto, kallisto\|bustools, SCope; spec 2 and 3 | spec 3.0.0; a SCope loom with `--scope` | diff --git a/docs/reference/formats/visium.md b/docs/reference/formats/visium.md index 5ab58ea..c2cd42d 100644 --- a/docs/reference/formats/visium.md +++ b/docs/reference/formats/visium.md @@ -7,11 +7,13 @@ nav_order: 4 # Visium (Space Ranger) -A Space Ranger output folder for standard Visium (55 µm spots), from Space -Ranger 1 to 4: the counts, where each spot sits on the slide, the tissue +A Space Ranger output folder, for standard Visium (55 µm spots, Space Ranger +1 to 4) and for [Visium HD](#visium-hd) (2, 8 and 16 µm bins, Space Ranger 3 +and 4): the counts, where each spot or bin sits on the slide, the tissue images and the scale factors between them. The result is laid out as `squidpy.read.visium` lays it out, so scanpy's and squidpy's spatial plots -work on it, and CI checks it against squidpy. +work on it. CI checks standard Visium against squidpy, and Visium HD, which +squidpy does not read, against scanpy with pandas reading the positions. Give the `outs/` folder, or the run folder holding it. It is recognised by `spatial/` (the positions and `scalefactors_json.json`) beside a @@ -67,7 +69,42 @@ longest side. Refused, with the reason: - output of `spaceranger aggr`, whose matrix holds several libraries; -- Visium HD (`binned_outputs/`), whose positions are Parquet and are not - read yet. Each bin's `filtered_feature_bc_matrix.h5` converts as a - [10x HDF5 file](10x-h5.md) meanwhile; - a folder with the matrix as Matrix Market only, with no `.h5`. + +## Visium HD + +Space Ranger 3 and 4 write a Visium HD run's counts at three bin sizes, each +in its own folder under `binned_outputs/` (`square_002um`, `square_008um`, +`square_016um`), laid out as a standard Visium `outs/` but with the +positions as `spatial/tissue_positions.parquet`. Give the `outs/` folder and +`--bin-size` picks the bins, 8 µm by default as 10x recommends for +analysis; or give a bin folder itself. + +```bash +pip install 'pyadata-cli[spatial]' # pyarrow, which reads the Parquet positions + +adata convert outs -o bins_8um.h5ad +adata convert outs -o bins_2um.h5ad --bin-size 2 +adata convert outs/binned_outputs/square_016um -o bins_16um.h5ad +``` + +| Flag | Meaning | +|---|---| +| `--bin-size` | The bins to read, in µm (default 8; refused for standard Visium) | + +The result is laid out as above: `in_tissue`, `array_row` and `array_col` in +obs, `obsm["spatial"]` in full-resolution pixels (float64, as Space Ranger +records them), and the bin folder's own images and scale factors, which +carry `bin_size_um` and `microns_per_pixel`, in `uns["spatial"]`. + +The positions list every bin on the slide, about 11 million at 2 µm, where +the filtered matrix holds the bins under tissue. They are read twice, a +buffered stream of only the columns needed: the barcodes, matched to the +matrix's as they come, then the numbers, written in the matrix's order. On +10x's Visium HD tiny sample, the 2 µm bins (330,035 in the matrix) convert in +3.5 seconds with a peak of 470 MB with `--no-images`, and 610 MB with the +100 MB hires image; pandas alone takes about 1 GB to read the positions. `--no-images` leaves the +images out. + +The cell segmentation Space Ranger 4 adds (`segmented_outputs/`) is not read +yet; its matrix converts as a [10x HDF5 file](10x-h5.md). diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index f97e011..11ba8c7 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -152,12 +152,11 @@ A coordinate, centroid or projection column holds text. Missing values are fine (they become NaN); a word is not. The file has been edited, or is not what its name says. -### `This is Visium HD output, which is not read yet: its positions are Parquet.` +### `This Visium HD output has no 2 µm bins; it has 8 µm, 16 µm. Pick one with --bin-size.` -Visium HD's bin positions exist only as `tissue_positions.parquet`, which a -later release will read. Meanwhile, convert one bin size's matrix as a plain -10x matrix, without positions: -`adata convert binned_outputs/square_008um/filtered_feature_bc_matrix.h5 -o bins.h5ad`. +`--bin-size` names a folder under `binned_outputs/` (`square_002um` for 2). +Space Ranger writes 2, 8 and 16 µm bins, but a copied or trimmed output may +hold fewer; the message lists those there are. ### `This matrix holds 2 libraries (A1, B1), as spaceranger aggr writes` @@ -252,6 +251,11 @@ wins alongside those where it loses. pandas and scipy are not dependencies of `adata`; the `pickle` extra adds them, and only pycisTopic pickles need them. The container image has them. +### `Reading Visium HD positions needs pyarrow, which reads its Parquet tables: pip install 'pyadata-cli[spatial]'.` + +Visium HD records where each bin sits only as Parquet. The `spatial` extra +adds pyarrow, which only Visium HD needs; the container image has it. + ## Reading the output elsewhere ### anndata warns `OldFormatWarning` diff --git a/pyproject.toml b/pyproject.toml index 1cb4926..233ab1e 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -59,9 +59,15 @@ pickle = [ "pandas>=2.2", "scipy>=1.13", ] +# Reading Visium HD: its bin positions are Parquet only. Imported only by +# the Visium converter, and only for an HD input. +spatial = [ + "pyarrow>=17", +] dev = [ "pandas>=2.2", "scipy>=1.13", + "pyarrow>=17", "pytest>=8.3.4", "pytest-cov>=6.0.0", # Turns a hang into a failure with a traceback. Without it a stuck test diff --git a/src/adata/cli.py b/src/adata/cli.py index 71e3dbb..6b35503 100644 --- a/src/adata/cli.py +++ b/src/adata/cli.py @@ -530,6 +530,7 @@ def concat( "unfiltered": "--unfiltered", "library_id": "--library-id", "no_images": "--no-images", + "bin_size": "--bin-size", "dense": "--dense", "obs_names_attr": "--obs-names-attr", "var_names_attr": "--var-names-attr", @@ -712,6 +713,12 @@ def convert( help="Visium input: leave the tissue images out of uns['spatial'] " "(the scale factors stay)", ), + bin_size: Optional[int] = typer.Option( + None, + "--bin-size", + help="Visium HD input: the bin size to read, in µm (default 8; " + "Space Ranger writes 2, 8 and 16)", + ), dense: bool = typer.Option( False, "--dense", @@ -809,8 +816,9 @@ def convert( CellBender's own loader names them. A 10x matrix is read as scanpy's read_10x_h5 or read_10x_mtx (gex_only=False) reads it, keeping integer counts, and written as Cell Ranger writes it. A Space Ranger outs/ - folder (standard Visium) keeps its spot positions, images and scale - factors where squidpy puts them, and a Xenium bundle its cell table, + folder keeps its spot positions, images and scale factors where squidpy + puts them; a Visium HD one is read at the bin size --bin-size names + (needs pyadata-cli[spatial]); and a Xenium bundle its cell table, centroids and Xenium's own clusters, UMAP and PCA. MTX folders from Cell Ranger and STARsolo, velocyto's included, and Parse split-pipe output are read in two streaming passes, as are text tables (.csv, .tsv, .txt, @@ -840,6 +848,7 @@ def convert( adata convert Solo.out/Gene/filtered -o counts.h5ad adata convert Solo.out/Velocyto/raw -o velocyto.h5ad adata convert spaceranger_run/outs -o section.h5ad + adata convert visium_hd_run/outs -o bins.h5ad --bin-size 8 adata convert xenium_bundle -o cells.h5ad adata convert counts.h5ad -o counts_mtx --to 10x-mtx --layer counts adata convert cellbender_output.h5 -o cb.h5ad --raw raw_feature_bc_matrix.h5 @@ -946,6 +955,7 @@ def convert( unfiltered=unfiltered, library_id=library_id, no_images=no_images, + bin_size=bin_size, dense=dense, obs_names_attr=obs_names_attr, var_names_attr=var_names_attr, diff --git a/src/adata/conversion/base.py b/src/adata/conversion/base.py index 9931fdd..9f8f2d7 100644 --- a/src/adata/conversion/base.py +++ b/src/adata/conversion/base.py @@ -47,6 +47,7 @@ class Options: # Reading Space Ranger output. library_id: Optional[str] = None no_images: bool = False + bin_size: Optional[int] = None # Reading a loom. dense: bool = False diff --git a/src/adata/conversion/spatial.py b/src/adata/conversion/spatial.py index 93ceff5..360afde 100644 --- a/src/adata/conversion/spatial.py +++ b/src/adata/conversion/spatial.py @@ -11,14 +11,16 @@ from dataclasses import dataclass from pathlib import Path -from typing import Any, Callable, Dict, List, Optional, Sequence +from typing import Any, Callable, Dict, Iterable, Iterator, List, Optional, Sequence, Tuple import numpy as np from adata.conversion.features import STRING_STEP from adata.core.scatter import RowScatter +from adata.elements import spec from adata.elements.read import decode_str_array from adata.formats.table import TableScan, column_values, iter_table_blocks +from adata.storage import create_dataset @dataclass @@ -64,7 +66,9 @@ class BarcodeJoin: own. While each block equals the matrix's next barcodes it only compares them; at the first difference it switches to looking each value up in a sorted copy of the barcodes, a numpy array rather than a dict, whose - per-string overhead is several times the barcode itself. + per-string overhead is several times the barcode itself. Barcodes are + held as bytes, a quarter of numpy's unicode: Visium HD's 2 µm bins number + in the millions. """ def __init__(self, barcodes: Any) -> None: @@ -77,10 +81,10 @@ def __init__(self, barcodes: Any) -> None: self._order: Optional[np.ndarray] = None def _slice(self, lo: int, hi: int) -> np.ndarray: - return np.asarray(decode_str_array(np.asarray(self._barcodes[lo:hi])), dtype=str) + return _as_bytes(np.asarray(self._barcodes[lo:hi])) - def __call__(self, values: Sequence[str]) -> None: - values = np.asarray(values, dtype=str) + def __call__(self, values: Sequence[Any]) -> None: + values = _as_bytes(values) lo, hi = self._done, self._done + values.size if self._in_order: if hi <= self.n_out and np.array_equal(self._slice(lo, hi), values): @@ -96,7 +100,7 @@ def _leave_order(self) -> None: self._targets = [np.arange(self._done, dtype=np.int64)] wanted = np.concatenate( [self._slice(lo, min(lo + STRING_STEP, self.n_out)) for lo in range(0, self.n_out, STRING_STEP)] - or [np.zeros(0, dtype=str)] + or [np.zeros(0, dtype="S1")] ) self._order = np.argsort(wanted, kind="stable") self._ranked = wanted[self._order] @@ -116,6 +120,7 @@ def finish(self) -> Join: if self._in_order: self._leave_order() target = np.concatenate(self._targets) if self._targets else np.zeros(0, dtype=np.int64) + self._targets = [] # the blocks, as big again as the target counts = np.bincount(target[target >= 0], minlength=self.n_out) missing = np.flatnonzero(counts == 0) return Join( @@ -124,33 +129,53 @@ def finish(self) -> Join: n_extra=int((target < 0).sum()), n_missing=int(missing.size), n_repeated=int((counts > 1).sum()), - example_missing=str(self._slice(missing[0], missing[0] + 1)[0]) if missing.size else "", + example_missing=( + self._slice(missing[0], missing[0] + 1)[0].decode("utf-8", "replace") + if missing.size else "" + ), ) +def _as_bytes(values: Any) -> np.ndarray: + """Strings, bytes or numbers as a fixed-width bytes array, for comparing.""" + array = np.asarray(values) + if array.dtype.kind == "S": + return array + if array.dtype.kind == "O" and array.size and isinstance(array.flat[0], bytes): + return array.astype("S") + return np.char.encode(np.asarray(decode_str_array(array), dtype=str), "utf-8") + + @dataclass class Sink: """One array to fill from a table: `values` turns a block's columns into the block's values; `fill` is what rows the table lacks get.""" dest: Any - values: Callable[[Dict[str, List[str]]], np.ndarray] + values: Callable[[Dict[str, Any]], np.ndarray] fill: Any = None -def write_rows(scan: TableScan, join: Join, sinks: Sequence[Sink], scratch_parent: Any) -> None: - """Fill each sink from one read of the table, in the matrix's order.""" +def table_blocks(scan: TableScan) -> Iterator[Tuple[int, Dict[str, List[str]]]]: + """A CSV table's rows for `write_rows`: (rows in the block, its columns).""" + for names, block in iter_table_blocks(scan): + yield len(names), block + + +def write_rows( + blocks: Iterable[Tuple[int, Dict[str, Any]]], join: Join, sinks: Sequence[Sink], scratch_parent: Any +) -> None: + """Fill each sink from one read of a table's `blocks`, in the matrix's order.""" if join.target is None: lo = 0 - for names, block in iter_table_blocks(scan): - hi = lo + len(names) + for n, block in blocks: for sink in sinks: - sink.dest[lo:hi] = sink.values(block) - lo = hi + sink.dest[lo : lo + n] = sink.values(block) + lo += n return with RowScatter(join.target, join.n_out, scratch_parent) as scatter: columns = [scatter.column(sink.dest, sink.fill) for sink in sinks] - for _, block in iter_table_blocks(scan): + for _, block in blocks: for column, sink in zip(columns, sinks): column.add(sink.values(block)) @@ -178,10 +203,29 @@ def numbers(scan: TableScan, names: Sequence[str], block: Dict[str, List[str]], ) -def read_image(path: Path) -> np.ndarray: - """An image's pixels as the file holds them: uint8 for an 8-bit PNG, as - squidpy reads them, H x W or H x W x C.""" +#: Image rows copied out of Pillow per write. +IMAGE_ROWS = 512 + + +def write_image(parent: Any, name: str, path: Path) -> None: + """An image's pixels as the file holds them, uint8 for an 8-bit PNG as + squidpy reads them, H x W or H x W x C, written a strip of rows at a time. + + A PNG decodes only whole, but copying it out in strips spares a second + copy of it: Visium HD's hires image is some 100 MB of pixels. + """ from PIL import Image with Image.open(path) as image: - return np.asarray(image) + image.load() + width, height = image.size + out = None + for top in range(0, height, IMAGE_ROWS): + strip = np.asarray(image.crop((0, top, width, min(top + IMAGE_ROWS, height)))) + if out is None: + out = create_dataset( + parent, name, shape=(height,) + strip.shape[1:], dtype=strip.dtype, + chunks=(min(IMAGE_ROWS, height),) + strip.shape[1:], + ) + spec.set_encoding(out, spec.ARRAY) + out[top : top + strip.shape[0]] = strip diff --git a/src/adata/conversion/visium.py b/src/adata/conversion/visium.py index 5fcbebb..2a69c67 100644 --- a/src/adata/conversion/visium.py +++ b/src/adata/conversion/visium.py @@ -20,7 +20,9 @@ a few thousand however large the data, and are matched to the matrix's barcodes as they are scanned. -Visium HD is not read yet: its positions are Parquet. +Visium HD is read too, one bin size at a time (`--bin-size`, 8 µm by +default): each `binned_outputs/square_NNNum/` folder is laid out as above, +with its positions as Parquet, which pyarrow reads (the `spatial` extra). """ from __future__ import annotations @@ -29,7 +31,7 @@ from contextlib import ExitStack from dataclasses import dataclass from pathlib import Path -from typing import Any, Dict, List, Optional, Sequence +from typing import Any, Dict, List, Optional, Sequence, Union import h5py import numpy as np @@ -38,7 +40,7 @@ from adata.conversion import tenx from adata.conversion.base import ConversionPlan, Converter, Options from adata.conversion.features import STRING_STEP, Target, describe_import, write_obs_index, write_var -from adata.conversion.spatial import BarcodeJoin, Join, Sink, numbers, read_image, require_numeric, write_rows +from adata.conversion.spatial import BarcodeJoin, Join, Sink, table_blocks, numbers, write_image, require_numeric, write_rows from adata.core.subset import fan_out_sparse_matrix from adata.elements.read import decode_str_array from adata.elements.write import ( @@ -49,6 +51,7 @@ write_scalar, ) from adata.formats.json_data import write_json_value +from adata.formats.parquet import ParquetTable, open_table, require_pyarrow from adata.formats.table import TableScan, column_values, create_columns, scan_table from adata.formats.textio import open_binary from adata.storage import create_dataset, is_zarr_path, open_store @@ -60,15 +63,27 @@ "barcode", "in_tissue", "array_row", "array_col", "pxl_row_in_fullres", "pxl_col_in_fullres", ) -_POSITION_FILES = ("tissue_positions.csv", "tissue_positions_list.csv") +_POSITION_FILES = ("tissue_positions.csv", "tissue_positions_list.csv", "tissue_positions.parquet") +#: The bin size an HD output is read at unless --bin-size says otherwise: +#: the one 10x recommends for analysis. +DEFAULT_BIN_UM = 8 _IMAGES = ("hires", "lowres") _FILTERED, _RAW = "filtered_feature_bc_matrix", "raw_feature_bc_matrix" def _is_hd(folder: Path) -> bool: - return (folder / "binned_outputs").is_dir() or ( - folder / "spatial" / "tissue_positions.parquet" - ).exists() + """A Visium HD `outs/`, whose bins sit one folder per size below it.""" + return (folder / "binned_outputs").is_dir() + + +def _bin_sizes(outs: Path) -> Dict[int, Path]: + """Each bin size an HD output holds, in µm, and its folder.""" + sizes = {} + for folder in (outs / "binned_outputs").glob("square_*um"): + digits = folder.name[len("square_"):-len("um")] + if digits.isdigit() and folder.is_dir(): + sizes[int(digits)] = folder + return dict(sorted(sizes.items())) def _matrix_files(folder: Path, name: str) -> List[Path]: @@ -99,11 +114,8 @@ def _locate(path: Path) -> Optional[Path]: def sniff(path: Path) -> bool: - """A Space Ranger output folder (or the run folder holding its `outs/`). - - Visium HD folders are claimed too, only so that they are refused by name - rather than taken for a Cell Ranger 2 MTX layout. - """ + """A Space Ranger output folder (or the run folder holding its `outs/`), + a Visium HD one included, or one of its bin folders.""" path = Path(path) return path.is_dir() and not is_zarr_path(path) and _locate(path) is not None @@ -113,8 +125,10 @@ class _Plan: matrix: Any matrix_path: Path targets: List[Target] - positions: TableScan + #: The CSV positions' scan, or the Parquet ones' footer. + positions: Union[TableScan, ParquetTable] join: Join + bin_um: Optional[int] library_id: str scalefactors: Dict[str, Any] images: Dict[str, Path] @@ -125,11 +139,12 @@ class _Plan: def _bundle(source: Path, options: Options) -> tuple: """(the outs folder, the matrix file) for this input.""" if source.is_file(): - if "--unfiltered" in options.given: - raise ValueError( - "--unfiltered picks between the matrices of a Space Ranger outs " - f"folder; {source.name} is a matrix already. Give the folder." - ) + for flag in ("--unfiltered", "--bin-size"): + if flag in options.given: + raise ValueError( + f"{flag} picks among the matrices of a Space Ranger output " + f"folder; {source.name} is a matrix already. Give the folder." + ) return source.parent, source outs = _locate(source) if outs is None: @@ -139,10 +154,18 @@ def _bundle(source: Path, options: Options) -> tuple: "*_feature_bc_matrix.h5." ) if _is_hd(outs): + sizes = _bin_sizes(outs) + size = DEFAULT_BIN_UM if options.bin_size is None else options.bin_size + if size not in sizes: + raise ValueError( + f"This Visium HD output has no {size} µm bins; it has " + f"{', '.join(f'{s} µm' for s in sizes) or 'none'}. Pick one with --bin-size." + ) + outs = sizes[size] + elif options.bin_size is not None: raise ValueError( - "This is Visium HD output, which is not read yet: its positions are " - "Parquet. Convert one bin's matrix as a 10x matrix in the meantime " - "(binned_outputs/square_008um/filtered_feature_bc_matrix.h5)." + "--bin-size picks among a Visium HD output's bins; this is " + "standard Visium, or one bin size already." ) name = _RAW if options.unfiltered else _FILTERED found = _matrix_files(outs, name) @@ -191,8 +214,43 @@ def _library_id(attrs: Dict[str, Any], outs: Path, options: Options) -> str: return outs.parent.name if outs.name == "outs" else outs.name +def _bin_um(scalefactors: Dict[str, Any]) -> Optional[int]: + """An HD bin folder's bin size, from its scale factors.""" + size = scalefactors.get("bin_size_um") + return int(round(float(size))) if size is not None else None + + +def _scan_parquet(path: Path, joiner: BarcodeJoin) -> ParquetTable: + """Visium HD's positions: the footer, then the barcodes, matched as read.""" + require_pyarrow("Visium HD positions") + table = open_table(path) + missing = [c for c in POSITION_COLUMNS if c not in table.types] + if missing: + raise ValueError(f"{path.name} has no {', '.join(missing)} column.") + text = [c for c in POSITION_COLUMNS[1:] if not _numeric_arrow(table.types[c])] + if text: + raise ValueError(f"{path.name}: {', '.join(text)} {'is' if len(text) == 1 else 'are'} not numeric.") + # Space Ranger marks these not null; an edited file may not be, and a + # null cast to int64 would be a made-up number. Missing pixels are NaN. + gaps = {c: n for c, n in table.null_counts(POSITION_COLUMNS[:4]).items() if n} + if gaps: + raise ValueError( + f"{path.name} has missing values in " + + ", ".join(f"{c} ({n:,})" for c, n in gaps.items()) + "." + ) + for _, block in table.blocks(["barcode"]): + joiner(block["barcode"]) + return table + + +def _numeric_arrow(type_name: str) -> bool: + return type_name.startswith(("int", "uint", "float", "double", "halffloat")) + + class VisiumImport(Converter): - options = tenx.TenxImport.options | frozenset({"--unfiltered", "--library-id", "--no-images"}) + options = tenx.TenxImport.options | frozenset( + {"--unfiltered", "--library-id", "--no-images", "--bin-size"} + ) def plan( self, @@ -207,11 +265,14 @@ def plan( spatial = outs / "spatial" position_file = next((spatial / n for n in _POSITION_FILES if (spatial / n).is_file()), None) scalefactor_file = spatial / "scalefactors_json.json" - if position_file is None or not scalefactor_file.is_file(): - raise ValueError( - f"{outs} has no spatial/tissue_positions.csv and " - "spatial/scalefactors_json.json beside the matrix." - ) + lacking = [ + what for what, present in ( + ("tissue positions (spatial/tissue_positions.csv or .parquet)", position_file is not None), + ("spatial/scalefactors_json.json", scalefactor_file.is_file()), + ) if not present + ] + if lacking: + raise ValueError(f"{outs} has no {' and no '.join(lacking)} beside the matrix.") with h5py.File(matrix_path, "r") as handle: if not tenx.sniff(matrix_path): @@ -220,18 +281,22 @@ def plan( library_id = _library_id(matrix.attrs, outs, options) group = handle[matrix.group] joiner = BarcodeJoin(group["barcodes"]) - positions = scan_table( - position_file, - index="barcode", - names=None if _has_header(position_file) else POSITION_COLUMNS, - on_index=joiner, - ) + if position_file.suffix == ".parquet": + positions = _scan_parquet(position_file, joiner) + else: + positions = scan_table( + position_file, + index="barcode", + names=None if _has_header(position_file) else POSITION_COLUMNS, + on_index=joiner, + ) + require_numeric(positions, POSITION_COLUMNS[1:]) filtered_barcodes = "filtered_barcodes" in group join = joiner.finish() - require_numeric(positions, POSITION_COLUMNS[1:]) join.check(f"spatial/{position_file.name}") scalefactors = json.loads(scalefactor_file.read_text()) + bin_um = _bin_um(scalefactors) if position_file.suffix == ".parquet" else None images: Dict[str, Path] = {} if not options.no_images: for name in _IMAGES: @@ -248,7 +313,8 @@ def plan( dst_format=dst_format, outputs=outputs, summary=( - f"Space Ranger (Visium) → {dst_format}, {described}, positions from " + f"Space Ranger (Visium{f' HD, {bin_um} µm bins' if bin_um else ''}) → " + f"{dst_format}, {described}, positions from " f"{position_file.name}, library {library_id!r}" + (f", images {', '.join(images)}" if images else "") ), @@ -259,6 +325,7 @@ def plan( targets=targets, positions=positions, join=join, + bin_um=bin_um, library_id=library_id, scalefactors=scalefactors, images=images, @@ -270,7 +337,6 @@ def plan( def write(self, plan: ConversionPlan, staged: Sequence[Path], console: Console) -> None: detail: _Plan = plan.detail matrix = detail.matrix - images = {name: read_image(path) for name, path in detail.images.items()} with h5py.File(detail.matrix_path, "r") as handle, ExitStack() as stack: group = handle[matrix.group] @@ -283,7 +349,7 @@ def write(self, plan: ConversionPlan, staged: Sequence[Path], console: Console) _write_obs(root, group, detail) write_var(root, target) tenx._write_uns(root, matrix) - _write_spatial_uns(root, detail, images) + _write_spatial_uns(root, detail) with console.status("[cyan]Copying X...[/]"): fan_out_sparse_matrix( @@ -306,8 +372,42 @@ def _write_obs(root: Any, group: h5py.Group, detail: _Plan) -> None: ) ensure_anndata_skeleton(root) - scan = detail.positions pixels = ("pxl_col_in_fullres", "pxl_row_in_fullres") # x, y + if isinstance(detail.positions, ParquetTable): + order = _write_parquet_positions(root, obs, detail, pixels) + else: + order = _write_csv_positions(root, obs, detail, pixels) + if detail.filtered_barcodes: + write_dense(obs, "filtered_barcodes", np.asarray(group["filtered_barcodes"][...]).astype(bool)) + order.append("filtered_barcodes") + set_column_order(obs, order) + + +def _write_parquet_positions(root: Any, obs: Any, detail: _Plan, pixels: Sequence[str]) -> List[str]: + """Visium HD's positions: ints in obs, float pixels in `obsm["spatial"]`. + + A second read of the file, of these columns only: the barcodes were + matched in the first. + """ + n = detail.join.n_out + names = list(POSITION_COLUMNS[1:4]) + columns = {} + for name in names: + columns[name] = create_dataset(obs, name, shape=(n,), dtype=np.int64) + spec.set_encoding(columns[name], spec.ARRAY) + spatial = create_dataset(root["obsm"], "spatial", shape=(n, 2), dtype=np.float64) + spec.set_encoding(spatial, spec.ARRAY) + sinks = [Sink(dest, lambda block, name=name: block[name].astype(np.int64)) for name, dest in columns.items()] + sinks.append(Sink( + spatial, lambda block: np.stack([block[p].astype(np.float64) for p in pixels], axis=1) + )) + write_rows(detail.positions.blocks([*names, *pixels]), detail.join, sinks, root) + return names + + +def _write_csv_positions(root: Any, obs: Any, detail: _Plan, pixels: Sequence[str]) -> List[str]: + n = detail.join.n_out + scan = detail.positions kinds = {c.name: scan.dtype_of(c) for c in scan.columns} columns, lookups = create_columns(obs, scan, n, exclude=pixels) dtype = np.int64 if all(kinds[p] == "int64" for p in pixels) else np.float64 @@ -325,22 +425,22 @@ def _write_obs(root: Any, group: h5py.Group, detail: _Plan) -> None: for name, dest in columns.items() ] sinks.append(Sink(spatial, lambda block: numbers(scan, pixels, block, dtype))) - write_rows(scan, detail.join, sinks, root) + write_rows(table_blocks(scan), detail.join, sinks, root) + return list(columns) - order = list(columns) - if detail.filtered_barcodes: - write_dense(obs, "filtered_barcodes", np.asarray(group["filtered_barcodes"][...]).astype(bool)) - order.append("filtered_barcodes") - set_column_order(obs, order) +def _write_spatial_uns(root: Any, detail: _Plan) -> None: + """`uns["spatial"][library_id]`, as squidpy writes it. -def _write_spatial_uns(root: Any, detail: _Plan, images: Dict[str, np.ndarray]) -> None: - """`uns["spatial"][library_id]`, as squidpy writes it.""" + A PNG decodes only whole, so each image is read as it is written and + let go before the next: Visium HD's hires image is 6,000 pixels on its + longest side, some 100 MB of pixels. + """ library = write_mapping(write_mapping(root["uns"], "spatial"), detail.library_id) - if images: + if detail.images: holder = write_mapping(library, "images") - for name, pixels in images.items(): - write_dense(holder, name, pixels) + for name, path in detail.images.items(): + write_image(holder, name, path) write_json_value(library, "scalefactors", detail.scalefactors) metadata = write_mapping(library, "metadata") for key in ("chemistry_description", "software_version"): diff --git a/src/adata/conversion/xenium.py b/src/adata/conversion/xenium.py index 83abbd8..383bc6f 100644 --- a/src/adata/conversion/xenium.py +++ b/src/adata/conversion/xenium.py @@ -41,7 +41,7 @@ from adata.conversion import tenx from adata.conversion.base import ConversionPlan, Converter, Options from adata.conversion.features import STRING_STEP, Target, describe_import, write_obs_index, write_var -from adata.conversion.spatial import BarcodeJoin, Join, Sink, numbers, require_numeric, write_rows +from adata.conversion.spatial import BarcodeJoin, Join, Sink, table_blocks, numbers, require_numeric, write_rows from adata.core.subset import fan_out_sparse_matrix from adata.elements import spec from adata.elements.read import decode_str_array @@ -301,7 +301,7 @@ def _write_cells(root: Any, obs: Any, table: _Table) -> List[str]: for name, dest in columns.items() ] sinks.append(Sink(spatial, lambda block: numbers(scan, _CENTROIDS, block))) - write_rows(scan, join, sinks, root) + write_rows(table_blocks(scan), join, sinks, root) if join.n_extra: # Categories only the left-out cells used go, as anndata's @@ -329,7 +329,7 @@ def _write_clusters(root: Any, obs: Any, table: _Table) -> None: lookup = {c: i for i, c in enumerate(categories)} codes = create_categorical(obs, "graphclust", categories, join.n_out) write_rows( - scan, join, + table_blocks(scan), join, [Sink(codes, lambda block: np.array([lookup.get(v, -1) for v in block["Cluster"]], dtype=np.int64), -1)], root, ) @@ -346,7 +346,7 @@ def _write_projection(root: Any, key: str, table: _Table) -> None: out = create_dataset(root["obsm"], key, shape=(join.n_out, len(names)), dtype=np.float64) spec.set_encoding(out, spec.ARRAY) write_rows( - scan, join, + table_blocks(scan), join, [Sink(out, lambda block: numbers(scan, names, block), np.nan)], root, ) diff --git a/src/adata/formats/parquet.py b/src/adata/formats/parquet.py new file mode 100644 index 0000000..622e258 --- /dev/null +++ b/src/adata/formats/parquet.py @@ -0,0 +1,111 @@ +"""Parquet tables, read a batch of rows at a time. + +Visium HD writes its bin positions only as Parquet. pyarrow reads it, and +comes with the `spatial` extra, so it is imported here alone and only when +a Parquet file is read. + +Parquet stores each column apart, so a pass reads only the columns it asks +for, and the file is read as a buffered stream: by default pyarrow reads a +whole column chunk of a row group at once, and Space Ranger writes its +positions as one row group, some 11 million rows for 2 µm bins. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from pathlib import Path +from typing import Dict, FrozenSet, Iterator, Sequence, Tuple + +import numpy as np + +#: Rows handed out per block. +BATCH_ROWS = 1 << 16 + +#: Bytes pyarrow reads from the file at a time. Measured on Space Ranger's +#: 2 µm positions, this halves the peak of reading their numeric columns. +READ_BUFFER = 1 << 20 + + +def require_pyarrow(what: str) -> None: + try: + import pyarrow.parquet # noqa: F401 + except ImportError: + raise ValueError( + f"Reading {what} needs pyarrow, which reads its Parquet tables: " + "pip install 'pyadata-cli[spatial]'." + ) from None + + +@dataclass +class ParquetTable: + path: Path + n_rows: int + #: Each column's Arrow type, by name, in the file's order. + types: Dict[str, str] + #: Columns the schema says may hold nulls. + nullable: FrozenSet[str] = frozenset() + + def null_counts(self, columns: Sequence[str]) -> Dict[str, int]: + """Nulls in each column: none for one the schema says cannot hold + them, else from the footer's statistics, else counted by a pass.""" + import pyarrow.parquet as pq + + counts = {c: 0 for c in columns} + unknown = [] + metadata = pq.ParquetFile(self.path).metadata + names = [metadata.schema.column(i).name for i in range(metadata.num_columns)] + for name in columns: + if name not in self.nullable: + continue + at = names.index(name) + stats = [metadata.row_group(g).column(at).statistics for g in range(metadata.num_row_groups)] + if all(s is not None and s.has_null_count for s in stats): + counts[name] = sum(s.null_count for s in stats) + else: + unknown.append(name) + if unknown: + for _, block in self.blocks(unknown): + for name in unknown: + counts[name] += int(np.count_nonzero(_is_null(block[name]))) + return counts + + def blocks( + self, columns: Sequence[str], batch_rows: int = BATCH_ROWS + ) -> Iterator[Tuple[int, Dict[str, np.ndarray]]]: + """(rows, {column: values}) for `columns`, in file order. Strings + come as bytes, which is how a barcode join compares them.""" + import pyarrow as pa + import pyarrow.parquet as pq + + handle = pq.ParquetFile(self.path, buffer_size=READ_BUFFER, pre_buffer=False) + for batch in handle.iter_batches(batch_size=batch_rows, columns=list(columns)): + values: Dict[str, np.ndarray] = {} + for name in columns: + column = batch.column(name) + if pa.types.is_string(column.type) or pa.types.is_large_string(column.type): + column = column.cast(pa.binary()) + values[name] = column.to_numpy(zero_copy_only=False) + yield batch.num_rows, values + + +def open_table(path: Path) -> ParquetTable: + """A Parquet file's row count and column types, from its footer alone.""" + import pyarrow.parquet as pq + + handle = pq.ParquetFile(path) + schema = handle.schema_arrow + return ParquetTable( + path=Path(path), + n_rows=handle.metadata.num_rows, + types={field.name: str(field.type) for field in schema}, + nullable=frozenset(field.name for field in schema if field.nullable), + ) + + +def _is_null(values: np.ndarray) -> np.ndarray: + """What `to_numpy` makes of an Arrow null: NaN in floats, None in objects.""" + if values.dtype.kind == "f": + return np.isnan(values) + if values.dtype.kind == "O": + return np.array([v is None for v in values], dtype=bool) + return np.zeros(values.shape, dtype=bool) diff --git a/tests/conftest.py b/tests/conftest.py index 4f7019f..e686233 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -1114,6 +1114,28 @@ def _pycistopic_classes() -> Dict[str, type]: return classes +def _as_pycistopic_writes(build): + """Build a pickle's frames as pycisTopic's pandas 1.5 did: strings as + object columns. pandas 3 with pyarrow installed (the `spatial` extra) + makes them ArrowStringArray, which no real pycisTopic pickle holds and + the importer's allowlist refuses.""" + import functools + + @functools.wraps(build) + def wrapped(*args, **kwargs): + import pandas as pd + + try: + context = pd.option_context("future.infer_string", False) + except (KeyError, pd.errors.OptionError): # pandas without the option + return build(*args, **kwargs) + with context: + return build(*args, **kwargs) + + return wrapped + + +@_as_pycistopic_writes def write_cistopic_pickle( path: Path, *, @@ -1204,6 +1226,7 @@ def write_cistopic_pickle( return CistopicTruth(path, obj) +@_as_pycistopic_writes def write_imputed_pickle(path: Path, *, n_cells: int = 15, n_features: int = 8, seed: int = 0) -> CistopicTruth: """A CistopicImputedFeatures: a dense features x cells `mtx`.""" import pickle @@ -1543,3 +1566,87 @@ def write_xenium_bundle( umap=umap, counts=counts, ) + + +# --------------------------------------------------------------------------- +# Space Ranger output for Visium HD +# +# As Space Ranger 3 and 4 write it (checked against 10x's SR 4.0.1 tiny HD +# sample): one folder per bin size under binned_outputs/, each laid out as a +# standard Visium outs/ but with its positions as Parquet -- every bin on +# the slide, in grid order, where the filtered matrix holds those under +# tissue in its own order. + + +@dataclass +class VisiumHDTruth: + outs: Path + bins: Dict[int, VisiumTruth] # by bin size in µm + + +def write_visium_hd_outs(folder: Path, *, sizes: Sequence[int] = (8, 16), grid: int = 6, seed: int = 0) -> VisiumHDTruth: + import pyarrow as pa + import pyarrow.parquet as pq + from PIL import Image + + rng = np.random.default_rng(seed) + outs = folder / "outs" + (outs / "spatial").mkdir(parents=True) + n_features = 4 + ids = [f"ENSMUSG{i:05d}" for i in range(n_features)] + names = [f"G{i}" for i in range(n_features)] + bins: Dict[int, VisiumTruth] = {} + for size in sizes: + side = grid * 16 // size + where = outs / "binned_outputs" / f"square_{size:03d}um" + (where / "spatial").mkdir(parents=True) + every = [f"s_{size:03d}um_{r:05d}_{c:05d}-1" for r in range(side) for c in range(side)] + rows = np.repeat(np.arange(side), side) + cols = np.tile(np.arange(side), side) + in_tissue = (rng.random(len(every)) < 0.6).astype(np.uint8) + pxl_row = rows * size + rng.random(len(every)) + pxl_col = cols * size + rng.random(len(every)) + table = pa.table({ + "barcode": every, + "in_tissue": pa.array(in_tissue, pa.uint8()), + "array_row": pa.array(rows, pa.uint32()), + "array_col": pa.array(cols, pa.uint32()), + "pxl_row_in_fullres": pxl_row, + "pxl_col_in_fullres": pxl_col, + }) + pq.write_table(table, where / "spatial" / "tissue_positions.parquet", compression="zstd") + + kept = rng.permutation(np.flatnonzero(in_tissue)) + counts = rng.integers(0, 4, size=(kept.size, n_features)) + write_10x_h5(where / "filtered_feature_bc_matrix.h5", counts, barcodes=[every[i] for i in kept], ids=ids, names=names) + write_10x_h5(where / "raw_feature_bc_matrix.h5", np.zeros((len(every), n_features), dtype=int), barcodes=every, ids=ids, names=names) + for path in (where / "filtered_feature_bc_matrix.h5", where / "raw_feature_bc_matrix.h5"): + with h5py.File(path, "a") as f: + f.attrs["library_ids"] = np.array([b"tiny_mouse_brain"]) + f.attrs["chemistry_description"] = "Visium HD v1 - 3'" + f.attrs["software_version"] = "spaceranger-4.0.1" + scalefactors = { + "spot_diameter_fullres": float(size), "bin_size_um": float(size), + "microns_per_pixel": 1.0, "tissue_lowres_scalef": 0.1, + "fiducial_diameter_fullres": 328.9, "tissue_hires_scalef": 1.0, + "regist_target_img_scalef": 1.0, + } + (where / "spatial" / "scalefactors_json.json").write_text(json.dumps(scalefactors)) + pictures = {} + for key, shape in (("hires", (12, 10)), ("lowres", (4, 3))): + pixels = rng.integers(0, 256, size=shape + (3,), dtype=np.uint8) + Image.fromarray(pixels).save(where / "spatial" / f"tissue_{key}_image.png") + Image.fromarray(pixels).save(outs / "spatial" / f"tissue_{key}_image.png") + pictures[key] = pixels + bins[size] = VisiumTruth( + outs=where, + barcodes=[every[i] for i in kept], + in_tissue=in_tissue[kept].astype(np.int64), + array_row=rows[kept].astype(np.int64), + array_col=cols[kept].astype(np.int64), + spatial=np.stack([pxl_col, pxl_row], axis=1)[kept], + images=pictures, + scalefactors=scalefactors, + counts=counts, + ) + return VisiumHDTruth(outs=outs, bins=bins) diff --git a/tests/test_cistopic.py b/tests/test_cistopic.py index 3f3aba3..3365be7 100644 --- a/tests/test_cistopic.py +++ b/tests/test_cistopic.py @@ -244,8 +244,10 @@ def test_repeated_column_labels_keep_the_first_with_a_warning(tmp_path): frame, and the conversion failed.""" truth = write_cistopic_pickle(tmp_path / "obj.pkl", model=False, projections=False) cell_data = truth.obj.cell_data - truth.obj.cell_data = pd.concat([cell_data, cell_data[["TSS_enrichment"]] * 2], axis=1) - truth.path.write_bytes(pickle.dumps(truth.obj, protocol=4)) + # As pycisTopic's pandas would: object strings, not pandas 3's Arrow ones. + with pd.option_context("future.infer_string", False): + truth.obj.cell_data = pd.concat([cell_data, cell_data[["TSS_enrichment"]] * 2], axis=1) + truth.path.write_bytes(pickle.dumps(truth.obj, protocol=4)) out = tmp_path / "out.h5ad" result = _ok(truth.path, "-o", out) assert "Not written: obs column 'TSS_enrichment', a name a dataframe reserves or repeats" in _out(result) diff --git a/tests/test_external_readers.py b/tests/test_external_readers.py index 828678d..8305c7e 100644 --- a/tests/test_external_readers.py +++ b/tests/test_external_readers.py @@ -25,6 +25,7 @@ from adata.cli import app from tests.conftest import ( + write_visium_hd_outs, write_visium_outs, write_xenium_bundle, write_10x_h5, @@ -1146,3 +1147,42 @@ def test_xenium_imports_as_10x_s_scanpy_recipe_reads_it(tmp_path, ids, shuffled) _convert(str(truth.bundle), "-o", str(out), "--var-names", "gene_symbols") assert _run(["scanpy"], _COMPARE_XENIUM, str(truth.bundle), str(out)) == [] + +# Visium HD's bins as scanpy users read them, squidpy having no HD reader: +# the bin folder's matrix, the Parquet positions reindexed to it, and the +# pixel columns as obsm["spatial"], [col, row] as for standard Visium. +_COMPARE_VISIUM_HD = r""" +import json, sys, warnings +from pathlib import Path +warnings.filterwarnings("ignore") +import numpy as np, pandas as pd, anndata as ad, scanpy as sc + +folder, h5ad = Path(sys.argv[1]), sys.argv[2] +ref = sc.read_10x_h5(folder / "filtered_feature_bc_matrix.h5") +positions = pd.read_parquet(folder / "spatial" / "tissue_positions.parquet").set_index("barcode") +positions = positions.reindex(ref.obs_names) +got = ad.read_h5ad(h5ad) +problems = [] +if list(ref.obs_names) != list(got.obs_names): + problems.append("obs_names differ") +if not np.array_equal(ref.X.toarray(), got.X.toarray()): + problems.append("X differs") +for column in ("in_tissue", "array_row", "array_col"): + if not np.array_equal(positions[column].to_numpy().astype(np.int64), got.obs[column].to_numpy()): + problems.append(f"obs[{column!r}] differs") +expected = positions[["pxl_col_in_fullres", "pxl_row_in_fullres"]].to_numpy() +if not np.array_equal(expected, got.obsm["spatial"]): + problems.append("obsm['spatial'] differs") +scalefactors = json.loads((folder / "spatial" / "scalefactors_json.json").read_text()) +if got.uns["spatial"]["tiny_mouse_brain"]["scalefactors"] != scalefactors: + problems.append("scalefactors differ") +print("RESULT " + json.dumps(problems)) +""" + + +def test_visium_hd_imports_as_scanpy_and_pandas_read_it(tmp_path): + hd = write_visium_hd_outs(tmp_path / "run") + out = tmp_path / "hd.h5ad" + _convert(str(hd.outs), "-o", str(out), "--var-names", "gene_symbols") + assert _run(["scanpy", "pyarrow"], _COMPARE_VISIUM_HD, str(hd.bins[8].outs), str(out)) == [] + diff --git a/tests/test_performance.py b/tests/test_performance.py index d249a6d..9f23727 100644 --- a/tests/test_performance.py +++ b/tests/test_performance.py @@ -981,26 +981,32 @@ def measure(n: int) -> int: def _spatial_bundle(path: Path, kind: str, n: int, shuffled: bool) -> Path: - from tests.conftest import write_visium_outs, write_xenium_bundle + from tests.conftest import write_visium_hd_outs, write_visium_outs, write_xenium_bundle if kind == "visium": return write_visium_outs(path, n_spots=n, n_extra=n // 4).outs + if kind == "visium-hd": + # 16 µm bins on a grid of side ~sqrt(n): every bin on the slide in the + # positions, about 60% of them in the matrix. + return write_visium_hd_outs(path, sizes=(16,), grid=max(2, int(n ** 0.5))).outs return write_xenium_bundle(path, n_cells=n, shuffled=shuffled, extra_cells=n // 8 if shuffled else 0).bundle @pytest.mark.parametrize( - "kind,shuffled", [("visium", True), ("xenium", False), ("xenium", True)] + "kind,shuffled", [("visium", True), ("visium-hd", True), ("xenium", False), ("xenium", True)] ) def test_spatial_import_work_grows_linearly(tmp_path, kind, shuffled): """The positions or cell table are matched to the matrix while scanned, and written straight through when in its order (Xenium's own) or through the scatter buckets when not: linear either way, never a lookup per row.""" from adata.commands.convert import convert_store + from adata.conversion import Options def measure(n: int) -> int: source = _spatial_bundle(tmp_path / f"sp{kind}{shuffled}{n}", kind, n, shuffled) + options = Options(bin_size=16) if kind == "visium-hd" else Options() with count_io() as io: - convert_store(source, tmp_path / f"osp{kind}{shuffled}{n}.h5ad", QUIET) + convert_store(source, tmp_path / f"osp{kind}{shuffled}{n}.h5ad", QUIET, options=options) return io.work assert_grows_linearly(measure, what=f"convert {kind} (shuffled={shuffled})", axis="n_cells") diff --git a/tests/test_visium.py b/tests/test_visium.py index d375e86..10104aa 100644 --- a/tests/test_visium.py +++ b/tests/test_visium.py @@ -19,7 +19,7 @@ from typer.testing import CliRunner from adata.cli import app -from tests.conftest import write_visium_outs +from tests.conftest import write_visium_hd_outs, write_visium_outs ad = pytest.importorskip("anndata") @@ -196,11 +196,6 @@ def test_a_repeated_position_is_refused(tmp_path): assert "more than one row for 1 of the matrix's barcodes" in _refused(truth.outs, "-o", tmp_path / "out.h5ad") -def test_visium_hd_is_refused_by_name(tmp_path): - outs = tmp_path / "hd" / "outs" - (outs / "binned_outputs" / "square_008um").mkdir(parents=True) - message = _refused(outs, "-o", tmp_path / "out.h5ad") - assert "Visium HD output, which is not read yet" in message def test_aggr_output_is_refused(tmp_path): @@ -264,3 +259,120 @@ def test_a_library_id_that_cannot_be_a_key_is_refused(tmp_path, name): nested as uns['spatial']['a']['b'].""" truth = write_visium_outs(tmp_path / "run") assert "cannot be a key of uns" in _refused(truth.outs, "-o", tmp_path / "out.h5ad", "--library-id", name) + + +# --------------------------------------------------------------------------- +# Visium HD + + +def _check(adata, truth) -> None: + assert list(adata.obs_names) == truth.barcodes + for column in ("in_tissue", "array_row", "array_col"): + assert adata.obs[column].dtype == np.int64, column + np.testing.assert_array_equal(adata.obs[column], getattr(truth, column)) + assert adata.obsm["spatial"].dtype == np.float64 + np.testing.assert_array_equal(adata.obsm["spatial"], truth.spatial) + np.testing.assert_array_equal(adata.X.toarray(), truth.counts) + + +@pytest.mark.parametrize("suffix", [".h5ad", ".zarr"]) +def test_an_hd_output_reads_its_8_um_bins_by_default(tmp_path, suffix): + hd = write_visium_hd_outs(tmp_path / "run") + out = tmp_path / f"out{suffix}" + + result = _ok(hd.outs, "-o", out) + + assert "Visium HD, 8 µm bins" in _out(result) + adata = _read(out) + truth = hd.bins[8] + _check(adata, truth) + library = adata.uns["spatial"]["tiny_mouse_brain"] + assert library["scalefactors"]["bin_size_um"] == 8.0 + np.testing.assert_array_equal(library["images"]["hires"], truth.images["hires"]) + + +def test_bin_size_picks_another_bin_folder_and_a_bin_folder_reads_alone(tmp_path): + hd = write_visium_hd_outs(tmp_path / "run") + _ok(hd.outs, "-o", tmp_path / "a.h5ad", "--bin-size", "16") + _check(_read(tmp_path / "a.h5ad"), hd.bins[16]) + _ok(hd.bins[16].outs, "-o", tmp_path / "b.h5ad") + _check(_read(tmp_path / "b.h5ad"), hd.bins[16]) + + +def test_unfiltered_reads_every_bin_on_the_slide(tmp_path): + hd = write_visium_hd_outs(tmp_path / "run", sizes=(16,)) + _ok(hd.outs, "-o", tmp_path / "out.h5ad", "--bin-size", "16", "--unfiltered") + assert _read(tmp_path / "out.h5ad").n_obs == 36 + + +def test_a_bin_size_the_output_lacks_names_those_it_has(tmp_path): + hd = write_visium_hd_outs(tmp_path / "run") + message = _refused(hd.outs, "-o", tmp_path / "out.h5ad", "--bin-size", "2") + assert "has no 2 µm bins; it has 8 µm, 16 µm. Pick one with --bin-size" in message + + +def test_bin_size_on_standard_visium_is_refused(tmp_path): + truth = write_visium_outs(tmp_path / "run") + assert "--bin-size picks among a Visium HD output's bins" in _refused( + truth.outs, "-o", tmp_path / "out.h5ad", "--bin-size", "8" + ) + + +def test_hd_without_pyarrow_names_the_extra(tmp_path, monkeypatch): + import builtins + + hd = write_visium_hd_outs(tmp_path / "run") + real_import = builtins.__import__ + + def no_pyarrow(name, *args, **kwargs): + if name.startswith("pyarrow"): + raise ImportError(name) + return real_import(name, *args, **kwargs) + + monkeypatch.setattr(builtins, "__import__", no_pyarrow) + message = _refused(hd.outs, "-o", tmp_path / "out.h5ad") + assert "pip install 'pyadata-cli[spatial]'" in message + assert not list(tmp_path.glob("out*")) + + +def test_bin_size_with_a_matrix_given_is_refused(tmp_path): + """It was ignored: the 8 µm matrix named was read, exit 0.""" + hd = write_visium_hd_outs(tmp_path / "run") + message = _refused( + hd.bins[8].outs / "filtered_feature_bc_matrix.h5", "-o", tmp_path / "out.h5ad", + "--from", "visium", "--bin-size", "16", + ) + assert "--bin-size picks among the matrices" in message + + +def test_bin_size_zero_is_a_size_the_output_lacks_not_the_default(tmp_path): + """0 read as 'not given', and the 8 µm bins were written.""" + hd = write_visium_hd_outs(tmp_path / "run") + assert "has no 0 µm bins" in _refused(hd.outs, "-o", tmp_path / "out.h5ad", "--bin-size", "0") + + +def test_missing_values_in_hd_positions_are_refused_before_writing(tmp_path): + """A null in an integer column became an arbitrary number in obs.""" + import pyarrow as pa + import pyarrow.parquet as pq + + hd = write_visium_hd_outs(tmp_path / "run", sizes=(16,)) + path = hd.bins[16].outs / "spatial" / "tissue_positions.parquet" + table = pq.read_table(path) + rows = table.column("array_row").to_pylist() + rows[0] = None + table = table.set_column(table.schema.get_field_index("array_row"), "array_row", pa.array(rows, pa.uint32())) + pq.write_table(table, path) + + message = _refused(hd.outs, "-o", tmp_path / "out.h5ad", "--bin-size", "16") + + assert "tissue_positions.parquet has missing values in array_row (1)" in message + assert not list(tmp_path.glob("out*")) + + +def test_a_bin_folder_without_scale_factors_names_what_it_lacks(tmp_path): + hd = write_visium_hd_outs(tmp_path / "run", sizes=(16,)) + (hd.bins[16].outs / "spatial" / "scalefactors_json.json").unlink() + message = _refused(hd.outs, "-o", tmp_path / "out.h5ad", "--bin-size", "16") + assert "has no spatial/scalefactors_json.json beside the matrix" in message + assert "tissue_positions.csv" not in message diff --git a/uv.lock b/uv.lock index 2e0a7fd..0238921 100644 --- a/uv.lock +++ b/uv.lock @@ -501,6 +501,7 @@ dependencies = [ dev = [ { name = "anndata" }, { name = "pandas" }, + { name = "pyarrow" }, { name = "pytest" }, { name = "pytest-cov" }, { name = "pytest-timeout" }, @@ -510,6 +511,9 @@ pickle = [ { name = "pandas" }, { name = "scipy" }, ] +spatial = [ + { name = "pyarrow" }, +] [package.metadata] requires-dist = [ @@ -519,6 +523,8 @@ requires-dist = [ { name = "pandas", marker = "extra == 'dev'", specifier = ">=2.2" }, { name = "pandas", marker = "extra == 'pickle'", specifier = ">=2.2" }, { name = "pillow", specifier = ">=12.1.0" }, + { name = "pyarrow", marker = "extra == 'dev'", specifier = ">=17" }, + { name = "pyarrow", marker = "extra == 'spatial'", specifier = ">=17" }, { name = "pytest", marker = "extra == 'dev'", specifier = ">=8.3.4" }, { name = "pytest-cov", marker = "extra == 'dev'", specifier = ">=6.0.0" }, { name = "pytest-timeout", marker = "extra == 'dev'", specifier = ">=2.3.0" }, @@ -528,7 +534,57 @@ requires-dist = [ { name = "typer", specifier = ">=0.20.0" }, { name = "zarr", specifier = ">=3.1.5" }, ] -provides-extras = ["pickle", "dev"] +provides-extras = ["pickle", "spatial", "dev"] + +[[package]] +name = "pyarrow" +version = "26.0.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/ec/34/17c34cb38e5d940e38f0f0d9fdfa0e8a506676409ea9b85aff7e3079f831/pyarrow-26.0.0.tar.gz", hash = "sha256:0cccd36e00ea3afeb52ded61f2721ce71f604853d70c45365c58324eb773d6ae", size = 1239433, upload-time = "2026-10-09T08:26:25.315Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b3/60/6793778f2617cce469383dac0ba08c4f2401cf342df0c7b9ca53939d9b46/pyarrow-26.0.0-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:90ddaf7c625307ad52f31a9b25c34fe5e4897c7529ee3481135822b2b6842ff1", size = 36333953, upload-time = "2026-10-09T08:14:00.387Z" }, + { url = "https://files.pythonhosted.org/packages/db/81/f944cc63ce8a753e5fbff25de6d1d475ebd7fffdf9cf98c65130294fc896/pyarrow-26.0.0-cp312-cp312-macosx_12_0_x86_64.whl", hash = "sha256:ee341973f78a0b46e073d065e88e75026a9c584051e97f98a0d05d96c6bac7dd", size = 38688456, upload-time = "2026-10-09T08:14:04.344Z" }, + { url = "https://files.pythonhosted.org/packages/f5/2d/7e5c722fa5d5d9f3b75e62fe11694b34217664d4f05ac88031197166b277/pyarrow-26.0.0-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:01c863a18bd9c8412453dd0d92de6d0ee7b2b3d6fb079d9734a4b2a3c8bd4453", size = 50867603, upload-time = "2026-10-09T08:14:09.115Z" }, + { url = "https://files.pythonhosted.org/packages/88/e4/9cd356d906e71bd79b0c3fc5c9a54e01a0020dcf14c152ccfbcb503c7298/pyarrow-26.0.0-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:6a628922ba20705fa964ca73e4ef959c2fb2f14b9bbec5589a6a1e68e6257c85", size = 53931932, upload-time = "2026-10-09T08:14:24.051Z" }, + { url = "https://files.pythonhosted.org/packages/bb/e4/5bae3133b7fe04c24907a20f3bc1fba388cbbde659199e7b76445982047a/pyarrow-26.0.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:954d971b363b16ee41f89389a4053315dc71265f2ce5c2468eb0a910b1166268", size = 54444720, upload-time = "2026-10-09T08:14:31.214Z" }, + { url = "https://files.pythonhosted.org/packages/ba/b4/ee422493bb6dafdbef776cfe2c2a73106a1063a79bf4e78d1e5f51176885/pyarrow-26.0.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:5d5768d03426abe6526d5274adefa00abf00a7f81118c46e98b5a46390f5549e", size = 57388949, upload-time = "2026-10-09T08:14:38.964Z" }, + { url = "https://files.pythonhosted.org/packages/54/3c/1783aab1dac28e175dcf26dfc7123725efc474caecaed91e8a34cb89cad0/pyarrow-26.0.0-cp312-cp312-win_amd64.whl", hash = "sha256:cc903e1069e9dd5e9dcf780324c0112e27e051e422ecfaff574fb33ed65d9160", size = 28567581, upload-time = "2026-10-09T08:14:44.279Z" }, + { url = "https://files.pythonhosted.org/packages/4d/35/ca95493712af97c46a312945c8e9d16b21c5fe2f148be5466168d0290505/pyarrow-26.0.0-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:a6ca849f90cf73fe361f08a5762c783ead9671e4548c1f558cc637b54c9103f2", size = 36336700, upload-time = "2026-10-09T08:14:51.399Z" }, + { url = "https://files.pythonhosted.org/packages/69/ef/b1a675f79c9babfd4fcd99af62141d3c2d1a78a524e311b0c6b80110445a/pyarrow-26.0.0-cp313-cp313-macosx_12_0_x86_64.whl", hash = "sha256:c2ba350957076b1b3a22f549261dc3e9c67ca20816d8bd5f79d7b9c69be4c4c2", size = 38698502, upload-time = "2026-10-09T08:14:57.114Z" }, + { url = "https://files.pythonhosted.org/packages/3b/7c/cea852a832a327a8de797b3a68e5c25ce0f5aa1d20503807671bd90ec642/pyarrow-26.0.0-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:e3b190ba1d3d22a5a8758597f797111b77d433473744352a184a5ee0a42d672e", size = 50865064, upload-time = "2026-10-09T08:20:01.614Z" }, + { url = "https://files.pythonhosted.org/packages/4f/d6/e95834b29360092376fe4da9956ba41bb7b021869efe6ee9d4172d05cb15/pyarrow-26.0.0-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:240bd18a7487f8767616a948a69dd4e740a8bc36a1c9da49e4dc9a32c5c2faed", size = 53926722, upload-time = "2026-10-09T08:23:10.829Z" }, + { url = "https://files.pythonhosted.org/packages/e0/7f/98257444e2aea2e1fddceee3af3bd2077236d550428413f80393bd1f888d/pyarrow-26.0.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:2b5fcd69c0e1107b79e55839877db5a6ed04651b73fd6fec581d09e230bed5e4", size = 54443093, upload-time = "2026-10-09T08:23:16.971Z" }, + { url = "https://files.pythonhosted.org/packages/88/ca/dac99cfb25cfa62bf7194600cc99abc14a6bd2af50d7fdb7f15eeaf6e202/pyarrow-26.0.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:f7444ea6975c49a857c68f9bd8fa11acae96dede63d120ffb3bf0a603ea82516", size = 57381937, upload-time = "2026-10-09T08:23:24.95Z" }, + { url = "https://files.pythonhosted.org/packages/c0/ed/138d29fddaf803b90f4527e124bb6aaddc18aaf4a6c50fd0a5f577c94989/pyarrow-26.0.0-cp313-cp313-win_amd64.whl", hash = "sha256:3de30a7432b48b98b9decbd9e25a53bb9251d202c2e6c5a29a50869592ccb117", size = 28478571, upload-time = "2026-10-09T08:23:30.535Z" }, + { url = "https://files.pythonhosted.org/packages/8c/32/01858422a37f083911c2bb4d15cc32c5eeaa9d9b2bf5ddedee995a7146a6/pyarrow-26.0.0-cp314-cp314-macosx_12_0_arm64.whl", hash = "sha256:5780d487ff6c6ed7b42298609680d87fe0036e529a9dc2e1105364bce9697f50", size = 36378402, upload-time = "2026-10-09T08:23:36.537Z" }, + { url = "https://files.pythonhosted.org/packages/00/85/f6b5976c2878b752d0804d371684e0495a71de296b6dc6559e6fbaa4311a/pyarrow-26.0.0-cp314-cp314-macosx_12_0_x86_64.whl", hash = "sha256:a0e4e92eeb088f1d7c2c04d6c7de8434c75abb4b4ccf0bbcd045aa7164c68d93", size = 38733074, upload-time = "2026-10-09T08:23:42.873Z" }, + { url = "https://files.pythonhosted.org/packages/81/bc/c90fcbbcf893631e23dab1b0fb3fa29a508a8614326571b03c0894eda00b/pyarrow-26.0.0-cp314-cp314-manylinux_2_28_aarch64.whl", hash = "sha256:eaf9e7cc7ab59f6c760232bbde18f64d559bbc50544841303bfb32be53533297", size = 50929201, upload-time = "2026-10-09T08:23:50.507Z" }, + { url = "https://files.pythonhosted.org/packages/ec/c1/0c1ff38ab7df1b2cf54cf0ad9f19a516c4e416c6c9b4c966cc2c9d587f77/pyarrow-26.0.0-cp314-cp314-manylinux_2_28_x86_64.whl", hash = "sha256:ab6914db225d7f399652ae1f08588dfbc9efe617612715701e3d9d5cfa5ca19f", size = 53951865, upload-time = "2026-10-09T08:23:57.692Z" }, + { url = "https://files.pythonhosted.org/packages/9f/70/6a6b170496925472adad45a32528770fc8632db35fc60d4edd1e9ce1be0b/pyarrow-26.0.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:41dd3661ef40790a78870052ad7a58ad827b27c67a4511f06962eb9e9b74d19b", size = 54496388, upload-time = "2026-10-09T08:24:05.23Z" }, + { url = "https://files.pythonhosted.org/packages/a8/32/033ef9dba80976820190e292a10a5a23e9406572b76bbeb4d685d90e5c8d/pyarrow-26.0.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:6e949744dcfc2d379808f7013c5f9cafaf0f817656dff7d46c6931528dd1784b", size = 57411588, upload-time = "2026-10-09T08:24:12.043Z" }, + { url = "https://files.pythonhosted.org/packages/1e/ff/a74892c50aaf1f9f744a84493e08a2f99221e77c39d2d4a926de21a99edf/pyarrow-26.0.0-cp314-cp314-win_amd64.whl", hash = "sha256:4a5fa8dc70dd50808990ff36faf44088e357b353d86c7682dd92d4b78d4c97d5", size = 29237858, upload-time = "2026-10-09T08:24:58.106Z" }, + { url = "https://files.pythonhosted.org/packages/03/10/f0ee0976ef08a851a743c57608917ac9a47623f688b9ee0efe5429975ba1/pyarrow-26.0.0-cp314-cp314t-macosx_12_0_arm64.whl", hash = "sha256:e2a1856e9565fe2679863b372478c681806aebbf7d0a6e72f33e77f804e647d6", size = 36495870, upload-time = "2026-10-09T08:24:16.479Z" }, + { url = "https://files.pythonhosted.org/packages/27/ca/0bc431a509bf10b4472dbb94f4184752ecbbddeb7f467152dac0fdaed469/pyarrow-26.0.0-cp314-cp314t-macosx_12_0_x86_64.whl", hash = "sha256:4bcba83299cb2b8f8e443d36c6ba6269a5034431879015fb0719495df8a14de2", size = 38819754, upload-time = "2026-10-09T08:24:20.875Z" }, + { url = "https://files.pythonhosted.org/packages/61/59/2be41d26af7a07fb71581fb753cae396403ba1a2978355fd553929d44a9a/pyarrow-26.0.0-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:3a4d235876f14b4136b4d616ec42eb469ea0d6ead336cae631aa1dd29b21c962", size = 50933671, upload-time = "2026-10-09T08:24:27.199Z" }, + { url = "https://files.pythonhosted.org/packages/4b/cb/b6d5048cf3178be9678f5c9c60040199894b2f69c3439c87ced91fd24da9/pyarrow-26.0.0-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:210cc9b83888b87cdc8f793eebb264f22b20d0dedbedefc73b9687a7047b4747", size = 53906419, upload-time = "2026-10-09T08:24:33.536Z" }, + { url = "https://files.pythonhosted.org/packages/09/2b/23e30fbd776c81d18d134d2592eb60daca13e8a57ab087d0fa042f9d9f3d/pyarrow-26.0.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:ca77c43ca55bfc9a4eeb1f0cd5f093f08731b77c24cdba0829035f084959b0bb", size = 54527960, upload-time = "2026-10-09T08:24:41.292Z" }, + { url = "https://files.pythonhosted.org/packages/e2/23/fce251cd6b0546dfc181b00d5c8ef1c95a8c4cae83266bc3dfd5f719c62c/pyarrow-26.0.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:290a74c48e9491b436fd5edacfadf357943f82aa45c81110bd83a69aab33d1cf", size = 57388010, upload-time = "2026-10-09T08:24:48.186Z" }, + { url = "https://files.pythonhosted.org/packages/44/a5/0126fb0ef8d59bf257bdd68bb41623b72afc6e81790a0b4ac863a0f58861/pyarrow-26.0.0-cp314-cp314t-win_amd64.whl", hash = "sha256:515a10dae2a1d236bc9c9209d0317acb6746ea63cd4f98704904af7156d90ed1", size = 29406123, upload-time = "2026-10-09T08:24:53.387Z" }, + { url = "https://files.pythonhosted.org/packages/ed/66/8ada1b5165359d84b4b9b5384742304d1081da670f77d458fd9c9b8a2161/pyarrow-26.0.0-cp315-cp315-macosx_12_0_arm64.whl", hash = "sha256:e890816e5ee89c74a0f8b9379fe8b5ba83f46132b2a0bbb9b1c21359ec30dfda", size = 36373215, upload-time = "2026-10-09T08:25:03.067Z" }, + { url = "https://files.pythonhosted.org/packages/c4/83/74f10c3d803a6834b2acab21847724d4bdbc74d246eb17321432844707f3/pyarrow-26.0.0-cp315-cp315-macosx_12_0_x86_64.whl", hash = "sha256:9db18a9dc0af52135c9eac549d80a7a882696efbe5406cf882b044525d4ecc2e", size = 38730866, upload-time = "2026-10-09T08:25:07.924Z" }, + { url = "https://files.pythonhosted.org/packages/e2/5a/ea2fa2163b1bd8ff73efd39c4060be63fd6ddec03e7887a471acd1e042a4/pyarrow-26.0.0-cp315-cp315-manylinux_2_28_aarch64.whl", hash = "sha256:734312d3d99088d9ec28c5b17bad40389bd8373a1afc10acb60b83fd217af087", size = 50924443, upload-time = "2026-10-09T08:25:13.864Z" }, + { url = "https://files.pythonhosted.org/packages/78/80/8c47b6cf8cfd42826df65193eff026c1cc81fa6cb213a3c3f5d203e6f67a/pyarrow-26.0.0-cp315-cp315-manylinux_2_28_x86_64.whl", hash = "sha256:24f892fdf1ae1942d69d3f7742e2f49960ec95277cfb1a70b8a1d91f4a96d935", size = 53948540, upload-time = "2026-10-09T08:25:19.305Z" }, + { url = "https://files.pythonhosted.org/packages/69/1f/3a506a76d944ec5c5e4b7f01d8d0446b392a6fb384de627a12e503f616b4/pyarrow-26.0.0-cp315-cp315-musllinux_1_2_aarch64.whl", hash = "sha256:879331ddea2a26479fa18fade71e6facf684a6cf19f67daec3775c871569e8e5", size = 54494863, upload-time = "2026-10-09T08:25:24.517Z" }, + { url = "https://files.pythonhosted.org/packages/3d/50/08c4bb04d651788d2eaca78065743f4f6ded974d4ef96ae3c473993e9d0c/pyarrow-26.0.0-cp315-cp315-musllinux_1_2_x86_64.whl", hash = "sha256:5b827650e874f1f9f9392524ea3e9e3e8a245de5ba64acca1f81ab188090afb9", size = 57409877, upload-time = "2026-10-09T08:25:31.157Z" }, + { url = "https://files.pythonhosted.org/packages/d4/f3/c64781fbd7b6d3c07993b698c14944d0d195f07e800fa931c486ae6ab36a/pyarrow-26.0.0-cp315-cp315-win_amd64.whl", hash = "sha256:8e8e28c464552b5ca03e30d4504168c4425ce383884f8611b00e972f9fd933fc", size = 29236658, upload-time = "2026-10-09T08:26:22.607Z" }, + { url = "https://files.pythonhosted.org/packages/06/55/2ee3729daea999f19f061f03898d4895a242c4cd94f26e1324e5fdfbfe10/pyarrow-26.0.0-cp315-cp315t-macosx_12_0_arm64.whl", hash = "sha256:ce28748cbeb0f29c3ce9603782979c7117580fc76f16aa3ca448b38a22281adb", size = 36489011, upload-time = "2026-10-09T08:25:37.64Z" }, + { url = "https://files.pythonhosted.org/packages/6a/7d/3eb17f601f2bf13eda5f2ed28956379ca628b4dda97619cbb1cb1721622d/pyarrow-26.0.0-cp315-cp315t-macosx_12_0_x86_64.whl", hash = "sha256:106bb9290fc6fd9a84138a9440038ef184bac86463543c5ff099229cb30d996c", size = 38808480, upload-time = "2026-10-09T08:25:43.579Z" }, + { url = "https://files.pythonhosted.org/packages/0e/e3/f0047360b0f4bfc031b256dc0aec3837a61f245b2fb70f8363438e2db665/pyarrow-26.0.0-cp315-cp315t-manylinux_2_28_aarch64.whl", hash = "sha256:2e4a413046eba9896e632925066c74095182200ba32e19ff0166bf64d2f936ac", size = 50923273, upload-time = "2026-10-09T08:25:51.445Z" }, + { url = "https://files.pythonhosted.org/packages/38/d9/56d9fb91210407df31cbeb9b91138601c88c7c8fb5f6bf773b20d65509bf/pyarrow-26.0.0-cp315-cp315t-manylinux_2_28_x86_64.whl", hash = "sha256:d58798c4d8d629700058e9afc1e16b9801023f3ce4dc1c92d945e79b5ffe4e98", size = 53900905, upload-time = "2026-10-09T08:25:59.554Z" }, + { url = "https://files.pythonhosted.org/packages/cf/40/8e8a7e9e027c731520c7eb179dd00a153b76ebf0bc11d213c6c8f8502851/pyarrow-26.0.0-cp315-cp315t-musllinux_1_2_aarch64.whl", hash = "sha256:645917e976671debabf854abab6e2b75c571ca4f82adc33a2d338697f7c27d93", size = 54518345, upload-time = "2026-10-09T08:26:07.125Z" }, + { url = "https://files.pythonhosted.org/packages/be/89/1e768a3fdb88d34e708ad2dc00dbf8e4e30290784eb84198d59308963bea/pyarrow-26.0.0-cp315-cp315t-musllinux_1_2_x86_64.whl", hash = "sha256:7c3fda041e7078802589cf257750323ee3d0cd1e56e53a9b20ec845697fb3d28", size = 57379403, upload-time = "2026-10-09T08:26:13.624Z" }, + { url = "https://files.pythonhosted.org/packages/96/be/7b81a44d6a8e70581dcc1d6f01541f9000a973b1e5d75394aec91e7b179a/pyarrow-26.0.0-cp315-cp315t-win_amd64.whl", hash = "sha256:68cd662e9e2b00876a131950cf32336ace2d0865e1f9418763e3d3be8481dfa4", size = 29389953, upload-time = "2026-10-09T08:26:18.277Z" }, +] [[package]] name = "pydantic"