Repository navigation
convert: Visium HD bins (spatial extra) - #81
Conversation
A Space Ranger 3 or 4 HD outs/ converts at the bin size --bin-size names (8 µm by default, as 10x recommends for analysis); a bin folder given directly reads as it is. Its positions are Parquet only, read with pyarrow, which comes with a new `spatial` extra (and is in the image). - formats/parquet.py reads a buffered stream, one set of columns per pass: the barcodes for the join, then the numbers for the write. On 10x's SR 4.0.1 tiny sample, 2 µm bins (11.2 million positions, 330,035 in the matrix) convert in 3.5 s with a 470 MB peak without images, where pandas alone takes 1.1 GB to read the positions. - The barcode join holds barcodes as bytes, a quarter of numpy's unicode. - Images are written a strip of rows at a time as they are read, not loaded up front: HD's hires image is 6,000 pixels, some 100 MB. Checked on the real sample at 2, 8 and 16 µm against pandas reading the same Parquet, and in CI against scanpy and pandas. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Visium page gains a Visium HD section (layout, --bin-size, the spatial extra, how the Parquet positions stream, measured cost); the formats tables, convert guide and help, Home, README, pipelines guide, Troubleshooting, Developers and AGENTS.md name the extra. STATUS records the work. With pyarrow in the dev environment, pandas 3 builds string columns as ArrowStringArray, so the pycisTopic fixtures pickled a class no real pycisTopic pickle holds (pycisTopic pins pandas 1.5), and the importer's allowlist rightly refused it. The fixtures now build object strings, as pycisTopic's pandas does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.
Test Results (py3.12)1 986 tests 1 986 ✅ 5m 35s ⏱️ Results for commit f1d6ffe. ♻️ This comment has been updated with latest results. |
Test Results (py3.13)1 986 tests 1 986 ✅ 4m 40s ⏱️ Results for commit f1d6ffe. ♻️ This comment has been updated with latest results. |
- --bin-size with a matrix file given is refused, as --unfiltered is; it was ignored and the named matrix read. - --bin-size 0 is a size the output lacks, not "not given". - Nulls in the HD positions' integer columns are refused before writing (from the schema or footer statistics, else counted); they became made-up numbers in obs. - A bin folder missing its scale factors or positions says which, without standard Visium's file names; the bin folder used is the one found. - The Visium page says HD is checked against scanpy and pandas, not squidpy, and which peak is the --no-images one. - STATUS: this goes out as 0.17.1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reads Visium HD, the follow-up STATUS listed first after 0.17.0. Space Ranger 3 and 4 write an HD run's counts at 2, 8 and 16 µm, each in a folder under
binned_outputs/laid out like a standard Visiumouts/, but with the positions as Parquet only. pyarrow reads them, through a newspatialextra, which the Docker image also gets.The output is laid out as for standard Visium:
in_tissue,array_row,array_col;obsm["spatial"]:[x, y]in full-resolution pixels, float64;uns["spatial"][library]: the bin folder's images and scale factors (bin_size_um,microns_per_pixel).Memory
Measured on 10x's SR 4.0.1 tiny HD sample, at the 2 µm bins: 11.2 million positions, 330,035 bins in the matrix.
formats/parquet.pyreads a buffered stream, one set of columns per pass: first the barcodes, which the join matches as they come, then the numbers, for the write. Buffering halves the numeric pass.BarcodeJoinholds barcodes as bytes, a quarter of the size of numpy's unicode.Checked
--bin-sizedefault and choice, and a bin folder given on its own;--unfiltered;--bin-sizeon standard Visium;pyadata-cli[spatial].ArrowStringArray, which real pycisTopic pickles never hold (pycisTopic pins pandas 1.5). The importer refused it, as designed. The fixtures now build object strings.uv run pytest -m "not integration" --cov=adata --cov-fail-under=90: 1,981 passed, coverage 95.5%. The local site build passes html-proofer.Not included: Space Ranger 4's cell segmentation (
segmented_outputs/, GeoJSON). STATUS lists it as a follow-up.🤖 Generated with Claude Code