Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,8 @@ Each layer calls only the layers below it:
- `csr_build.py` builds a CSR matrix from entries in any order, through
row-range buckets on disk. The transpose uses it, and so does anything
that reads entries out of order;
- `scatter.py` writes rows read in one order (a table beside a matrix) in
another, through the same kind of scratch buckets;
- `query.py` and `select.py` hold the expression language behind
`--obs-query`.
- **`elements/`**: the AnnData on-disk spec.
Expand Down Expand Up @@ -99,7 +101,11 @@ Each layer calls only the layers below it:
an output only with `--to`. Order in `FORMATS` matters: the first sniffer
that matches wins, so a more specific format (`cellbender`, which is also a
10x matrix) goes before the general one, and an output extension picks the
first format that can be written.
first format that can be written. The folder formats `visium` and `xenium`
go before `10x-mtx`, which would take a Space Ranger `outs/` for a Cell
Ranger 2 genome layout. `conversion/spatial.py` matches a table to a
matrix's barcodes while `scan_table` reads it (`on_index`), and writes it
in the matrix's order.
- **`formats/`**: the per-file-type halves of `export` and `import`: npy, mtx,
csv, json and images. `mtx.py` is a streaming Matrix Market codec (two
passes in, block writes out) shared by `import`/`export sparse` and the
Expand Down
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,8 @@ without loading them into memory and without a Python session. `adata` inspects,
subsets, splits, concatenates, recasts and converts stores, and moves elements in
and out as CSV, JSON, `.npy`, `.mtx` or images. It implements the AnnData on-disk
spec directly against h5py and zarr, so it streams stores larger than RAM, and
converts to and from 10x, STARsolo, CellBender, Parse, loom, MuData and more.
converts to and from 10x, STARsolo, Visium, Xenium, CellBender, Parse, loom, MuData
and more.

**Documentation: <https://cellgeni.github.io/adata-cli/>**

Expand Down
35 changes: 31 additions & 4 deletions STATUS.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,40 @@ The conversion epic's sub-issues (#21–#30) are all done and ticked in #20.

## In progress

Nothing. The documentation site is live (#76, released in 0.16.2); the
spatial work below can resume.
### Spatial import: standard Visium and Xenium (`feat/spatial`, for 0.17.0)

Decisions:
- Standard Visium (Space Ranger 1–4) and Xenium (XOA 1–4) first, with no new
dependency: both come as CSV, JSON and PNG. Visium HD, whose positions are
Parquet only, waits for a `spatial` extra with pyarrow.
- Laid out as squidpy (Visium) and spatialdata-io's table (Xenium) lay
them out; images stay the PNGs' uint8 pixels. Xenium brings its own
clusters, UMAP and PCA, read from `analysis.tar.gz` as it ships.
- A table beside the matrix is matched to its barcodes while scanned
(`conversion/spatial.py`), and written straight through when in the
matrix's order (Xenium's cells, on every real bundle checked) or through
`core/scatter.py` when not.

Steps:
- [x] `table.py`: headerless files, `on_index`, reusable columns
- [x] `core/scatter.py`
- [x] Visium: `visium.py`, flags `--library-id`, `--no-images`, `--unfiltered`
- [x] Xenium: `xenium.py`, analysis from the folder or the archive
- [x] Performance guards; squidpy and 10x-recipe comparisons; `import-xenium` benchmark
- [x] Real data by hand: Visium SR 1.1 and SR 2.1, Xenium XOA 3.0 and 4.0 tiny
- [x] Docs: two format pages, tables, guide, Troubleshooting, Developers
- [ ] PR, review, merge; release 0.17.0

## Follow-ups noticed

- Visium HD bins and Space Ranger 4's cell segmentation, behind a `spatial`
extra (pyarrow) for their Parquet positions.
- Spatial bundles whose matrix is a MEX folder only (refused for now, with a
pointer to `--from 10x-mtx`).
- Space Ranger's own `analysis/` (clusters, UMAP) for Visium, as Xenium's.
- CellBender's `_write_embeddings` onto `core/scatter.py`, which generalises
it.

- Fixed in 0.16.2 (#77): `export image` without `-o` crashed (`'NoneType'
object has no attribute 'parent'`); it now writes PNG to stdout, as every
other export writes there. Found by the docs review.
Expand All @@ -37,7 +66,5 @@ spatial work below can resume.

## Next candidates (#31)

- Spatial: Visium / Xenium (matrix via 10x h5, coordinates to
`obsm["spatial"]`)
- alevin-fry USA mode, kallisto | bustools
- Seurat `.h5Seurat`
37 changes: 37 additions & 0 deletions benchmarks/cases.py
Original file line number Diff line number Diff line change
Expand Up @@ -489,6 +489,43 @@ def _py(body: str) -> str:
],
tags=["headline"],
),
Case(
name="import-xenium",
question=(
"Read a Xenium bundle of the same counts -- the cell matrix, a "
"cells.csv.gz of centroids and per-cell counts, and analysis "
"clusters -- into an h5ad with obsm['spatial']. adata-cli matches "
"the cell table to the matrix as it scans it and streams both; "
"10x's scanpy recipe loads the matrix and reads the tables with "
"pandas. adata-cli keeps the matrix's own gzip, scanpy writes lzf."
),
sidecar="xenium",
contenders=[
Contender(
"adata-cli",
argv=["adata", "convert", "{sidecar}", "-o", "{output}"],
),
Contender(
"scanpy",
env="scanpy",
script=_py(
"import os, pandas as pd, scanpy as sc\n"
"from pathlib import Path\n"
"bundle = Path(os.environ['BENCH_SIDECAR'])\n"
"obj = sc.read_10x_h5(bundle / 'cell_feature_matrix.h5', gex_only=False)\n"
"cells = pd.read_csv(bundle / 'cells.csv.gz', index_col='cell_id')\n"
"cells = cells.reindex(obj.obs_names)\n"
"obj.obsm['spatial'] = cells[['x_centroid', 'y_centroid']].to_numpy()\n"
"obj.obs = cells.drop(columns=['x_centroid', 'y_centroid'])\n"
"clusters = pd.read_csv(bundle / 'analysis/clustering/"
"gene_expression_graphclust/clusters.csv', index_col='Barcode')\n"
"obj.obs['graphclust'] = clusters['Cluster'].astype(str)"
".reindex(obj.obs_names).astype('category')\n"
"obj.write_h5ad(OUT, compression=COMPRESSION)\n"
),
),
],
),
Case(
name="import-csv",
question=(
Expand Down
7 changes: 4 additions & 3 deletions benchmarks/page_template.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,11 @@ for time, and a table that hid the cost would not be worth publishing.

## What is measured

Nineteen cases on a GitHub-hosted runner: concat (inner, outer, `--merge same`),
Twenty cases on a GitHub-hosted runner: concat (inner, outer, `--merge same`),
subset by query, h5ad→zarr (`convert`), `view`, `ls`, `create`, `export` dataframe /
array / sparse, `import dataframe`, MTX folder import (`convert`), `split`, `recast` narrowing and layout, and a
peak-RSS-against-input-size sweep. Inputs
array / sparse, `import dataframe`, MTX folder, text table and Xenium bundle imports
(`convert`), `split`, `recast` narrowing and layout, and a peak-RSS-against-input-size
sweep. Inputs
are 50,000 × 20,000 at 5% density, plus the 2,000 × 36,601 shape that hung in 0.5.1.
`recast` narrowing starts from a copy of the main input stored as float64 values
with int64 indices, once with X uncompressed (as `write_h5ad` leaves it) and once
Expand Down
70 changes: 70 additions & 0 deletions benchmarks/run.py
Original file line number Diff line number Diff line change
Expand Up @@ -187,6 +187,8 @@ def _build_sidecar(case: Case, inputs: List[Path], workdir: Path) -> Optional[Pa
return _build_mtx_sidecar(case, inputs, workdir)
if case.sidecar == "dense-csv":
return _build_dense_csv_sidecar(case, inputs, workdir)
if case.sidecar == "xenium":
return _build_xenium_sidecar(case, inputs, workdir)
if case.sidecar != "csv":
return None
# Named for its input too: a work directory reused across tiers must not
Expand Down Expand Up @@ -245,6 +247,74 @@ def _build_dense_csv_sidecar(case: Case, inputs: List[Path], workdir: Path) -> P
return path


def _build_xenium_sidecar(case: Case, inputs: List[Path], workdir: Path) -> Path:
"""A Xenium bundle holding the first input's counts, as XOA lays one out.

The matrix is exported by adata-cli (integer counts, then the 10x h5),
and the cell table and clusters are written in the matrix's order, as
Xenium writes them, with made-up centroids and per-cell counts.
"""
import gzip

import h5py
import numpy as np

from adata.elements.read import element_len, read_str_chunk, resolve_index

path = workdir / f"{case.name}-{inputs[0].stem}-sidecar"
if path.exists():
return path
partial = path.with_name(path.name + ".partial")
if partial.exists():
shutil.rmtree(partial)
partial.mkdir()
counts = workdir / f"{case.name}-{inputs[0].stem}-counts.h5ad"
if counts.exists():
counts.unlink()
for command in (
["adata", "recast", str(inputs[0]), "X", "-o", str(counts), "--dtype", "int32"],
["adata", "convert", str(counts), "-o", str(partial / "cell_feature_matrix.h5"),
"--gene-names-column", "gene_symbol", "--feature-types-column",
"feature_type", "--force"],
):
done = subprocess.run(command, capture_output=True, text=True)
if done.returncode != 0:
raise RuntimeError(f"could not build the Xenium sidecar:\n{done.stderr or done.stdout}")
counts.unlink()
(partial / "experiment.xenium").write_text('{"pixel_size": 0.2125, "major_version": 5}')

rng = np.random.default_rng(0)
clusters_dir = partial / "analysis" / "clustering" / "gene_expression_graphclust"
clusters_dir.mkdir(parents=True)
with h5py.File(inputs[0], "r") as handle, gzip.open(
partial / "cells.csv.gz", "wt", compresslevel=6
) as cells, open(clusters_dir / "clusters.csv", "w") as clusters:
index, _ = resolve_index(handle["obs"], "obs")
n = element_len(index)
cells.write(
"cell_id,x_centroid,y_centroid,transcript_counts,control_probe_counts,"
"cell_area,nucleus_area,segmentation_method\n"
)
clusters.write("Barcode,Cluster\n")
step = 50_000
for lo in range(0, n, step):
names = read_str_chunk(index, lo, min(lo + step, n), {}, parent_group=handle["obs"])
k = len(names)
x, y = rng.uniform(0, 8000, k), rng.uniform(0, 8000, k)
counts_, probes = rng.integers(0, 500, k), rng.integers(0, 3, k)
area, nucleus = rng.uniform(20, 300, k), rng.uniform(5, 60, k)
method = rng.choice(["Segmented by boundary stain", "Segmented by nucleus expansion"], k)
cluster = rng.integers(1, 20, k)
cells.writelines(
f"{c},{x[i]:.4f},{y[i]:.4f},{counts_[i]},{probes[i]},{area[i]:.3f},{nucleus[i]:.3f},{method[i]}\n"
for i, c in enumerate(names)
)
clusters.writelines(f"{c},{cluster[i]}\n" for i, c in enumerate(names))
# Renamed only when whole: a failed build must not be reused as a sidecar.
partial.rename(path)
return path


def _build_mtx_sidecar(case: Case, inputs: List[Path], workdir: Path) -> Path:
"""A Cell Ranger 3 MTX folder holding the first input's counts.

Expand Down
12 changes: 9 additions & 3 deletions docs/developers.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,9 @@ cli.py ──► commands/ ──► core/, conversion/, formats/ ──► elem
- **`core/`** holds the algorithms that stream. `subset.py`'s writer serves
`subset` and `split`, and reads each matrix once for every output.
`csr_build.py` builds CSR from entries in any order through row-range buckets
on disk; the transpose and every out-of-order reader use it.
on disk; the transpose and every out-of-order reader use it. `scatter.py`
does the same for the rows of a table that comes in another order than the
matrix beside it.
- **`elements/spec.py`** is the single source of truth for `encoding-type` and
`encoding-version`. `read.py` accepts every layout from anndata 0.7 on;
`write.py` always writes the current one; `strings.py` reconciles how HDF5
Expand All @@ -76,9 +78,13 @@ a converter to another format.

1. Add a `Format` to `FORMATS` in `conversion/__init__.py`, with a content
sniffer. The first sniffer to match wins, so a specific format
(`cellbender`, which is also a 10x matrix) goes before a general one.
(`cellbender`, which is also a 10x matrix) goes before a general one, and
the folder formats (`visium`, `xenium`) go before `10x-mtx`, which would
otherwise claim their MEX subfolders.
2. Write the converter module; `tenx.py` is the model, and `features.py` holds
the feature table every 10x-style format shares.
the feature table every 10x-style format shares. A format that pairs a 10x
matrix with a table (`visium.py`, `xenium.py`) uses `spatial.py` to match
the table to the matrix's barcodes and write it in their order.
3. Add fixtures shaped like the real tool's output to `tests/conftest.py`, and
a comparison against the reader people use in
`tests/test_external_readers.py`.
Expand Down
9 changes: 6 additions & 3 deletions docs/guides/convert.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,9 @@ to turn a loom into a 10x matrix, convert it to `.h5ad` first, then from that.

```
10x HDF5, 10x/STARsolo MTX ─┐ ┌─► 10x HDF5, 10x MTX
CellBender, Parse, text ├─► .h5ad ◄──► .zarr ◄──┤
loom, HDF5, pycisTopic │ ├─► loom (SCope too)
Visium, Xenium, CellBender │ │
Parse, text, HDF5 ├─► .h5ad ◄──► .zarr ◄──┤
loom, pycisTopic │ ├─► loom (SCope too)
MuData ─────────────────────┘ └─► MuData
```

Expand All @@ -38,7 +39,9 @@ no extension.

| You have | Convert with | Read the page |
|---|---|---|
| Cell Ranger, Space Ranger or Xenium `*_feature_bc_matrix.h5` | `adata convert in.h5 -o out.h5ad` | [10x HDF5](../reference/formats/10x-h5.md) |
| Cell Ranger's `*_feature_bc_matrix.h5` | `adata convert in.h5 -o out.h5ad` | [10x HDF5](../reference/formats/10x-h5.md) |
| A Space Ranger `outs/` folder (standard Visium) | `adata convert outs -o out.h5ad` | [Visium](../reference/formats/visium.md) |
| A Xenium output bundle | `adata convert bundle -o out.h5ad` | [Xenium](../reference/formats/xenium.md) |
| CellBender `output.h5` | `adata convert output.h5 -o out.h5ad` | [CellBender](../reference/formats/cellbender.md) |
| A `matrix.mtx` folder from Cell Ranger, STARsolo or GEO | `adata convert folder -o out.h5ad` | [MTX folders](../reference/formats/10x-mtx.md) |
| A Parse Biosciences split-pipe run | `adata convert run -o out.h5ad` | [Parse](../reference/formats/parse.md) |
Expand Down
7 changes: 4 additions & 3 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,10 +68,11 @@ Status goes to stderr and data to stdout, so every command pipes cleanly.
checks this against stores written by anndata 0.8 to 0.13, and checks that
each of those releases reads what `adata` writes.
- **Other formats, through `convert`**: 10x Genomics HDF5 and MTX folders
(Cell Ranger, STARsolo, velocyto), CellBender output, Parse Biosciences
runs, looms (velocyto's and SCope's), text tables, plain HDF5 matrices,
(Cell Ranger, STARsolo, velocyto), Visium and Xenium spatial outputs with
their coordinates and images, CellBender output, Parse Biosciences runs,
looms (velocyto's and SCope's), text tables, plain HDF5 matrices,
pycisTopic pickles and MuData. CI checks them against the readers people
use for them: scanpy, loompy, anndata, mudata.
use for them: scanpy, squidpy, loompy, anndata, mudata.
- **Outputs that keep the source's choices**: Zarr version, index dtypes,
chunking and compression carry over unless a flag changes them.
- **Outputs that appear whole or not at all**: every store is written beside
Expand Down
6 changes: 5 additions & 1 deletion docs/reference/commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,8 @@ adata convert filtered_feature_bc_matrix.h5 -o counts.h5ad
adata convert counts.h5ad -o for_cellbender.h5 --layer counts
adata convert Solo.out/Gene/filtered -o counts.h5ad
adata convert counts.h5ad -o counts_mtx --to 10x-mtx
adata convert spaceranger_run/outs -o section.h5ad
adata convert xenium_bundle -o cells.h5ad
adata convert splitpipe_out -o parse.h5ad
adata convert velocyto.loom -o velocyto.h5ad
adata convert counts.h5ad -o counts.loom
Expand All @@ -143,7 +145,9 @@ adata convert multiome.h5ad -o multiome.h5mu
|---|---|---|---|
| `h5ad` | `.h5ad` | yes | yes |
| `zarr` | `.zarr` (v2 or v3) | yes | yes |
| [`10x-h5`](formats/10x-h5.md) | `.h5` | Cell Ranger 2 and 3+, Space Ranger, Xenium | Cell Ranger 3+ layout |
| [`10x-h5`](formats/10x-h5.md) | `.h5` | Cell Ranger 2 and 3+; the matrix alone of Space Ranger and Xenium | Cell Ranger 3+ layout |
| [`visium`](formats/visium.md) | a folder | Space Ranger `outs/`: counts, spot positions, images (standard Visium; not HD yet) | no |
| [`xenium`](formats/xenium.md) | a folder | a Xenium bundle: counts, cells and centroids, Xenium's analysis | no |
| [`cellbender`](formats/cellbender.md) | `.h5` | CellBender `remove-background` 0.2 and later | no |
| [`10x-mtx`](formats/10x-mtx.md) | a folder (`--to 10x-mtx`) | Cell Ranger 2 and 3+, STARsolo (velocyto too) | Cell Ranger 3+ layout, gzipped |
| [`parse`](formats/parse.md) | a folder | Parse Biosciences split-pipe: a run, a sample or a `DGE_*` folder | no |
Expand Down
5 changes: 4 additions & 1 deletion docs/reference/formats/10x-h5.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,10 @@ nav_order: 1

Cell Ranger's `filtered_feature_bc_matrix.h5` and `raw_feature_bc_matrix.h5`,
and the same layout from Space Ranger, Xenium and CellBender. The file is
recognised by its content, whatever its name.
recognised by its content, whatever its name. Given alone, a Space Ranger or
Xenium matrix converts without its positions: convert the
[Space Ranger folder](visium.md) or the [Xenium bundle](xenium.md) to keep
them.

```bash
adata convert filtered_feature_bc_matrix.h5 -o counts.h5ad
Expand Down
2 changes: 1 addition & 1 deletion docs/reference/formats/cistopic.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: pycisTopic pickles
parent: Formats
grand_parent: Reference
nav_order: 8
nav_order: 10
---

# pycisTopic pickles
Expand Down
2 changes: 1 addition & 1 deletion docs/reference/formats/hdf5.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: HDF5 matrices
parent: Formats
grand_parent: Reference
nav_order: 7
nav_order: 9
---

# HDF5 matrices
Expand Down
4 changes: 3 additions & 1 deletion docs/reference/formats/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,11 @@ flags every conversion takes (`--output`, `--from`, `--to`, `--compression`,

| Format | `--from` / `--to` | Read | Write |
|---|---|---|---|
| [10x Genomics HDF5](10x-h5.md) | `10x-h5` | Cell Ranger 2 and 3+, Space Ranger, Xenium | Cell Ranger 3+ layout |
| [10x Genomics HDF5](10x-h5.md) | `10x-h5` | Cell Ranger 2 and 3+; the matrix alone of Space Ranger and Xenium | Cell Ranger 3+ layout |
| [CellBender](cellbender.md) | `cellbender` | `remove-background` 0.2 and later, with its latents | no |
| [10x and STARsolo MTX folders](10x-mtx.md) | `10x-mtx` | Cell Ranger 2 and 3+, STARsolo (velocyto too), GEO prefixes | Cell Ranger 3+ layout, gzipped |
| [Visium (Space Ranger)](visium.md) | `visium` | an `outs/` folder: counts, spot positions, tissue images, scale factors (Space Ranger 1–4; not HD yet) | no |
| [Xenium](xenium.md) | `xenium` | an output bundle: counts, cells and centroids, Xenium's clusters, UMAP and PCA | no |
| [Parse Biosciences](parse.md) | `parse` | split-pipe: a run, a sample or a `DGE_*` folder | no |
| [loom and SCope](loom.md) | `loom` | loompy, velocyto, kallisto\|bustools, SCope; spec 2 and 3 | spec 3.0.0; a SCope loom with `--scope` |
| [Text tables](text.md) | `text` | a dense matrix in `.csv`, `.tsv`, `.txt` and the like, gzipped too | no |
Expand Down
2 changes: 1 addition & 1 deletion docs/reference/formats/loom.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: loom and SCope
parent: Formats
grand_parent: Reference
nav_order: 5
nav_order: 7
---

# loom and SCope looms
Expand Down
2 changes: 1 addition & 1 deletion docs/reference/formats/mudata.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: MuData
parent: Formats
grand_parent: Reference
nav_order: 9
nav_order: 11
---

# MuData
Expand Down
2 changes: 1 addition & 1 deletion docs/reference/formats/parse.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: Parse Biosciences
parent: Formats
grand_parent: Reference
nav_order: 4
nav_order: 6
---

# Parse Biosciences split-pipe
Expand Down
2 changes: 1 addition & 1 deletion docs/reference/formats/text.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: Text tables
parent: Formats
grand_parent: Reference
nav_order: 6
nav_order: 8
---

# Text tables
Expand Down
Loading
Loading