Optimize modelarrayio to-modelarray runtime

.gitignore

@@ -73,6 +73,12 @@ instance/
 docs/_build/
 docs/generated/
 docs/auto_examples/
+docs/sg_execution_times.rst
+
+# Downloaded/generated benchmark data and results (one set per suite)
+benchmarks/*/cohorts/
+benchmarks/*/data/
+benchmarks/*/results/
 
 # PyBuilder
 target/

README.rst

@@ -71,6 +71,39 @@ Both backends expose a similar API:
 Notes and minor differences:
 
 * Chunking vs tiling: HDF5 uses chunks; TileDB uses tiles. We compute tile sizes analogous to chunk sizes to keep write/read patterns similar.
-* Compression: HDF5 uses ``gzip`` by default; TileDB defaults to ``zstd`` with shuffle for better speed/ratio. You can switch to ``gzip`` for parity.
-* Metadata: HDF5 stores ``column_names`` as a dataset attribute; TileDB stores names as JSON metadata on the array/group.
+* Compression: HDF5 defaults to ``gzip`` level 4; TileDB defaults to ``zstd`` level 5. Compressed outputs use byte shuffle; ``--compression none`` disables the filter pipeline. You can switch TileDB to ``gzip`` for parity.
+* Metadata: HDF5 stores ``column_names`` as a sibling dataset under each scalar group. TileDB stores names as JSON metadata, and CIFTI outputs also retain their dedicated column-name arrays.
 * Layout: Both backends keep dimensions in the same order and use zero-based indices.
+
+Fast conversion and input sources
+=================================
+
+``modelarrayio to-modelarray`` stages one scalar at a time into a typed,
+C-contiguous matrix, then writes that matrix in one backend submission. Use
+``--workers N`` to bound how many input sources are loaded concurrently. This
+meaning applies to local paths and ``s3://`` URLs, every modality, and both
+backends; it does not cap HDF5 compression threads or TileDB's internal pools.
+
+``--workers`` previously controlled concurrent TileDB scalar writes.
+``--s3-workers`` is accepted as a deprecated alias for one minor release.
+Matching values are accepted, conflicting values are rejected, and the alias
+will be removed in the next breaking release.
+
+Local files and materialized DataLad/git-annex symlinks are read directly.
+ModelArrayIO never runs ``datalad get`` implicitly; a broken annex symlink
+reports an actionable error. S3 sources use one shared connection pool sized
+for the loader concurrency. Set ``MODELARRAYIO_S3_ANON=1`` for public buckets.
+
+``--dtype float32`` is the faster, smaller default. ``--dtype float64`` keeps
+source precision through decoding, staging, and storage, at twice the staging
+memory. On the committed 20-subject fixture, three warm runs showed HDF5's
+median write phase increasing from 0.00090 s to 0.00138 s (1.54x) and compressed
+output from 37,460 to 54,355 bytes (1.45x). That fixture is too small for a
+throughput claim; use the reproducible `benchmark harness
+<benchmarks/to_modelarray/README.md>`_ to measure the medium ABIDE cohort on deployment
+hardware.
+
+Pass ``--timing-jsonl PATH`` to append structured per-scalar and total runtime
+records. Outputs are written to a same-filesystem temporary path and validated
+before publication. File replacement is atomic; existing TileDB directories
+use a recoverable rename-and-rollback sequence.

benchmarks/README.md

@@ -0,0 +1,32 @@
+# ModelArrayIO benchmarks
+
+Each benchmark suite lives in its own folder, named after the `modelarrayio` subcommand it
+measures (with underscores). A suite is self-contained: its scripts, committed fixtures, pinned
+results (`baselines/`), rendered findings (`reports/`), its own `README.md`, and the gitignored
+inputs and outputs it produces.
+
+| Suite | Measures | Guide |
+| --- | --- | --- |
+| [`to_modelarray/`](to_modelarray/) | `modelarrayio to-modelarray` runtime and output correctness across NIfTI, CIFTI and MIF on HDF5 and TileDB, including the original-vs-optimized comparison and the CUBIC (HPC) validation ladder | [`to_modelarray/README.md`](to_modelarray/README.md) |
+
+## Suite layout
+
+Each suite keeps the same three workspace folders, ignored by Git through the
+`benchmarks/*/{data,cohorts,results}/` patterns in `.gitignore`:
+
+- `data/`: downloaded or synthesized input files.
+- `cohorts/`: generated cohort CSVs. Locally materialized cohorts contain absolute paths into
+  `data/`, so regenerate them (or rewrite the path prefix) after moving the checkout.
+- `results/<run-id>/`: per-run outputs, logs, timings and `run_meta` files. Set
+  `MODELARRAYIO_BENCHMARK_RESULTS_DIR` to write them elsewhere (for example cluster scratch).
+
+Run every command from the repository root.
+
+## Adding a suite
+
+Create `benchmarks/<subcommand>/` with its own `README.md`, `fixtures/`, `baselines/` and
+`reports/`, and add a row to the table above. When a second suite needs a tool or input that
+currently lives in `to_modelarray/` (the shared modules `harness_common.py` and
+`output_compare.py`, the cohort generators, `source_identity.py`, the old-commit launcher in
+`baselines/`, or a downloaded cohort), move it to `benchmarks/common/` and update both suites;
+until then, keep each tool and its data with the suite that uses it.