[Docs] groups, vector point source, block-wise reading and CLI reference

CHANGELOG.md

@@ -8,6 +8,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Documentation
+- New guides: command line (`validate`, `fetch`, `run`, `export`, `plot`) and block-wise reading (the block budget, which operators and sources are read by blocks, measured runs).
+- `groups` of `majority` and `minority` (MapBiomas level-1 classes) in the operators page, the pipeline-file parameters and the MapBiomas guide; the point-source example of `distance`; the contents of `run.json`.
+
 ---
 
 ## [0.6.0] - 2026-10-14

docs/architecture/operators.md

@@ -60,6 +60,11 @@ These operators receive a high-resolution array snapped to the target grid origi
 Each of the operators above accepts `params = {"subcells": n}`, a cap on the fine pixels per cell
 along each axis (see `GridAligner._align_fine`).
 
+`majority` and `minority` accept `params = {groups = {label = [codes]}}`: the map is read at a coarser level, and the
+result is the label of the group that covers most (least) of the cell. The labels are integers (TOML keys are written as
+strings, `"1"`), a code cannot be in two groups, and codes in no group, and nodata, are ignored. With it one derivation
+turns a map of leaf classes into the MapBiomas level-1 classes; see [MapBiomas](../guides/mapbiomas.md#level-1-classes).
+
 The last three also produce `coverage_purity` and `dominance_purity` as coordinates of the output `DataArray` (persisted in the Zarr alongside the variable).
 
 ### Proximity — vector (and passthrough for raster)

docs/guides/blocks.md

@@ -0,0 +1,62 @@
+# Block-wise reading
+
+A national raster does not fit in memory: one year of MapBiomas for Brazil has 154,470 × 146,501 pixels, 22.6 GB at one
+byte per pixel. DisSCube does not load it. It processes the grid in **blocks of rows and columns**, and the peak
+memory depends on the size of a block, not on the size of the source.
+
+## What happens in a block
+
+For each block of grid cells DisSCube
+
+1. finds the window of the source that covers the block (in the source's CRS, clipped to the source);
+2. reads only that window — MapBiomas and PRODES are tiled GeoTIFFs, so the rest is not fetched;
+3. reprojects it with GDAL to the grid projection (nearest neighbour onto fine pixels for the class operators, which
+   count them; averaging for `mean`);
+4. aggregates the pixels into the values of the block's cells, and writes them to the product.
+
+The blocks of one source are written to the same product, so the result is the grid, not a set of tiles.
+
+## How big a block is
+
+The budget is the number of **source pixels** aligned at once: 50 million by default. It is measured, not assumed:
+a strip of grid rows that is wide in a projected grid can cover a far larger area of a longitude/latitude source than
+its height suggests. So DisSCube starts from the whole grid and halves a block along its larger side until the block
+that covers the most source pixels is within the budget. If the whole grid is within it, there are no blocks.
+
+Change the budget with the environment variable `DISSCUBE_BLOCK_PIXELS` (or `block_pixels=` in
+`CubeClient.derive`): a smaller budget lowers the peak and makes more, smaller reads.
+
+## When a source is not read by blocks
+
+A source is derived whole when
+
+- it is **not a raster** (a vector layer is reprojected, clipped to the grid and used whole; it is small);
+- a variable uses an operator that is not block-safe. The block-safe ones are `majority`, `minority`, `percentage`,
+  `mean`, `median`, `std`, `sum`, `min` and `max`. `distance`, `min_distance`, `count`, `area`, `attribute`, `presence` and
+  `network_cost` look at neighbours or at the whole layer;
+- a variable uses `fill = "nearest"`, because a cell then depends on its neighbours;
+- the file cannot be opened as a plain raster (a NetCDF subdataset, some URLs).
+
+The Brazil Data Cube source (STAC) is not block-read either: the window of each item is read whole, and the items of the
+period are reduced to a per-pixel `median` (or the `reducer` of the source) before the aggregation. It is the case with
+the highest peak of the runs below, from a small source.
+
+## Time
+
+Each derived variable re-reads and re-aligns the source, so a pipeline with several variables from the same raster
+makes several passes over it. Reading is what takes the time.
+
+## Measured runs
+
+On one core of a desktop computer (Intel Core i7-7700T, 15 GB of RAM), reported in
+[disscube-grids](https://github.com/LambdaGeo/disscube-grids), one folder per case:
+
+| Case | Source | Grid | Time | Peak memory |
+|---|---|---|---|---|
+| Brazil | MapBiomas C11, 2000 (22.6 × 10⁹ pixels) | 5.28 km; 900 × 1,040 | 1 h 43 min | 3.9 GB |
+| Legal Amazon | PRODES, 2008 and 2024 (≈ 9.7 × 10⁹ pixels each) | 5.28 km; 680 × 520 | 27 min | 2.9 GB |
+| Pará, classes | MapBiomas 2024 and PRODES 2024 | 5 km; 280 × 291 | 2 min 12 s | 1.2 GB |
+| Pará, NDVI | BDC MODIS NDVI, 16-day composites | 5 km; 280 × 291 | 7 min 15 s | 4.2 GB (source not block-read) |
+
+Each case was run once; the times show that the run is feasible, not how fast it is compared with other tools. Peak
+memory is the maximum resident set size reported by `/usr/bin/time -v`.

docs/guides/cli.md

@@ -0,0 +1,80 @@
+# Command line
+
+The `disscube` command works on a [pipeline file](pipeline_files.md). Five subcommands:
+
+| Command | What it does | Network |
+|---|---|---|
+| `disscube validate FILE` | checks the pipeline file and prints the plan; nothing is fetched | no |
+| `disscube fetch FILE` | downloads the remote `file` sources of the pipeline and checks their SHA-256 | yes |
+| `disscube run FILE` | fetches the sources, derives the variables and, with `--output`, exports them | yes |
+| `disscube export FILE -o OUT` | exports variables from a workspace that already has them | no |
+| `disscube plot TIF` | draws the bands of an exported GeoTIFF in a PNG | no |
+
+## `validate`
+
+```bash
+disscube validate sources.toml          # the plan, or the first error
+disscube validate sources.toml --json   # the same as JSON
+```
+
+An unknown operator, a `percentage` without `class_code`, a source that does not exist or a parameter the operator does
+not know fail here, before any download.
+
+## `fetch`
+
+Downloads the `file` sources that declare a `url` (and a `sha256`) into the workspace cache, and verifies them. It is
+the step to run once before working offline.
+
+## `run`
+
+```bash
+disscube run sources.toml --workspace data/sources --output data/sources.tif
+```
+
+| Option | Meaning |
+|---|---|
+| `--workspace DIR` | where the run keeps its files; default: the `workspace` of the file, otherwise a folder named after the file |
+| `--output`, `-o FILE` | export the derived variables: GeoTIFF, or netCDF if the name ends in `.nc` |
+| `--dry-run` | simulate the plan without downloading or computing |
+| `--json` | print the execution report as JSON |
+| `-v`, `--verbose` | log each step |
+
+The workspace receives `catalog.db`, `store/`, `raw/` and `run.json` (see
+[what a run leaves behind](pipeline_files.md#what-a-run-leaves-behind)). The exported GeoTIFF has one band per variable
+and per year, named after the variable; `-o` with an existing workspace redoes only what changed.
+
+## `export`
+
+```bash
+disscube export sources.toml -o data/cellspace.tif --workspace data/sources --variables l1_class
+```
+
+Exports from an existing workspace without deriving anything. `--variables` selects some of the derived variables
+(default: all); `--workspace` defaults to `data/cube` or the workspace of the file. The same output is made by
+`run --output`.
+
+## `plot`
+
+```bash
+pip install 'disscube[plot]'            # needs matplotlib
+disscube plot data/sources.tif -o look.png --cols 2 --crop
+```
+
+A quick look, not a publication figure: one panel per band; the scale follows the values (shares in [0, 1] as 0 to
+100 %, a few integer codes as classes, otherwise continuous between the 2nd and 98th percentiles); cells without data
+are white.
+
+| Option | Meaning |
+|---|---|
+| `--output`, `-o FILE` | PNG; default: the GeoTIFF's name with `.png` |
+| `--bands NAME …` | only these bands or variables |
+| `--year N` | only the bands of this year |
+| `--cols N` | panels per row |
+| `--shared-colorbar` | one colour bar for all panels, when all are shares |
+| `--crop` | trim the empty margin (the bounding box of the cells with data) |
+| `--dpi N` | resolution of the PNG (default 150) |
+
+## Exit status
+
+`0` when the command finished; `2` for an error of the pipeline file, a missing optional extra or an unreadable
+file, with the message on the standard error (as JSON when `--json` is given).

docs/guides/mapbiomas.md

@@ -49,6 +49,30 @@ example open sea outside the mapped area). The reader treats 0 as nodata and
 writes the local file with `nodata=0`, so it never counts as a class in
 `majority` or `percentage`.
 
+## Level-1 classes
+
+The leaf classes of the legend are grouped in five level-1 classes: Forest, Herbaceous and shrubby vegetation, Farming,
+Non vegetated area and Water. `majority` reads a map at that level with `groups`, in one derivation and without a
+reclassified copy of the map. For Collection 11 (legend of 13 August 2026):
+
+```toml
+[[derive]]
+target = "l1_class"
+source = "lulc_{year}"
+operator = "majority"
+
+[derive.params.groups]
+"1" = [1, 3, 4, 5, 6, 7, 49]                                  # forest, savanna, mangrove, floodable forest, flooded savanna, wooded sandbank
+"2" = [10, 11, 12, 29, 32, 50, 77, 84]                        # wetland, grassland, herbaceous and shrub formation, rocky outcrop, tidal flat, salt marsh, herbaceous sandbank
+"3" = [9, 14, 15, 18, 19, 20, 21, 35, 36, 39, 40, 41, 46, 47, 48, 62]   # pasture, crops, forest plantation, mosaic of uses
+"4" = [22, 23, 24, 25, 30, 75, 91]                            # beach and dune, urban, mining, photovoltaic, wind farm, other
+"5" = [26, 31, 33]                                            # river, lake and ocean; aquaculture
+```
+
+A code that is in no group is ignored, as is nodata (0): a class missing from the table is a class that disappears from
+the map, so check the table against the legend of the collection in use. The share of a class, instead of the dominant
+one, is `operator = "percentage"` with `class_code = [3, 4, 5, 6, 49]` (a list is the share in any of them).
+
 ## Classes
 
 `disscube.sources.mapbiomas.CLASSES` names the classes most common in

docs/guides/pipeline_files.md

@@ -104,6 +104,32 @@ relative to the pipeline file, so a folder with its TOML and data is portable.
 
 Every source block also accepts `name` and `years`.
 
+### A vector source: distance to points
+
+A point, line or polygon layer is declared as a `file` source, like a raster, and the operator is the one that
+summarises it in a cell. The distance from each cell to the nearest municipal seat, measured in a projected CRS
+(`EPSG:5880`, in metres) although the grid is in BDC Albers:
+
+```toml
+[[source]]
+id = "seats"
+type = "file"
+path = "zip://ana/GEOFT_CIDADE_2016.zip"      # a shapefile inside a zip
+url = "https://…/GEOFT_CIDADE_2016.zip"       # where it is downloaded from, if not there
+sha256 = "0311f4267c4e…"                        # checked after the download
+
+[[derive]]
+target = "dist_seats"
+source = "seats"
+operator = "distance"
+params = { crs = "EPSG:5880" }
+```
+
+The distance is computed for every cell of the grid, including those outside the area of interest (the maps may mask
+them), and the layer is not clipped to the grid: a point outside it can be the nearest one. A vector source is
+not read by blocks (see [Block-wise reading](blocks.md)); it is small. The complete file, for the state of Pará, is
+`cases/para/seats.toml` in [disscube-grids](https://github.com/LambdaGeo/disscube-grids).
+
 ### Taking files as they come
 
 A `file` source can say how to read the file, so that the data need no
@@ -147,6 +173,7 @@ of = ["state_paved", "federal_paved"]
 |---|---|---|
 | `distance` | `crs` | measure in this CRS — e.g. a projected one, for metres on a geographic grid |
 | `percentage`, `majority`, `minority`, `std`, `median` | `subcells` | at most this many fine pixels per cell along each axis: bounds the memory of a fine source over a large grid (a 100 m raster on a 1/12° grid would give ~90 × 90) |
+| `majority`, `minority` | `groups` | `{label = [codes]}`: read a map of leaf classes at a coarser level; the result is the label of the group (codes in no group are ignored). See [MapBiomas](mapbiomas.md#level-1-classes) |
 | `sum` | `area` | vector polygons: share the value in proportion to the intersected area (default `false`) |
 | `sum` | `column` | vector: numeric column to sum (default: the target name) |
 | `std` | `ddof` | `0` (default, population) or `1` (sample standard deviation) |
@@ -175,6 +202,12 @@ other loops or conditions: what does not fit belongs in Python.
 └── run.json            this run: DisSCube version, pipeline file and checksum, sources, products
 ```
 
+`run.json` (a copy is kept in `runs/<pipeline file name>.json`) holds, in this order: the DisSCube version that ran
+(`disscube`); the `pipeline` file, with its SHA-256 and name; the `started` and `finished` times (UTC); the `grid`;
+the `sources`, each with its `id`, `type`, the local `file`, the SHA-256 `checksum` of that file and its `time`; the
+`derived` products, each with its `target`, `source`, `spec_hash`, `times` and the Zarr `file`; and, if the run exported,
+the `exported` path. With the pipeline file, the checksums and the version, the same grid can be rebuilt and checked.
+
 Each provenance file gains a `pipeline` entry with the pipeline file's path,
 name and SHA-256, so every derived variable can be traced back to the exact
 file that declared it.

mkdocs.yml

@@ -40,6 +40,8 @@ nav:
       - TerraME Fill Cells Correspondence: terrame_fill_correspondence.md
   - Guides:
       - Pipeline files (TOML): guides/pipeline_files.md
+      - Command line: guides/cli.md
+      - Block-wise reading: guides/blocks.md
       - BDC Integration: guides/bdc.md
       - MapBiomas: guides/mapbiomas.md
       - "Land-cover maps: PRODES and SITS": guides/landcover.md