diff --git a/CHANGELOG.md b/CHANGELOG.md
index b197282..a79dd8b 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,6 +4,52 @@ Format based on [Keep a Changelog](https://keepachangelog.com/pl/1.1.0/);
semantic versioning. The version of the data schema is separate from the version of the package
and is reported as `SCHEMA_VERSION`.
+## 1.6.0 (2026-10-24)
+
+### Added
+- **Charts** (`gridqueue.plots`, optional extra `pip install "gridqueue[plot]"`): field coverage,
+ quality rules, column-shift audit, stage of the proceeding, requested capacity by class and
+ voltage, Kaplan-Meier processing-time curves with censoring and a 95 % band, node loading,
+ flows / transitions / linkage audit of two editions, location resolution, a four-panel
+ overview, and `save_all`, which draws every chart the data supports and lists the skipped ones
+ with the reason. A chart is a view of a result, never a second computation of it. Figures are
+ created without `pyplot`, so the module runs unchanged in scripts, notebooks and the browser.
+- **CLI**: `gridqueue compare` (flows, transitions, linkage precision and sensitivity of two
+ editions) and `gridqueue plot` (every chart, PNG/SVG/PDF).
+- `parse_sources`: any number of files of one or several publishers into one panel; every file is
+ parsed by a fresh adapter instance, so each document keeps the as-at date it states.
+ `split_editions` separates two or more editions of one register dropped together.
+- `kaplan_meier_by_group`: Kaplan-Meier summary per group (or per combination of columns) with the
+ inclusion criterion of the article.
+- `geocode_frame`: writes the resolver's result back to the panel (`wspolrzedne` as `"lat,lon"`,
+ `_geo_pewnosc`, `_geo_dokladnosc`, `_geo_powod`); source coordinates are kept.
+- `node_loading(..., napiecie=...)`: voltage filter of the queue. `napiecie="110"` is the base
+ variant of the article (median 6.500 / 4.688, 29 of 36 / 33 of 37 for ENERGA-OPERATOR,
+ 31 August 2026); the default is the all-voltage upper variant (8.611 / 7.638).
+- `gridqueue.examples`: a synthetic panel with node rows, two synthetic editions and a small
+ gazetteer, labelled as synthetic, for trying every function without a real disclosure.
+- **Browser page, simple view** (the default): drop the documents and get the main charts, a few
+ sentences with the key numbers, and one ZIP with charts, data and report. Two editions of one
+ register dropped together are recognised by their as-at dates and compared.
+- **Browser page, full view**: several files at once, manual choice of the adapter, a later edition
+ (flows, transitions, linkage audit), an optional gazetteer, schema validation and the column-shift
+ audit, processing times and node loading, all charts (PNG 300 dpi and SVG), and export to CSV,
+ XLSX, JSON and one ZIP archive; two synthetic demonstration modes; a table of what runs where.
+- **Colab notebook**: installs 1.6.0 with charts; adds schema validation, the column-shift audit,
+ key sensitivity (checked against the article), ENERGA-OPERATOR processing times and node loading
+ (both checked against the Supplementary Material), the resolver on a synthetic gazetteer, all
+ charts and a downloadable results archive.
+
+### Fixed
+- **Censoring date of `processing_time`.** The default censored every open application at the
+ single latest publication date of the frame, node rows included. ENERGA-OPERATOR publishes its
+ applications as at 30 June and its node capacities as at 31 August 2026, so every censored time
+ was two months too long (median for consumption at 110 kV 256 d instead of 240 d). Each row is
+ now censored at its own publication date, which is also correct for panels of several publishers.
+- **Interval of the Kaplan-Meier median**: the lower and upper bounds were swapped
+ (`mediana_dolna > mediana > mediana_gorna`).
+- `kaplan_meier` raised `KeyError` for a group in which every observation is censored.
+
## 1.5.2 (2026-10-17)
### Added
diff --git a/CITATION.cff b/CITATION.cff
index 6f277ec..004d51f 100644
--- a/CITATION.cff
+++ b/CITATION.cff
@@ -7,8 +7,8 @@ abstract: >-
and the corresponding registers from other jurisdictions into a single set of rows in a 21-field
schema derived from the structure of the legal obligation.
type: software
-version: "1.2.1"
-date-released: "2026-10-04"
+version: "1.6.0"
+date-released: "2026-10-24"
license: Apache-2.0
repository-code: "https://github.com/s-matysik/gridqueue"
url: "https://s-matysik.github.io/gridqueue/"
diff --git a/CODE_METADATA.md b/CODE_METADATA.md
index 1667694..f315ff6 100644
--- a/CODE_METADATA.md
+++ b/CODE_METADATA.md
@@ -8,13 +8,13 @@ listed again in the final section.
| Nr | Item | Value |
|----|------|-------|
-| C1 | Current code version | `1.2.1` (declared identically in `pyproject.toml` and in `gridqueue.__version__`; CI asserts the two agree on every push) |
-| C2 | Permanent link to code / repository used for this code version | https://github.com/s-matysik/gridqueue (tag `v1.2.1`) |
-| C3 | Permanent link to Reproducible Capsule | ⟦TEAM DECISION⟧, archive DOI. The repository is ready for a Zenodo deposit of tag `v1.2.1`; enabling the integration requires the team's account |
+| C1 | Current code version | `1.6.0` (declared identically in `pyproject.toml` and in `gridqueue.__version__`; CI asserts the two agree on every push) |
+| C2 | Permanent link to code / repository used for this code version | https://github.com/s-matysik/gridqueue (tag `v1.6.0`) |
+| C3 | Permanent link to Reproducible Capsule | ⟦TEAM DECISION⟧, archive DOI. The repository is ready for a Zenodo deposit of tag `v1.6.0`; enabling the integration requires the team's account |
| C4 | Legal Code License | Apache-2.0, `LICENSE.txt` at the distribution root. Input data licences are fixed and separate: OpenStreetMap gazetteer under ODbL 1.0; operators' statutory publications under the terms each publisher applies to its own site |
| C5 | Code versioning system used | git |
| C6 | Software code languages, tools and services used | Python, `requires-python >= 3.10`; build backend `setuptools.build_meta` (`setuptools>=68`); test runner `pytest` |
-| C7 | Compilation requirements, operating environments and dependencies | No compilation (pure Python). Runtime: `pandas>=2.0`, `pdfplumber>=0.11`, `openpyxl>=3.1`, `requests>=2.31`, `pyarrow>=14`. Test extra: `pytest>=8.0`. Build: `setuptools>=68`. The location resolver needs no third-party string-matching library; matching is implemented in `geoloc.py` |
+| C7 | Compilation requirements, operating environments and dependencies | No compilation (pure Python). Runtime: `pandas>=2.0`, `pdfplumber>=0.11`, `openpyxl>=3.1`, `requests>=2.31`, `pyarrow>=14`. Optional extra `plot`: `matplotlib>=3.7` (charts). Test extra: `pytest>=8.0`, `matplotlib>=3.7`. Build: `setuptools>=68`. The location resolver needs no third-party string-matching library; matching is implemented in `geoloc.py` |
| C8 | If available, link to developer documentation / manual | https://s-matysik.github.io/gridqueue/, eight generated pages (schema with legal basis per field, adapters, extraction method, quality rules, measured validation, CLI, limitations). `README.md` at the distribution root duplicates the quick path |
| C9 | Support email for questions | ⟦TEAM DECISION⟧, corresponding author's institutional address |
@@ -22,8 +22,8 @@ listed again in the final section.
| Nr | Item | Value |
|----|------|-------|
-| S1 | Current software version | `1.2.1`, matching C1 |
-| S2 | Permanent link to executables of this version | Source distribution only, `gridqueue-1.2.1.tar.gz`, pure Python, no binaries, no compiled extensions |
+| S1 | Current software version | `1.6.0`, matching C1 |
+| S2 | Permanent link to executables of this version | Source distribution only, `gridqueue-1.6.0.tar.gz`, pure Python, no binaries, no compiled extensions |
| S3 | Legal Software License | Apache-2.0, same licence as C4 |
| S4 | Computing platforms / Operating Systems | Linux, macOS, Windows × Python 3.10, 3.11, 3.12, 3.13, a twelve-cell matrix exercised by GitHub Actions (`.github/workflows/ci.yml`) on every push and pull request. The measured figures reported in the paper were produced on macOS/arm64 with Python 3.11.15 |
| S5 | Installation requirements & dependencies | As C7. No GPU, no network access required at run time once the source documents and the gazetteer are local |
@@ -113,7 +113,7 @@ blocker is server-side client filtering, the site returns a rejection page to th
control the operator put in place deliberately, so it was not attempted. Electricity North West:
rows behind the same access control as UK Power Networks; metadata only.
-**Size.** Read from the built source distribution `gridqueue-1.2.1.tar.gz`: 7 core modules under `src/gridqueue`, 2,125 lines, `geoloc.py` 672, `schema.py` 480, `quality.py` 370, `layout.py` 353, `cli.py` 148, `registry.py` 64, `__init__.py` 38, and 9 adapter modules, 1,777 lines (`pl_pse.py` 303, `pl_energa.py` 282, `_ptpiree.py` 278, `pl_boryszew.py` 232, `pl_stoen.py` 194, `uk_nationalgrid.py` 166, `pl_tauron.py` 161, `base.py` 156, `__init__.py` 5); 3,902 lines in total under `src`. `_ptpiree.py` is shared by the PSE and ENERGA adapters because both publishers fill the same agreed industry template, one as a spreadsheet and one as a print document. Tests: 7 files, 1,040 lines (`test_geoloc_wspolrzedne.py` 280, `test_schema.py` 164, `test_pse_energa.py` 147, `test_adapters.py` 144, `test_quality.py` 124, `test_geoloc.py` 95, `test_layout.py` 86). Runnable workflows shipped alongside the package: `examples/research_use_case.py` 168 lines, and `reproducibility/` 382 lines (`run_figures.py` 248, `run_validation.py` 134). `python -m pytest -q`: **153 passed, 2 skipped (155 collected), 0 failed, 0 errors**, the two skips are the PSE and ENERGA integration tests, which require the source documents and are skipped when those are absent from the working tree. The twelve-cell OS × Python matrix in CI runs the same suite.
+**Size.** Read from the built source distribution `gridqueue-1.6.0.tar.gz`: 7 core modules under `src/gridqueue`, 2,125 lines, `geoloc.py` 672, `schema.py` 480, `quality.py` 370, `layout.py` 353, `cli.py` 148, `registry.py` 64, `__init__.py` 38, and 9 adapter modules, 1,777 lines (`pl_pse.py` 303, `pl_energa.py` 282, `_ptpiree.py` 278, `pl_boryszew.py` 232, `pl_stoen.py` 194, `uk_nationalgrid.py` 166, `pl_tauron.py` 161, `base.py` 156, `__init__.py` 5); 3,902 lines in total under `src`. `_ptpiree.py` is shared by the PSE and ENERGA adapters because both publishers fill the same agreed industry template, one as a spreadsheet and one as a print document. Tests: 7 files, 1,040 lines (`test_geoloc_wspolrzedne.py` 280, `test_schema.py` 164, `test_pse_energa.py` 147, `test_adapters.py` 144, `test_quality.py` 124, `test_geoloc.py` 95, `test_layout.py` 86). Runnable workflows shipped alongside the package: `examples/research_use_case.py` 168 lines, and `reproducibility/` 382 lines (`run_figures.py` 248, `run_validation.py` 134). `python -m pytest -q`: **153 passed, 2 skipped (155 collected), 0 failed, 0 errors**, the two skips are the PSE and ENERGA integration tests, which require the source documents and are skipped when those are absent from the working tree. The twelve-cell OS × Python matrix in CI runs the same suite.
**Functionalities.** Document ingest with per-row provenance; layout-based extraction; unit
normalisation to the canonical kW (MW→kW, with the source unit retained in an extension column); resource-class and process-status vocabulary mapping; composite-date
diff --git a/README.md b/README.md
index 557e0ce..430ae7a 100644
--- a/README.md
+++ b/README.md
@@ -2,8 +2,8 @@
[](https://github.com/s-matysik/gridqueue/actions/workflows/ci.yml)
[](https://github.com/s-matysik/gridqueue)
-[](LICENSE.txt)
-[](https://s-matysik.github.io/gridqueue/)
+[](LICENSE.txt)
+[](https://s-matysik.github.io/gridqueue/)
[](https://github.com/s-matysik/gridqueue/releases)
[](https://colab.research.google.com/github/s-matysik/gridqueue/blob/main/notebooks/gridqueue_colab.ipynb)
@@ -17,10 +17,33 @@ public, but it is not processable.
`gridqueue` turns those publications into one set of rows with **measured accuracy**.
-State of release 0.3.1: **six adapters, 19 220 panel rows, all 21 schema fields
-populated**. Publishers: TAURON Dystrybucja 10 955 rows, ENERGA-OPERATOR 5 382,
-Stoen Operator 1 236, PSE 945, National Grid (GB) 694, Boryszew Green Energy&Gas 8.
-Documentation: **https://s-matysik.github.io/gridqueue/**
+Release 1.6.0: **nine adapters in five jurisdictions** (Poland, Spain, Ireland, Portugal, Great
+Britain), all 21 schema fields populated by at least one publisher, quality rules, a longitudinal
+module with a linkage audit, processing times with censoring, node loading, a location resolver and
+**charts for every result**. Documentation: **https://s-matysik.github.io/gridqueue/**
+
+## Three ways to run it
+
+| | Python library | Google Colab | In the browser |
+|---|---|---|---|
+| Install | `pip install` (below) | nothing | nothing |
+| Where | your machine | Google's free machine | your machine, offline after loading |
+| Parse, validate, quality rules, column-shift audit | yes | yes | yes |
+| Several files and publishers in one panel | yes | yes | yes |
+| Two editions: flows, transitions, linkage precision and sensitivity | yes | yes | yes |
+| Processing time (Kaplan-Meier), node loading | yes | yes | yes |
+| Coordinates from a gazetteer | yes | yes | yes, with your gazetteer file |
+| Charts (PNG, SVG) | yes | yes | yes |
+| Export | Parquet, CSV, XLSX, JSON | Parquet, CSV, XLSX, JSON | CSV, XLSX, JSON, ZIP |
+| Fetch documents from the publisher's server | yes | yes | no, the browser forbids it |
+
+* **Colab:** [](https://colab.research.google.com/github/s-matysik/gridqueue/blob/main/notebooks/gridqueue_colab.ipynb)
+* **Browser:** **https://s-matysik.github.io/gridqueue/app/**, drop the published documents on the
+ page. The *simple* view needs nothing else: it recognises the publisher, compares two editions
+ of one register when it gets them, and returns the main charts with one download. The *full*
+ view adds every input, option and table. The package runs in a WebAssembly interpreter;
+ nothing is uploaded.
+* **Library:** see *Installation*.
## What it is not
@@ -35,28 +58,33 @@ this package amounts to downloading a CSV. Poland is the case in which it is not
[](https://colab.research.google.com/github/s-matysik/gridqueue/blob/main/notebooks/gridqueue_colab.ipynb)
The notebook `notebooks/gridqueue_colab.ipynb` performs **a full research run in the browser**:
-it downloads two editions of the disclosure directly from the publisher, extracts the rows, assesses
-the quality, computes the queue flows together with the linkage precision and **compares the
-reproduced figures with those published in the paper, aborting execution on a divergence**. Under a
-minute on a free machine, with no accelerator and no configuration.
+it downloads two editions of the disclosure directly from the publisher, extracts and validates the
+rows, assesses the quality, computes the queue flows together with the linkage precision and key
+sensitivity and **compares the reproduced figures with those published in the paper, aborting
+execution on a divergence**. It then parses a distribution operator's three documents for the
+processing-time curves and node loading, runs the location resolver, draws every chart and exports
+the results as one archive. The PSE part takes about a minute on a free machine; the ENERGA part adds
+several minutes of PDF parsing and can be switched off.
The notebook does not reproduce the extraction accuracy against the reference set, because that set
-requires a human reading, nor does it resolve coordinates, because the gazetteer is a separate
-download under the ODbL licence.
+requires a human reading. The national gazetteer is a separate download under the ODbL licence, so the
+resolver is shown on a synthetic gazetteer, with a one-line change to use your own.
## Installation
One command, with no compilation and no external services:
```bash
-pip install git+https://github.com/s-matysik/gridqueue@v1.2.1
+pip install "gridqueue[plot] @ git+https://github.com/s-matysik/gridqueue@v1.6.0"
```
+The `plot` extra adds matplotlib for the charts; without it everything else works.
+
For work on the code, with the test extra:
```bash
git clone https://github.com/s-matysik/gridqueue && cd gridqueue
-pip install -e ".[test]"
+pip install -e ".[test,plot]"
```
Dependencies: `pandas`, `pdfplumber`, `openpyxl`, `requests`, `pyarrow`. The canonical unit of
@@ -94,8 +122,43 @@ gridqueue parse tauron.pdf --out tauron.parquet --report tauron.json
gridqueue parse --adapter uk_nationalgrid --out uk.parquet # downloads from the CKAN catalogue
gridqueue validate tauron.parquet --out walidacja.json
gridqueue export tauron.parquet stoen.parquet --out panel.parquet --only-schema
+gridqueue compare pse_07.parquet pse_08.parquet --label-a 2026-07-31 --label-b 2026-08-31 --out delta.json
+gridqueue plot energa.parquet --out charts/ --format png,svg # every chart the data supports
+gridqueue plot pse_07.parquet --edition-b pse_08.parquet --out charts/pse/
```
+## Charts
+
+`gridqueue.plots` draws every result of the package. Each function takes the object the analysis
+returns and gives back a matplotlib `Figure`, so a chart never recomputes a number: what it shows is
+what the analysis reports. What a disclosure does not contain is drawn as absence (a grey cell, a
+skipped node with the reason), never as zero.
+
+| Function | Shows |
+|---|---|
+| `plot_field_coverage` | share of rows populating each of the 21 fields, per publisher |
+| `plot_quality` | violations of the eight quality rules, stacked by publisher |
+| `plot_column_shift` | rows in which a value stands in a field it cannot belong to |
+| `plot_status_distribution` | applications by stage of the proceeding |
+| `plot_capacity_by_class` | requested capacity of the active queue by resource class and voltage |
+| `plot_processing_time` | Kaplan-Meier curves with censoring and a 95 % band |
+| `plot_node_loading` | queue over declared free capacity, per node, log scale |
+| `plot_edition_flows`, `plot_transitions`, `plot_linkage_audit`, `plot_edition_comparison` | two editions |
+| `plot_location_resolution` | where a coordinate comes from, and the reason where it does not |
+| `plot_overview` | four panels for a first look |
+| `save_all` | every chart the data supports, PNG/SVG/PDF; skipped ones listed with the reason |
+
+```python
+from gridqueue import parse_sources, plots
+
+panel, reports = parse_sources(["wnioski.pdf", "moc_odbiorcza.pdf", "moc_wytworcza.pdf"])
+plots.plot_processing_time(panel, by="klasa_zasobu") # displays in Jupyter / Colab
+plots.save_all(panel, "charts/", formats=("png", "svg"), dpi=300)
+```
+
+`gridqueue.examples` generates a synthetic panel, two synthetic editions and a small gazetteer, labelled
+as synthetic, for trying every function without a real disclosure.
+
## Schema
21 fields in three groups reflecting the structure of the duty (IDENTITY / CAPACITY / PROCESS),
@@ -203,7 +266,7 @@ disclosure; violations of that rule numbered 5 390 (28,044 %) against **771 (4,0
## Tests
```bash
-python -m pytest -q # 153 passed, 2 skipped (155 collected)
+python -m pytest -q # 221 passed, 2 skipped (223 collected)
```
The two skips are conditional: they are the integration tests of the PSE and ENERGA adapters, which
@@ -246,11 +309,8 @@ with a list of individual authors before the paper is submitted.
Apache-2.0 for **the code**. The input and output data have a separate status; official documents are not redistributed, and the geometric gazetteer is subject to ODbL with attribution and share-alike duties. Details in [`DATA_LICENSE.md`](DATA_LICENSE.md).
-Change history: [`CHANGELOG.md`](CHANGELOG.md)., see `LICENSE.txt`. This is a **working** choice, dictated by the fact that the target journal
-requires a licence file in the repository, and the text of Apache-2.0 does not require the rights
-holder's details to be entered in the file itself. A change to another licence from the permitted list
-(MIT, BSD, GPL) is one commit, as long as the repository has no external co-authors.
-
The licences of the input data are separate and fixed: the OpenStreetMap gazetteer under the ODbL 1.0
licence; the operators' statutory publications under their own terms of use, redistributed only
where that is permitted.
+
+Change history: [`CHANGELOG.md`](CHANGELOG.md).
diff --git a/docs/app/gridqueue-1.5.2-py3-none-any.whl b/docs/app/gridqueue-1.5.2-py3-none-any.whl
deleted file mode 100644
index 319eea9..0000000
Binary files a/docs/app/gridqueue-1.5.2-py3-none-any.whl and /dev/null differ
diff --git a/docs/app/gridqueue-1.6.0-py3-none-any.whl b/docs/app/gridqueue-1.6.0-py3-none-any.whl
new file mode 100644
index 0000000..c678d41
Binary files /dev/null and b/docs/app/gridqueue-1.6.0-py3-none-any.whl differ
diff --git a/docs/app/index.html b/docs/app/index.html
index 8ce2d79..fa882cb 100644
--- a/docs/app/index.html
+++ b/docs/app/index.html
@@ -4,7 +4,7 @@
gridqueue, run it in your browser
-
+
-
+
@@ -223,12 +270,13 @@
Runs locally, nothing is uploaded
Open source, Apache 2.0
Nine publishers, five jurisdictions
+ Analyses and charts included
Grid connection registers, as data you can use
Network operators across the European Union must publish the connection capacity
available at each node, at least quarterly. Most publish it as documents built for printing.
- Drop one below and get harmonised rows back, with no installation and no upload.
-
+ Drop them below and get harmonised rows, quality checks, analyses and charts back,
+ with no installation and no upload.
@@ -265,8 +313,8 @@ Grid connection registers, as data you can use
Rows you can analyse
- plus eight quality rules
- and a validation report
+ quality rules, analyses
+ and charts
EVERY STEP RUNS INSIDE THIS BROWSER TAB
@@ -276,7 +324,6 @@ Grid connection registers, as data you can use The schema is the invariant part. Adapters are the part that differs per publisher,
which is why adding a tenth publisher does not change anything downstream of it.
-
@@ -285,22 +332,22 @@ Grid connection registers, as data you can use
- Your document never leaves this computer. The page loads the real
+
Your documents never leave this computer. The page loads the real
gridqueue package into a Python interpreter compiled to WebAssembly and
- runs it locally. There is no server to upload anything to. It is the same package, at the same
- version, that produced the results in the accompanying article, so what you see here matches the
- command line output by construction.
+ runs it locally. There is no server to upload anything to. The charts are drawn by the same
+ gridqueue.plots functions as on the command line and in the notebook,
+ from the same numbers, so they match by construction.
Starting
Preparing the interpreter
-
First run downloads the interpreter and the libraries, about 30 MB.
+
First run downloads the interpreter and the libraries, about 45 MB.
Later visits start from the browser cache.
WebAssembly interpreter
- Data and imaging libraries
+ Data, imaging and charting libraries
PDF and spreadsheet readers
gridqueue package
@@ -309,42 +356,104 @@
Grid connection registers, as data you can use
- Give it a document
-
-
-
+ Give it documents
+
+
+ Simple, with charts
+ Full, every option
+
+
Drop the published documents. The page recognises the publisher and, for two
+ editions of one register, compares them. You get the main charts and one download.
+
Every input and option: later edition, gazetteer, adapter, group size, and every table.
+
+
+
+
+
+
Preparing, one moment
+
PDF or XLSX as published: one disclosure, all documents of one edition, or two editions
+
+
+
+
+ Try it on synthetic data
+ Try two synthetic editions
-
Drop a disclosure file here
-
PDF, XLSX or CSV, as published by one of the nine supported publishers
-
- or
+
+
+
Disclosure documents
+
One or more files: several documents of one edition (ENERGA, PSE and Stoen
+ split theirs), or editions of several publishers, which are stacked into one panel.
+
+
+
Preparing, one moment
+
PDF, XLSX or CSV, as published, several at once
+
+
+
+
+
+
+
Later edition (optional)
+
The same register at a later date. Adds flows, status transitions and the
+ linkage audit between the two editions.
+
+
+
Drop the later edition
+
+
+
+
+
+
Gazetteer (optional)
+
A JSON gazetteer with coordinates, in the layout read by
+ resolver_from_gazeter_json . Adds coordinates and the reason
+ wherever none can be given.
+
+
+
Drop a gazetteer JSON
+
+
+
+
+
+
-
-
-
-
- Run a synthetic example
-
-
No real disclosure needed. Shows the schema and the eight quality rules.
+
+
+
Adapter detect from the file
+
Label of the edition
+
Label of the later edition
+
Minimum group size, curves
+
+
+
+
+ Analyse the documents
+
Synthetic panel
+
Synthetic two editions
+
The synthetic data are generated by gridqueue.examples
+ and describe no real grid.
+
-
-
+
No sample document ships with this page. The disclosures are produced by commercial
undertakings discharging a statutory duty, their status for redistribution is unsettled, and
publishers replace them at least quarterly, so a bundled copy would misrepresent the register
within months. Fetch a current edition from the publisher instead.
-
Detection uses the file name first and, for PDFs, the text
- of the first page. Keep the name the publisher gave the file.
+ of the first page. Keep the name the publisher gave the file, or choose the adapter above. A CSV
+ written by gridqueue itself is read with the adapter
harmonised table .
Result
-
+
+
-
+
-
-
- Harmonised table, CSV
-
- Validation report, JSON
+
+ Everything, ZIP (charts 300 dpi PNG + SVG, data, report)
+
Harmonised table, CSV
+
See every table and option
+
Workbook, XLSX
+
Report, JSON
+
+
+
+
+ Analyses
+
+
+
Time from application to connection conditions, per resource class
+ (Kaplan-Meier, open applications censored at the as-at date, 95 % Greenwood interval of the median).
+
+
+
+
Node loading: requested capacity of the active queue over the declared free
+ capacity. Undefined where the publisher declares zero free capacity.
+
+
+
+
+
+
Harmonised rows
@@ -383,27 +524,52 @@
Harmonised rows
-
- Quality rules
+
+ Quality, validation and column shift
Every rule returns a count, never an exception. A violation is a property of the
disclosure, not a fault in the software, which is why the count is reported rather than raised.
+
-
+
Parse report
- Returned verbatim by the adapter, including whatever it could not resolve.
+ Returned verbatim by each adapter call, including whatever it could not resolve.
- What this page will not do
- It handles one document at a time. It does not geocode, because the gazetteer is too large to
- send to a browser, does not compare editions over time, and does not export Parquet. For those,
- install the package with pip install gridqueue , or open the
- Colab notebook , which reproduces the longitudinal analysis from the
- article and stops if the figures disagree with the published ones.
+ What runs where
+ The page, the notebook and the command line are three ways into one package. What the page
+ cannot do follows from the browser, not from a reduced copy of the code.
+
+ For everything at once, open the
+ Colab notebook ,
+ or install the package with pip install "gridqueue[plot] @ git+https://github.com/s-matysik/gridqueue@v1.6.0" .
-
+
diff --git a/docs/cli.md b/docs/cli.md
index a48181d..bcde9e7 100644
--- a/docs/cli.md
+++ b/docs/cli.md
@@ -11,6 +11,10 @@ gridqueue parse tauron.pdf --out tauron.parquet --report tauron.json
gridqueue parse --adapter uk_nationalgrid --out uk.parquet # downloads from the CKAN catalogue
gridqueue validate tauron.parquet --out walidacja.json
gridqueue export tauron.parquet stoen.parquet --out panel.parquet --only-schema
+gridqueue compare pse_07.parquet pse_08.parquet --label-a 2026-07-31 --label-b 2026-08-31 \
+ --out delta.json --transitions transitions.csv
+gridqueue plot energa.parquet --out charts/ --format png,svg
+gridqueue plot pse_07.parquet --edition-b pse_08.parquet --out charts/pse/
```
`parse` recognizes the publisher from the file name and, where that is not enough, from the content
@@ -24,6 +28,14 @@ violations along with the rows that cause them; a violation does not abort the r
`export` merges the results of several adapters into one panel. `--only-schema` omits the extension
columns carrying the `_` prefix.
+`compare` sets two editions of one register side by side: the flows of rows, the status transition
+matrix, the linkage precision on the fields held out of the key and the key sensitivity.
+`--node-key` uses the node key for node-level registers.
+
+`plot` draws every chart the dataset supports (see [Charts](wykresy.md)); with `--edition-b` it adds
+the comparison of two editions. Charts that need data the disclosure does not carry are skipped and
+listed with the reason. It needs the `plot` extra (matplotlib).
+
## Provenance of the result
Every row carries a provenance record: the source document, the page, the adapter version and the
diff --git a/docs/colab.md b/docs/colab.md
index c035cc7..c416941 100644
--- a/docs/colab.md
+++ b/docs/colab.md
@@ -14,28 +14,33 @@ without an accelerator.
1. installs the package from a **tagged release**, not from the main branch, so the result does not
depend on the current state of the repository;
-2. downloads two consecutive editions of the disclosure directly from the publisher and prints the
- checksums;
-3. recognizes the publisher **from the content of the document** and extracts the rows of the common
- schema;
-4. evaluates the eight quality control rules;
+2. downloads two consecutive editions of the transmission operator's disclosure directly from the
+ publisher and prints the checksums;
+3. recognizes the publisher **from the content of the document**, extracts the rows of the common
+ schema and validates them;
+4. evaluates the eight quality control rules and the column-shift audit;
5. computes the queue flows between the editions together with the **linkage precision** on the
- fields held out of the key;
+ fields held out of the key and the **key sensitivity**;
6. compares the reproduced figures with those published in the paper and **aborts execution on a
- divergence**.
+ divergence**;
+7. downloads ENERGA-OPERATOR's three documents of one edition and reproduces the processing-time
+ strata (Kaplan-Meier with censoring) and the node loading of the paper, with the same check;
+8. runs the location resolver on a synthetic gazetteer, draws every chart and packs the results into
+ one archive.
-Point six is the counterpart of the assertions in the `reproducibility/` directory. A divergence need
-not mean an error, the publisher may have replaced the edition, but it has to be visible rather than
-passed over in silence.
+Points six and seven are the counterpart of the assertions in the `reproducibility/` directory. A
+divergence need not mean an error, the publisher may have replaced the edition, but it has to be
+visible rather than passed over in silence. Point seven parses PDF documents and adds several
+minutes; it can be switched off with `RUN_ENERGA = False`.
## What the notebook does not do
It does not reproduce the extraction accuracy against the reference set, because that set requires a
-human reading. It does not resolve coordinates, because the geometric gazetteer is a separate
-download and is subject to ODbL, and the notebook should not quietly draw the user into licensing
-duties. It also does not run the cross-sectional example on this publisher, because **it does not
-meet the date coverage criterion** set before the result was computed; the notebook shows this
-openly instead of computing a median on a dozen or so percent of the records.
+human reading. It does not use the national geometric gazetteer, which is a separate download subject
+to ODbL; the resolver is shown on a synthetic gazetteer, and one line points it at your own. It does
+not compute processing times for the transmission operator, because that publisher **does not meet
+the date coverage criterion** set before the result was computed; the notebook shows this openly
+instead of computing a median on a dozen or so percent of the records.
## When a download fails
diff --git a/docs/index.md b/docs/index.md
index 8bd8174..32e16ae 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -5,7 +5,8 @@ title: gridqueue
# gridqueue
Harmonizer of statutory grid-connection disclosures for the electricity network.
-Version **1.5.2** · [repository](https://github.com/s-matysik/gridqueue)
+Version **1.6.0** · [repository](https://github.com/s-matysik/gridqueue) ·
+[run in the browser](app/) · [open in Colab](https://colab.research.google.com/github/s-matysik/gridqueue/blob/main/notebooks/gridqueue_colab.ipynb)
> **In English.** `gridqueue` parses the statutory grid-connection disclosures that Polish
> distribution system operators must publish under Article 7(8l) of the Energy Law, and emits
@@ -58,6 +59,7 @@ confidence level and a placement accuracy level.
* [Quality control rules](jakosc.md)
* [Measured validation](walidacja.md)
* [Longitudinal analysis, comparison of editions](wzdluzna.md)
+* [Charts](wykresy.md): every result as a figure, in the library, the command line, Colab and the browser
* [Running in Google Colab](colab.md)
* [Command line](cli.md)
* [Limitations](ograniczenia.md)
diff --git a/docs/wykresy.md b/docs/wykresy.md
new file mode 100644
index 0000000..1a91704
--- /dev/null
+++ b/docs/wykresy.md
@@ -0,0 +1,74 @@
+---
+title: Charts
+---
+
+# Charts
+
+`gridqueue.plots` draws every result of the package. It needs matplotlib, installed with the
+`plot` extra:
+
+```bash
+pip install "gridqueue[plot] @ git+https://github.com/s-matysik/gridqueue@v1.6.0"
+```
+
+## Principles
+
+* **A chart is a view of a result, never a second computation of it.** Every function takes the
+ object an analysis returns (a frame in the schema, an `EditionDelta`, a `LinkageAudit`, the frame
+ of `node_loading`) and calls the same functions the report uses, so the numbers in a title or a
+ legend are the numbers of the report.
+* **Absence is drawn as absence.** A field the publisher does not disclose is a grey `-` cell, not
+ a zero; a node with zero declared free capacity has an undefined loading and is counted in the
+ title, not drawn as infinite; a coordinate that cannot be given is shown with its reason.
+* **Inclusion criteria are fixed in advance.** Processing-time curves are drawn only for groups of
+ at least `min_n` applications, and `save_all` skips them, with the reason, when fewer than 80 % of
+ the applications carry both dates.
+* Figures are created without `pyplot`: the same code runs in a script, in Jupyter or Colab (a
+ returned figure displays itself) and in the browser page.
+
+## Functions
+
+| Function | Input | Shows |
+|---|---|---|
+| `plot_field_coverage(df)` | panel | share of rows populating each of the 21 fields, per publisher |
+| `plot_quality(df)` | panel | violations of the eight rules, evaluated and stacked per publisher |
+| `plot_column_shift(df)` | panel | rows with a value in a field it cannot belong to |
+| `plot_status_distribution(df, normalize=True)` | panel | applications by stage of the proceeding |
+| `plot_capacity_by_class(df)` | panel | requested capacity of the active queue by class and voltage |
+| `plot_processing_time(df, by="klasa_zasobu")` | panel | Kaplan-Meier curves, censored at the as-at date, 95 % band |
+| `plot_node_loading(L)` | `node_loading(...)` | queue over declared free capacity per node, log scale |
+| `plot_edition_flows(delta)` | `compare_editions(...)` | shared, new and absent rows, with the warning segments |
+| `plot_transitions(delta)` | `compare_editions(...)` | status transition matrix; backward moves outlined |
+| `plot_linkage_audit(audit, recall)` | `linkage_audit`, `linkage_recall` | confirmed, uncertain, false and missed pairs |
+| `plot_edition_comparison(delta, audit, recall)` | all three | the three views above in one figure |
+| `plot_location_resolution(df)` | panel, optionally after `geocode_frame` | source of each coordinate, or the reason for its absence |
+| `plot_overview(df)` | panel | coverage, quality, stage and capacity in four panels |
+| `save_figure(fig, path)` | figure | PNG, SVG or PDF from the suffix, 300 dpi by default |
+| `save_all(df, out_dir, edition_b=...)` | panel | every chart the data supports; skipped ones listed |
+
+## Examples
+
+```python
+from gridqueue import (parse_sources, compare_editions, linkage_audit, linkage_recall,
+ assign_to_nodes, node_loading, station_node_map, plots)
+
+panel, _ = parse_sources(["wnioski.pdf", "moc_odbiorcza.pdf", "moc_wytworcza.pdf"])
+plots.plot_processing_time(panel, by="klasa_zasobu", min_n=30)
+
+L = node_loading(assign_to_nodes(panel, station_node_map(panel)), napiecie="110")
+plots.save_figure(plots.plot_node_loading(L), "node_loading.pdf")
+
+a, b = parse_sources(["edycja_07.xlsx"])[0], parse_sources(["edycja_08.xlsx"])[0]
+d = compare_editions(a, b, "2026-07-31", "2026-08-31")
+plots.plot_edition_comparison(d, linkage_audit(a, b), linkage_recall(a, b))
+```
+
+From the command line:
+
+```bash
+gridqueue plot panel.parquet --out charts/ --format png,svg
+gridqueue plot edition_07.parquet --edition-b edition_08.parquet --label-a 2026-07-31 --label-b 2026-08-31 --out charts/
+```
+
+Without a real disclosure, `gridqueue.examples.synthetic_panel()` and `synthetic_editions()`
+produce labelled synthetic data on which every chart can be tried.
diff --git a/notebooks/gridqueue_colab.ipynb b/notebooks/gridqueue_colab.ipynb
index 0273c67..9f30688 100644
--- a/notebooks/gridqueue_colab.ipynb
+++ b/notebooks/gridqueue_colab.ipynb
@@ -8,27 +8,33 @@
"\n",
"[](https://colab.research.google.com/github/s-matysik/gridqueue/blob/main/notebooks/gridqueue_colab.ipynb)\n",
"\n",
- "This notebook runs the **full research workflow with nothing installed locally**: it downloads\n",
- "two editions of the statutory connection disclosure straight from the publisher, turns them into rows\n",
- "of the common schema, assesses the quality of the disclosure and reproduces the longitudinal analysis\n",
- "described in the article.\n",
+ "This notebook runs the **full research workflow with nothing installed locally**, and it exercises\n",
+ "every function of the package: parsing, schema validation, quality rules, the column-shift audit,\n",
+ "the longitudinal analysis with its linkage audit, processing times with censoring, node loading,\n",
+ "coordinates from a gazetteer, charts and export.\n",
"\n",
"**What you will see after running all the cells**\n",
"\n",
- "1. two official documents downloaded from the publisher's addresses, with checksums;\n",
- "2. rows in the 21-field schema derived from the structure of the legal duty;\n",
- "3. a report of the eight quality control rules;\n",
- "4. flows in the queue between editions together with the **linkage precision** measured on the\n",
- " fields excluded from the key;\n",
+ "1. two editions of the transmission system operator's disclosure downloaded from the publisher, with checksums;\n",
+ "2. rows in the 21-field schema derived from the structure of the legal duty, and their validation;\n",
+ "3. the eight quality rules and the column-shift audit, as tables and as charts;\n",
+ "4. flows in the queue between editions, the status transition matrix, the **linkage precision** measured\n",
+ " on fields excluded from the key and the **key sensitivity**;\n",
"5. a comparison of the reproduced numbers with the values published in the article: the cell\n",
- " **stops execution** if they diverge.\n",
+ " **stops execution** if they diverge;\n",
+ "6. for a distribution operator that discloses process dates and node capacity (ENERGA-OPERATOR):\n",
+ " processing-time curves with censoring and node loading;\n",
+ "7. coordinates from a gazetteer, with the reason wherever none can be given;\n",
+ "8. every chart saved as PNG and SVG, and the data exported as Parquet, CSV and XLSX.\n",
"\n",
- "Time for a full run on a free Colab machine: under a minute. No accelerator is needed.\n",
+ "**Time.** Sections 1 to 8 take about a minute on a free Colab machine. Sections 9 and 10 parse\n",
+ "three ENERGA-OPERATOR PDF documents, which takes several minutes; set `RUN_ENERGA = False` in\n",
+ "section 9 to skip them. No accelerator is needed.\n",
"\n",
"**What this notebook does not do.** It does not reproduce the extraction accuracy against the\n",
- "reference set, because that one requires a human reading, nor does it resolve coordinates, because the\n",
- "geometric gazetteer is a separate download under the ODbL licence. Both are described in the\n",
- "documentation."
+ "reference set, because that requires a human reading. The national gazetteer used in the article is\n",
+ "a separate download under the ODbL licence; section 11 shows the resolver on a small synthetic\n",
+ "gazetteer and how to load your own."
]
},
{
@@ -38,7 +44,7 @@
"## 1. Installation\n",
"\n",
"A single command, from a **tagged release**, so that the result does not depend on the state of the\n",
- "main branch."
+ "main branch. The `plot` extra installs matplotlib for the charts."
]
},
{
@@ -47,10 +53,11 @@
"execution_count": null,
"outputs": [],
"source": [
- "%pip install -q git+https://github.com/s-matysik/gridqueue@v1.2.1\n",
+ "%pip install -q \"gridqueue[plot] @ git+https://github.com/s-matysik/gridqueue@v1.6.0\"\n",
"\n",
"import gridqueue as gq\n",
- "print('gridqueue', gq.__version__, '| schema version', gq.SCHEMA_VERSION)"
+ "from gridqueue import plots\n",
+ "print('gridqueue', gq.__version__, '| schema version', gq.SCHEMA_VERSION, '| adapters', len(gq.REGISTRY))"
]
},
{
@@ -79,7 +86,7 @@
"WZOR = 'Informacje_publikowane_zgodnie_z_Art_7_ust_8l_ustawy_PE_stan_na_{}.xlsx'\n",
"EDYCJE = {'2026-07-31': '31_07_2026', '2026-08-31': '31_08_2026'}\n",
"\n",
- "NAGL = {'User-Agent': 'gridqueue-colab/1.2 (statutory grid connection disclosure)'}\n",
+ "NAGL = {'User-Agent': 'gridqueue-colab/1.6 (statutory grid connection disclosure)'}\n",
"RADA = (\" Download the current list from https://www.pse.pl, upload it through the Colab\\n\"\n",
" \" file panel and enter its name in the `pliki` dictionary below, for example:\\n\"\n",
" \" pliki['2026-08-31'] = pathlib.Path('moj_wykaz.xlsx')\")\n",
@@ -124,7 +131,9 @@
"## 3. Recognising the publisher and extracting the rows\n",
"\n",
"The publisher is **recognised from the content of the document**, not supplied by hand. The\n",
- "extraction rests on the position of words and on column geometry, not on matching text patterns."
+ "extraction rests on the position of words and on column geometry, not on matching text patterns.\n",
+ "`parse_sources` takes one or many files, of one or several publishers, and returns one panel together\n",
+ "with the report of every adapter call."
]
},
{
@@ -133,14 +142,13 @@
"execution_count": null,
"outputs": [],
"source": [
- "from gridqueue import detect_publisher, get_adapter\n",
+ "from gridqueue import parse_sources\n",
"\n",
- "ramki = {}\n",
+ "ramki, raporty = {}, {}\n",
"for etykieta, sciezka in pliki.items():\n",
- " klucz = detect_publisher(str(sciezka))\n",
- " wynik = get_adapter(klucz).parse(str(sciezka))\n",
- " ramki[etykieta] = wynik.frame\n",
- " print(f'{etykieta}: recognised {klucz} | {len(wynik.frame)} rows, {wynik.frame.shape[1]} columns')\n",
+ " ramki[etykieta], raporty[etykieta] = parse_sources([str(sciezka)])\n",
+ " r = raporty[etykieta][0]\n",
+ " print(f\"{etykieta}: recognised {r['adapter']} | {r['rows']} rows, {ramki[etykieta].shape[1]} columns\")\n",
"\n",
"ramki['2026-08-31'].head(3).T"
]
@@ -168,7 +176,31 @@
"pokrycie = (df[list(FIELDS)].notna().mean() * 100).round(2).sort_values(ascending=False)\n",
"print(f'schema {SCHEMA_VERSION}: {len(FIELDS)} fields, filled in {(pokrycie > 0).sum()}')\n",
"print(f'extension columns: {sum(1 for c in df.columns if c.startswith(\"_\"))}')\n",
- "pokrycie[pokrycie > 0].to_frame('coverage [%]')"
+ "pokrycie[pokrycie > 0].to_frame('coverage [%]')\n",
+ "\n",
+ "plots.plot_field_coverage(df)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Validation against the schema\n",
+ "\n",
+ "`validate_frame` checks every field against its type and controlled vocabulary and returns figures, not\n",
+ "an exception."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {},
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "from gridqueue import validate_frame\n",
+ "\n",
+ "walidacja = validate_frame(df)\n",
+ "{k: v for k, v in walidacja.items() if not isinstance(v, (list, dict)) or len(str(v)) < 300}"
]
},
{
@@ -192,7 +224,51 @@
"\n",
"raport = run_quality(df)\n",
"print('violations in total:', raport.as_dict()['naruszen_lacznie'], 'on', len(df), 'rows')\n",
- "raport.as_frame()"
+ "display(raport.as_frame())\n",
+ "plots.plot_quality(df)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Column-shift audit\n",
+ "\n",
+ "Counts rows in which a value stands in a field it cannot belong to: a number in a date field, status text\n",
+ "in a capacity field. With layout-based extraction the expected result is zero. It detects internal type\n",
+ "inconsistency, not disagreement with a reference."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {},
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "from gridqueue import audit_column_shift\n",
+ "\n",
+ "print(audit_column_shift(df).as_dict())\n",
+ "plots.plot_column_shift(df)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Stage of the proceeding and requested capacity\n",
+ "\n",
+ "Two descriptive views of the queue. Capacity counts applications still in the queue (filed, incomplete,\n",
+ "under review, conditions issued, agreement in force), the same set node loading uses."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {},
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "display(plots.plot_status_distribution(df, normalize=True))\n",
+ "plots.plot_capacity_by_class(df)"
]
},
{
@@ -243,12 +319,13 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "## 7. Linkage precision\n",
+ "## 7. Linkage precision and key sensitivity\n",
"\n",
"We close the indirect evidence with direct evidence. The audit compares every linked pair on **nine\n",
"fields excluded from the key**: entity name, object name, date the application was lodged and six\n",
"fields of the breakdown of capacity by technology. The key forces none of them to agree, so their\n",
- "agreement is independent evidence."
+ "agreement is independent evidence. Sensitivity is the converse question: how many true pairs the key\n",
+ "missed, recovered by identity on the same held-out fields."
]
},
{
@@ -257,15 +334,19 @@
"execution_count": null,
"outputs": [],
"source": [
- "from gridqueue import linkage_audit\n",
+ "from gridqueue import linkage_audit, linkage_recall\n",
"\n",
"audyt = linkage_audit(ramki['2026-07-31'], ramki['2026-08-31'])\n",
- "a = audyt.as_dict()\n",
+ "czulosc = linkage_recall(ramki['2026-07-31'], ramki['2026-08-31'])\n",
+ "a, c = audyt.as_dict(), czulosc.as_dict()\n",
"print(f\"pairs: {a['par']} | confirmed {a['potwierdzonych']}, \"\n",
" f\"uncertain {a['niepewnych']}, false {a['falszywych']}\")\n",
"print(f\"precision: from {a['precyzja_dolna']*100:.2f} to {a['precyzja_gorna']*100:.2f} %\")\n",
+ "print(f\"key sensitivity: {c['czulosc_klucza']:.4f} ({c['odzyskanych']} pairs missed by the key, recovered)\")\n",
"print('\\ndistribution of divergences on the held-out fields:')\n",
- "audyt.rozbieznosci['kategoria'].value_counts().to_frame('count')"
+ "display(audyt.rozbieznosci['kategoria'].value_counts().to_frame('count'))\n",
+ "\n",
+ "plots.plot_edition_comparison(delta, audyt, czulosc)"
]
},
{
@@ -292,9 +373,12 @@
" 'migawka_monotoniczna': False,\n",
"}\n",
"OPUBLIKOWANY_AUDYT = {'par': 804, 'potwierdzonych': 795, 'niepewnych': 6, 'falszywych': 3}\n",
+ "OPUBLIKOWANA_CZULOSC = 0.9652\n",
"\n",
"rozjazd = {k: (v, d[k]) for k, v in OPUBLIKOWANE.items() if d[k] != v}\n",
"rozjazd |= {k: (v, a[k]) for k, v in OPUBLIKOWANY_AUDYT.items() if a[k] != v}\n",
+ "if round(c['czulosc_klucza'], 4) != OPUBLIKOWANA_CZULOSC:\n",
+ " rozjazd['czulosc_klucza'] = (OPUBLIKOWANA_CZULOSC, round(c['czulosc_klucza'], 4))\n",
"\n",
"if rozjazd:\n",
" for k, (opub, otrzym) in rozjazd.items():\n",
@@ -302,23 +386,27 @@
" raise AssertionError('the reproduced numbers do not match the published ones; see above')\n",
"print('All numbers agree with the article.')\n",
"print(f\"share of forward transitions: {d['udzial_przejsc_w_przod']*100:.2f} %\")\n",
- "print(f\"linkage precision: from {a['precyzja_dolna']*100:.2f} to {a['precyzja_gorna']*100:.2f} %\")"
+ "print(f\"linkage precision: from {a['precyzja_dolna']*100:.2f} to {a['precyzja_gorna']*100:.2f} %\")\n",
+ "print(f\"key sensitivity: {c['czulosc_klucza']:.4f}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "## 9. Cross-sectional analysis: why not on this publisher\n",
+ "## 9. Cross-sectional analysis: processing time with censoring\n",
"\n",
"The cross-sectional example in the article measures the time from lodging the application to the issue\n",
"of conditions. The inclusion criterion was set **before** the result was computed: coverage of both\n",
- "dates above 80 % and at least 30 observations in a group. This publisher does **not** meet the\n",
- "criterion; the cell below shows why, and that is more honest than computing a median on a dozen or so\n",
- "per cent of the records.\n",
- "\n",
- "To run the cross-sectional analysis, upload a document of a publisher that does meet the criterion\n",
- "through the Colab file panel and give its path in the cell."
+ "dates above 80 % and at least 30 observations in a group. The transmission system operator does **not**\n",
+ "meet it; the first cell shows why, which is more honest than computing a median on a small share of\n",
+ "the records.\n",
+ "\n",
+ "A distribution operator that does meet it is ENERGA-OPERATOR, which splits one edition into three\n",
+ "documents: applications and refusals, and the available capacity of nodes in the demand and generation\n",
+ "directions. They are parsed **together**, so that application rows and node rows land in one panel.\n",
+ "Applications still open at the as-at date are **censored**, not dropped, and the curves are\n",
+ "Kaplan-Meier estimates with a Greenwood interval."
]
},
{
@@ -329,24 +417,143 @@
"source": [
"for etykieta, r in ramki.items():\n",
" pokr = {p: round(float(r[p].notna().mean()) * 100, 2) for p in ('data_wniosku', 'data_warunkow')}\n",
- " kwalifikuje = all(v > 80 for v in pokr.values())\n",
- " stan = 'met' if kwalifikuje else 'NOT MET'\n",
+ " stan = 'met' if all(v > 80 for v in pokr.values()) else 'NOT MET'\n",
" print(f\"{etykieta}: data_wniosku {pokr['data_wniosku']} %, \"\n",
- " f\"data_warunkow {pokr['data_warunkow']} % -> 80 % criterion: {stan}\")\n",
+ " f\"data_warunkow {pokr['data_warunkow']} % -> 80 % criterion: {stan}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {},
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "RUN_ENERGA = True # set to False to skip sections 9 and 10 (several minutes of PDF parsing)\n",
+ "\n",
+ "ENERGA = {\n",
+ " 'energa_wnioski_odmowy_20260831.pdf':\n",
+ " 'https://cdn-netpr.pl/file/mediakit/3063617/fa/2026_08_31_pusop_wnioski_i_odmowy_eop.pdf',\n",
+ " 'energa_moc_dostepna_odbiorcza_20260831.pdf':\n",
+ " 'https://cdn-netpr.pl/file/mediakit/3063619/5c/informacja_o_wartosci_lacznej_dostepnej_odbiorczej_mocy_przylaczeniowej_31_08_2026.pdf',\n",
+ " 'energa_moc_dostepna_wytworcza_20260831.pdf':\n",
+ " 'https://cdn-netpr.pl/file/mediakit/3063621/4a/informacja_o_wartosci_lacznej_dostepnej_wytworczej_mocy_przylaczeniowej_31_08_2026.pdf',\n",
+ "}\n",
+ "en = None\n",
+ "if RUN_ENERGA:\n",
+ " sciezki = []\n",
+ " for nazwa, url in ENERGA.items():\n",
+ " try:\n",
+ " r = requests.get(url, headers=NAGL, timeout=300)\n",
+ " except requests.exceptions.RequestException as e:\n",
+ " print(f'{nazwa}: connection failed ({type(e).__name__}); upload the file and add it to `sciezki`.')\n",
+ " continue\n",
+ " if r.status_code != 200 or len(r.content) < 10_000:\n",
+ " print(f'{nazwa}: HTTP {r.status_code}; the publisher has replaced the file. '\n",
+ " 'Download the current edition from energa-operator.pl and upload it.')\n",
+ " continue\n",
+ " pathlib.Path(nazwa).write_bytes(r.content)\n",
+ " sciezki.append(nazwa)\n",
+ " print(f'{nazwa}: {len(r.content):,} B | sha256 {hashlib.sha256(r.content).hexdigest()[:16]}…')\n",
+ " if sciezki:\n",
+ " import time\n",
+ " t0 = time.time()\n",
+ " en, raport_en = parse_sources(sciezki)\n",
+ " print(f'\\n{len(en):,} rows in {time.time() - t0:.0f} s:',\n",
+ " en['_encja'].value_counts().to_dict())"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {},
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "from gridqueue import kaplan_meier_by_group\n",
+ "\n",
+ "# Consumption strata of ENERGA-OPERATOR as published (Supplementary Material, Note S6):\n",
+ "# n, events, censored, KM median and its 95 % interval, in days.\n",
+ "OPUBLIKOWANE_KM = {'ODB / SN': (2073, 1982, 91, 57, 56, 58), 'ODB / 110 kV': (83, 43, 40, 240, 229, 265)}\n",
+ "\n",
+ "if en is not None:\n",
+ " km = kaplan_meier_by_group(en, by='klasa_zasobu', min_n=30)\n",
+ " display(km)\n",
+ " display(plots.plot_processing_time(en, by='klasa_zasobu', min_n=30))\n",
+ " # the same estimator split by resource class and voltage level, as in the article\n",
+ " km2 = kaplan_meier_by_group(en, by=['klasa_zasobu', 'poziom_napiecia'], min_n=30)\n",
+ " display(km2)\n",
+ " rozjazd_km = {}\n",
+ " for grupa, opub in OPUBLIKOWANE_KM.items():\n",
+ " r = km2.set_index('grupa').loc[grupa]\n",
+ " otrzym = (int(r.n), int(r.zdarzen), int(r.cenzurowanych), int(r.mediana),\n",
+ " int(r.mediana_dolna), int(r.mediana_gorna))\n",
+ " if otrzym != opub:\n",
+ " rozjazd_km[grupa] = (opub, otrzym)\n",
+ " if rozjazd_km:\n",
+ " raise AssertionError(f'processing times differ from the article (published, reproduced): {rozjazd_km}')\n",
+ " print('Processing times of the consumption strata agree with the article.')\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 10. Node loading\n",
+ "\n",
+ "Node loading λ(n) sets the requested capacity of the active queue at a node against the free capacity the\n",
+ "publisher declares there. Values above 1 mean the queue already exceeds what is declared available. A node\n",
+ "with zero declared free capacity has an undefined ratio and is reported separately, not as infinity.\n",
"\n",
- "# After uploading a qualifying document:\n",
- "# df2 = get_adapter(detect_publisher('moj_plik.pdf')).parse('moj_plik.pdf').frame\n",
- "# and then run examples/research_use_case.py on the exported panel."
+ "The article's **base variant** counts the 110 kV queue, because the free capacity a distribution operator\n",
+ "discloses for a node is that of its high-voltage busbars; the **upper variant** counts applications at\n",
+ "every voltage. Both are computed, and the base variant is checked against the published figures\n",
+ "(Supplementary Material, Note S7): the cell stops if they diverge.\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {},
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "from gridqueue import assign_to_nodes, node_loading, station_node_map\n",
+ "\n",
+ "OPUBLIKOWANE_WEZLY = {'generation': (6.500, 29, 36), 'demand': (4.688, 33, 37)} # Note S7, base variant\n",
+ "\n",
+ "if en is not None:\n",
+ " przyp = assign_to_nodes(en, station_node_map(en))\n",
+ " rozjazd_w = {}\n",
+ " for pole, nazwa in (('moc_wprowadzana', 'generation'), ('moc_pobierana', 'demand')):\n",
+ " for wariant, napiecie in (('base, 110 kV queue', '110'), ('upper, all voltages', None)):\n",
+ " L = node_loading(przyp, pole_mocy=pole, napiecie=napiecie)\n",
+ " v = L['obciazenie'].dropna()\n",
+ " print(f'{nazwa}, {wariant}: {len(v)} of {len(L)} node-directions with a defined loading, '\n",
+ " f'median {v.median():.3f}, {(v > 1).sum()} above 1')\n",
+ " if napiecie == '110':\n",
+ " otrzym = (round(float(v.median()), 3), int((v > 1).sum()), int(len(v)))\n",
+ " if otrzym != OPUBLIKOWANE_WEZLY[nazwa]:\n",
+ " rozjazd_w[nazwa] = (OPUBLIKOWANE_WEZLY[nazwa], otrzym)\n",
+ " display(plots.plot_node_loading(L, title=(\n",
+ " f'Node loading, {nazwa} direction, 110 kV active queue: median {v.median():.3f}, '\n",
+ " f'{(v > 1).sum()} of {len(v)} above 1')))\n",
+ " if rozjazd_w:\n",
+ " raise AssertionError(f'node loading differs from the article (published, reproduced): {rozjazd_w}')\n",
+ " print('Node loading agrees with the article.')\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "## 10. Export\n",
+ "## 11. Coordinates from a gazetteer\n",
+ "\n",
+ "The resolver matches each place description against a gazetteer and returns a point **or the reason\n",
+ "why there is none**: no locality declared, a station code with no dictionary, a locality outside the\n",
+ "gazetteer, several candidates. Absence is reported, not guessed.\n",
"\n",
- "We save the panel in a columnar format and as CSV. In Colab both files can be downloaded from the\n",
- "file panel."
+ "The national gazetteer used in the article is distributed under the ODbL and is not bundled. The cell\n",
+ "below runs the resolver on the **synthetic** panel and gazetteer of `gridqueue.examples`, which describe\n",
+ "no real grid. To use your own gazetteer, upload a JSON file in the layout documented in\n",
+ "`resolver_from_gazeter_json` and replace the path."
]
},
{
@@ -355,11 +562,81 @@
"execution_count": null,
"outputs": [],
"source": [
+ "import json\n",
+ "from gridqueue import examples, geocode_frame, resolver_from_gazeter_json\n",
+ "\n",
+ "pathlib.Path('gazetteer_synthetic.json').write_text(json.dumps(examples.synthetic_gazetteer()))\n",
+ "resolver = resolver_from_gazeter_json('gazetteer_synthetic.json') # or your own gazetteer\n",
+ "syn = geocode_frame(examples.synthetic_panel(), resolver)\n",
+ "print(syn['_geo_pewnosc'].value_counts().to_dict())\n",
+ "print(syn.loc[syn['wspolrzedne'].isna(), '_geo_powod'].value_counts().to_dict())\n",
+ "plots.plot_location_resolution(syn)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 12. All charts at once\n",
+ "\n",
+ "`plots.save_all` draws every chart the data supports and saves it as PNG and SVG; a chart that needs data\n",
+ "the disclosure does not carry is skipped with the reason. The same function is behind\n",
+ "`gridqueue plot` on the command line and the charts of the browser page."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {},
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "wynik = plots.save_all(ramki['2026-07-31'], 'charts/pse', edition_b=ramki['2026-08-31'],\n",
+ " labels=('2026-07-31', '2026-08-31'), formats=('png', 'svg'), dpi=300)\n",
+ "print('PSE skipped:', wynik.pop('_skipped'))\n",
+ "if en is not None:\n",
+ " wynik_en = plots.save_all(en, 'charts/energa', formats=('png', 'svg'), dpi=300)\n",
+ " print('ENERGA skipped:', wynik_en.pop('_skipped'))\n",
+ "plots.plot_overview(df if en is None else en)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 13. Export\n",
+ "\n",
+ "The panel in a columnar format, as CSV and as a workbook, together with the transition matrix, the audit\n",
+ "divergences and the charts, packed into one archive that Colab can download."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {},
+ "execution_count": null,
+ "outputs": [],
+ "source": [
+ "import pandas as pd\n",
+ "import shutil\n",
+ "\n",
"df.to_parquet('panel_pse_2026-08-31.parquet', index=False)\n",
"df.to_csv('panel_pse_2026-08-31.csv', index=False)\n",
+ "with pd.ExcelWriter('panel_pse_2026-08-31.xlsx') as w:\n",
+ " df.to_excel(w, sheet_name='panel', index=False)\n",
+ " raport.as_frame().to_excel(w, sheet_name='quality', index=False)\n",
+ " delta.przejscia.to_excel(w, sheet_name='transitions')\n",
"delta.przejscia.to_csv('przejscia_statusu.csv')\n",
"audyt.rozbieznosci.to_csv('audyt_dopasowania.csv', index=False)\n",
- "print('saved: panel (parquet, csv), transition matrix, audit divergences')"
+ "if en is not None:\n",
+ " en.to_parquet('panel_energa_2026-08-31.parquet', index=False)\n",
+ " km.to_csv('processing_time_energa.csv', index=False)\n",
+ "\n",
+ "archiwum = shutil.make_archive('gridqueue_results', 'zip', '.', 'charts')\n",
+ "print('saved: panels (parquet, csv, xlsx), transition matrix, audit divergences, charts ->', archiwum)\n",
+ "try:\n",
+ " from google.colab import files\n",
+ " files.download(archiwum)\n",
+ "except ImportError:\n",
+ " pass"
]
},
{
diff --git a/pyproject.toml b/pyproject.toml
index 5377418..a8eac24 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -4,14 +4,15 @@ build-backend = "setuptools.build_meta"
[project]
name = "gridqueue"
-version = "1.5.2"
-description = "Harmonizator ustawowych ujawnien przylaczeniowych do sieci elektroenergetycznej"
+version = "1.6.0"
+description = "Harmonizer of statutory grid-connection disclosures into panel data for queue research"
readme = "README.md"
requires-python = ">=3.10"
dependencies = ["pandas>=2.0", "pdfplumber>=0.11", "requests>=2.31", "pyarrow>=14", "openpyxl>=3.1"]
[project.optional-dependencies]
-test = ["pytest>=8.0"]
+plot = ["matplotlib>=3.7"]
+test = ["pytest>=8.0", "matplotlib>=3.7"]
[project.scripts]
gridqueue = "gridqueue.cli:main"
diff --git a/src/gridqueue/__init__.py b/src/gridqueue/__init__.py
index 7c34b36..10d03bf 100644
--- a/src/gridqueue/__init__.py
+++ b/src/gridqueue/__init__.py
@@ -19,9 +19,11 @@
extract_ruled_page, assign_words_to_columns, merge_continuation_rows,
audit_column_shift, ShiftAudit,
)
-from .geoloc import LocationResolver, Resolution, resolver_from_osm_json # noqa: F401
+from .geoloc import (LocationResolver, MultiLocalityResolver, Resolution, # noqa: F401
+ geocode_frame, resolver_from_gazeter_json, resolver_from_osm_json)
from .analysis import ( # noqa: F401
- PRZEDROSTKI_STACJI, KMResult, assign_to_nodes, kaplan_meier, node_loading,
+ PRZEDROSTKI_STACJI, KMResult, assign_to_nodes, kaplan_meier, kaplan_meier_by_group,
+ node_loading,
normalizuj_nazwe, processing_time, station_node_map,
)
from .quality import RULES, run_quality, QualityReport # noqa: F401
@@ -31,13 +33,27 @@
surrogate_key, EditionDelta, compare_editions,
HELD_OUT_FIELDS, LinkageAudit, linkage_audit,
)
-from .registry import REGISTRY, get_adapter, detect_publisher, declarations # noqa: F401
+from .registry import (REGISTRY, get_adapter, detect_publisher, declarations, # noqa: F401
+ parse_sources, split_editions)
-__version__ = "1.5.2"
+# Charts live in `gridqueue.plots` and synthetic data in `gridqueue.examples`. Both are imported
+# on first access, so that the core of the package does not require matplotlib:
+# from gridqueue import plots
+_LAZY_SUBMODULES = ("plots", "examples")
+
+
+def __getattr__(name):
+ if name in _LAZY_SUBMODULES:
+ import importlib
+
+ return importlib.import_module(f".{name}", __name__)
+ raise AttributeError(f"module 'gridqueue' has no attribute {name!r}")
+
+__version__ = "1.6.0"
__all__ = [
"NODE_KEY_FIELDS",
"RECALL_FIELDS", "LinkageRecall", "linkage_recall",
- "PRZEDROSTKI_STACJI", "KMResult", "assign_to_nodes", "kaplan_meier",
+ "PRZEDROSTKI_STACJI", "KMResult", "assign_to_nodes", "kaplan_meier", "kaplan_meier_by_group",
"node_loading", "normalizuj_nazwe", "processing_time", "station_node_map",
"SCHEMA", "SCHEMA_VERSION", "FIELDS", "CORE_FIELDS", "OPTIONAL_FIELDS", "GROUPS",
"CORE_ALTERNATIVES", "CORE_BY_ENTITY", "ENTITY_COLUMN", "DEFAULT_ENTITY",
@@ -46,10 +62,11 @@
"empty_frame", "schema_table", "ColumnGeometry", "detect_ruled_columns",
"detect_banded_columns", "extract_ruled_page", "assign_words_to_columns",
"merge_continuation_rows", "audit_column_shift", "ShiftAudit",
- "LocationResolver", "Resolution", "resolver_from_osm_json",
+ "LocationResolver", "Resolution", "resolver_from_osm_json", "resolver_from_gazeter_json", "geocode_frame",
+ "MultiLocalityResolver",
"RULES", "run_quality", "QualityReport",
"DEFAULT_KEY_FIELDS", "STATUS_ORDER", "TERMINAL_NEGATIVE",
"surrogate_key", "EditionDelta", "compare_editions",
"HELD_OUT_FIELDS", "LinkageAudit", "linkage_audit",
- "REGISTRY", "get_adapter", "detect_publisher", "declarations", "__version__",
+ "REGISTRY", "get_adapter", "detect_publisher", "declarations", "parse_sources", "split_editions", "__version__",
]
diff --git a/src/gridqueue/analysis.py b/src/gridqueue/analysis.py
index 8b577d4..dd70636 100644
--- a/src/gridqueue/analysis.py
+++ b/src/gridqueue/analysis.py
@@ -35,6 +35,7 @@
"node_loading",
"processing_time",
"kaplan_meier",
+ "kaplan_meier_by_group",
"KMResult",
]
@@ -218,6 +219,7 @@ def node_loading(
kolumna_grupy: str = "_nr_grupy",
statusy_czynne: Iterable[str] | None = None,
kolumna_encji: str = "_encja",
+ napiecie: str | None = None,
) -> pd.DataFrame:
"""Node loading: the capacity of the active queue against the disclosed available capacity.
@@ -233,6 +235,12 @@ def node_loading(
`moc_dostepna_zero` flag: the ratio there is UNDEFINED, not infinite,
and averaging it would be an error. The row nonetheless retains the capacity applied for,
because a queue at a node with no free capacity is information, not missing data.
+
+ ``napiecie`` keeps only applications whose ``poziom_napiecia`` contains the given text.
+ The base variant of the article counts the 110 kV queue (``napiecie="110"``), because the
+ available capacity a distribution operator discloses for a node is the capacity of its
+ high-voltage busbars; the default (``None``) counts applications at every voltage level,
+ which is the article's upper variant.
"""
if statusy_czynne is None:
statusy_czynne = ("WNIOSEK_ZLOZONY", "WNIOSEK_NIEKOMPLETNY",
@@ -245,6 +253,9 @@ def node_loading(
else df["moc_dostepna"].notna()
wnioski = df[~jest_wezel]
wnioski = wnioski[wnioski["status_procesu"].isin(set(statusy_czynne))]
+ if napiecie is not None and "poziom_napiecia" in wnioski.columns:
+ wnioski = wnioski[wnioski["poziom_napiecia"].astype("string")
+ .str.contains(napiecie, regex=False, na=False)]
wnioski = wnioski[wnioski["_grupa"].notna()]
suma = wnioski.groupby("_grupa")[pole_mocy].sum(min_count=1).rename("moc_wnioskowana")
liczba = wnioski.groupby("_grupa").size().rename("wnioskow")
@@ -279,24 +290,37 @@ def processing_time(
both kinds of observations, as well as `zdarzenie` equal to 1 for resolved ones and 0
for censored ones, in the form expected by the survival estimator.
- `na_dzien` by default takes the maximum of the publication-date field, and when that is
- absent, the maximum of the resolution dates.
+ By default (``na_dzien=None``) every open application is censored at the publication
+ date of ITS OWN row (``data_publikacji``), so that a panel of several publishers, or a
+ publisher whose node and application documents carry different as-at dates, censors
+ each application at the date its document describes. Rows without a publication date
+ are censored at the latest publication date of the application rows, and failing that
+ at the latest resolution date. A date given explicitly applies to every row.
+
+ Up to 1.5.2 the default was the single latest publication date of the whole frame,
+ node rows included; for ENERGA-OPERATOR (applications as at 30 June, nodes as at
+ 31 August 2026) that lengthened every censored time by two months.
"""
d = df.copy()
for c in (od, do):
d[c] = pd.to_datetime(d[c], errors="coerce")
if na_dzien is None:
- if "data_publikacji" in d:
- na_dzien = pd.to_datetime(d["data_publikacji"], errors="coerce").max()
- if na_dzien is None or pd.isna(na_dzien):
- na_dzien = d[do].max()
- na_dzien = pd.to_datetime(na_dzien)
+ wlasna = pd.to_datetime(d["data_publikacji"], errors="coerce") \
+ if "data_publikacji" in d else pd.Series(pd.NaT, index=d.index)
+ wnioski = d["_encja"].fillna("WNIOSEK").astype(str).str.upper() != "WEZEL" \
+ if "_encja" in d else pd.Series(True, index=d.index)
+ zapas = wlasna[wnioski].max()
+ if pd.isna(zapas):
+ zapas = d[do].max()
+ na_dzien = wlasna.fillna(zapas)
+ else:
+ na_dzien = pd.Series(pd.to_datetime(na_dzien), index=d.index)
ma_start = d[od].notna()
zdarzenie = ma_start & d[do].notna()
czas = pd.Series(pd.NA, index=d.index, dtype="Float64")
czas[zdarzenie] = (d.loc[zdarzenie, do] - d.loc[zdarzenie, od]).dt.days
uciete = ma_start & ~zdarzenie
- czas[uciete] = (na_dzien - d.loc[uciete, od]).dt.days
+ czas[uciete] = (na_dzien[uciete] - d.loc[uciete, od]).dt.days
d["czas_dni"] = czas
d["zdarzenie"] = zdarzenie.astype(int).where(ma_start)
d["na_dzien"] = na_dzien
@@ -357,18 +381,60 @@ def kaplan_meier(czas, zdarzenie, alpha: float = 0.05) -> KMResult:
se = S * math.sqrt(kum_war) if S > 0 else 0.0
wiersze.append({"t": float(t), "n_ryzyka": n_ryz, "zdarzen": zd, "S": S,
"S_dolne": max(0.0, S - z * se), "S_gorne": min(1.0, S + z * se)})
- tab = pd.DataFrame(wiersze)
+ # Explicit columns: a group in which every observation is censored has no event time,
+ # and a frame built from an empty list would have no columns at all (KeyError: 'S').
+ tab = pd.DataFrame(wiersze, columns=["t", "n_ryzyka", "zdarzen", "S", "S_dolne", "S_gorne"])
def pierwszy_ponizej(kol):
pod = tab[tab[kol] <= 0.5]
return float(pod["t"].iloc[0]) if len(pod) else float("nan")
+ # The lower band reaches 0.5 FIRST, so it gives the LOWER bound of the median; the upper
+ # band reaches it last and gives the upper bound. Versions up to 1.5.2 had the two
+ # swapped, which returned an interval with mediana_dolna > mediana > mediana_gorna.
return KMResult(
tabela=tab,
mediana=pierwszy_ponizej("S"),
- mediana_dolna=pierwszy_ponizej("S_gorne"),
- mediana_gorna=pierwszy_ponizej("S_dolne"),
+ mediana_dolna=pierwszy_ponizej("S_dolne"),
+ mediana_gorna=pierwszy_ponizej("S_gorne"),
n=n0,
zdarzen=int((d["e"] == 1).sum()),
cenzurowanych=int((d["e"] == 0).sum()),
)
+
+
+def kaplan_meier_by_group(
+ df: pd.DataFrame,
+ by: str | Sequence[str] | None = "klasa_zasobu",
+ od: str = "data_wniosku",
+ do: str = "data_warunkow",
+ min_n: int = 30,
+ na_dzien=None,
+) -> pd.DataFrame:
+ """Kaplan-Meier summary per group: size, events, censoring, median and its interval.
+
+ Combines :func:`processing_time` and :func:`kaplan_meier`. A group enters the table
+ when it has at least ``min_n`` measurable applications (start date present, time not
+ negative); this is the inclusion criterion of the article, fixed before the result is
+ computed. ``by`` may be a column, a list of columns (joined with ``" / "``) or ``None``
+ for the whole frame. Sorted by group size, largest first.
+ """
+ cols = ["grupa", "n", "zdarzen", "cenzurowanych", "udzial_cenzurowanych",
+ "mediana", "mediana_dolna", "mediana_gorna"]
+ if od not in df.columns or do not in df.columns:
+ return pd.DataFrame(columns=cols)
+ pt = processing_time(df, od=od, do=do, na_dzien=na_dzien)
+ if by is None:
+ klucz = pd.Series("all", index=pt.index)
+ else:
+ kol = [by] if isinstance(by, str) else list(by)
+ if any(k not in pt.columns for k in kol):
+ return pd.DataFrame(columns=cols)
+ klucz = pt[kol].astype("string").fillna("(empty)").agg(" / ".join, axis=1)
+ wiersze = []
+ for g, sub in pt.groupby(klucz, sort=False):
+ km = kaplan_meier(sub["czas_dni"].astype("float64"), sub["zdarzenie"].astype("float64"))
+ if km.n >= min_n:
+ wiersze.append({"grupa": str(g), **km.as_dict()})
+ out = pd.DataFrame(wiersze, columns=cols)
+ return out.sort_values("n", ascending=False, kind="stable").reset_index(drop=True)
diff --git a/src/gridqueue/cli.py b/src/gridqueue/cli.py
index 6c64af0..cf36c9b 100644
--- a/src/gridqueue/cli.py
+++ b/src/gridqueue/cli.py
@@ -1,4 +1,4 @@
-"""Command-line interface: ``gridqueue parse | validate | export``."""
+"""Command-line interface: ``gridqueue parse | validate | export | compare | plot``."""
from __future__ import annotations
@@ -91,6 +91,42 @@ def cmd_export(args) -> int:
return 0
+def cmd_compare(args) -> int:
+ from .longitudinal import (NODE_KEY_FIELDS, DEFAULT_KEY_FIELDS, compare_editions,
+ linkage_audit, linkage_recall)
+
+ a, b = _read_table(args.edition_a), _read_table(args.edition_b)
+ fields = NODE_KEY_FIELDS if args.node_key else DEFAULT_KEY_FIELDS
+ delta = compare_editions(a, b, args.label_a, args.label_b, fields=fields)
+ out = {"klucz": list(fields), "porownanie": delta.as_dict(),
+ "audyt_dopasowania": linkage_audit(a, b, fields=fields).as_dict(),
+ "czulosc_dopasowania": linkage_recall(a, b, fields=fields).as_dict()}
+ text = json.dumps(out, ensure_ascii=False, indent=2, default=str)
+ if args.out:
+ Path(args.out).write_text(text, encoding="utf-8")
+ if args.transitions:
+ delta.przejscia.to_csv(args.transitions)
+ print(text)
+ return 0
+
+
+def cmd_plot(args) -> int:
+ try:
+ from .plots import save_all
+ except ImportError as e: # pragma: no cover
+ print(str(e), file=sys.stderr)
+ return 2
+ df = _read_table(args.input)
+ b = _read_table(args.edition_b) if args.edition_b else None
+ saved = save_all(df, args.out, edition_b=b, labels=(args.label_a, args.label_b),
+ formats=[f.strip() for f in args.format.split(",") if f.strip()],
+ dpi=args.dpi, min_n=args.min_n, min_coverage=args.min_coverage)
+ skipped = saved.pop("_skipped", [])
+ print(json.dumps({"zapisane": {k: [str(p) for p in v] for k, v in saved.items()},
+ "pominiete": skipped}, ensure_ascii=False, indent=2))
+ return 0
+
+
def cmd_schema(args) -> int:
t = schema_table()
if args.out:
@@ -130,6 +166,32 @@ def build_parser() -> argparse.ArgumentParser:
help="omit extensions (columns with the _ prefix)")
p.set_defaults(func=cmd_export)
+ p = sub.add_parser("compare", help="compare two editions of one register (flows, transitions, "
+ "linkage precision and sensitivity)")
+ p.add_argument("edition_a", help="earlier edition (.parquet/.csv in the schema)")
+ p.add_argument("edition_b", help="later edition")
+ p.add_argument("--label-a", default="A")
+ p.add_argument("--label-b", default="B")
+ p.add_argument("--node-key", action="store_true",
+ help="use the node key (publikujacy, id_wezla) for NODE-entity registers")
+ p.add_argument("--out", help="JSON report file")
+ p.add_argument("--transitions", help="CSV file for the status transition matrix")
+ p.set_defaults(func=cmd_compare)
+
+ p = sub.add_parser("plot", help="draw every chart the dataset supports (needs matplotlib)")
+ p.add_argument("input", help="dataset in the schema (.parquet/.csv)")
+ p.add_argument("--edition-b", help="a later edition: adds the edition-comparison chart")
+ p.add_argument("--label-a", default="A")
+ p.add_argument("--label-b", default="B")
+ p.add_argument("--out", required=True, help="output directory")
+ p.add_argument("--format", default="png", help="comma-separated: png,svg,pdf")
+ p.add_argument("--dpi", type=int, default=200)
+ p.add_argument("--min-n", type=int, default=30,
+ help="minimum group size for the processing-time curves")
+ p.add_argument("--min-coverage", type=float, default=0.8,
+ help="minimum share of applications carrying both dates (default 0.8)")
+ p.set_defaults(func=cmd_plot)
+
p = sub.add_parser("schema", help="print the schema specification")
p.add_argument("--out")
p.set_defaults(func=cmd_schema)
diff --git a/src/gridqueue/examples.py b/src/gridqueue/examples.py
new file mode 100644
index 0000000..ad011a0
--- /dev/null
+++ b/src/gridqueue/examples.py
@@ -0,0 +1,166 @@
+"""Synthetic data in the schema, for trying the package without a real disclosure.
+
+Nothing here comes from a publisher. The frames are generated from a seed, they carry the
+publisher names ``"Synthetic operator A"`` and ``"Synthetic operator B"`` so that they
+cannot be mistaken for a register, and they are built to exercise every function of the
+package: application and node rows (for node loading), process dates with censoring (for
+the processing-time curves), two editions with linked, new, absent and status-changed rows
+(for the longitudinal module), and a small gazetteer with coordinates (for the resolver).
+
+The browser page and the Colab notebook use these frames for their demonstration mode.
+The figures they produce describe the generator, not any grid.
+"""
+
+from __future__ import annotations
+
+import random
+from datetime import date, timedelta
+
+import pandas as pd
+
+from .schema import empty_frame
+
+__all__ = ["SYNTHETIC_PUBLISHERS", "synthetic_panel", "synthetic_editions",
+ "synthetic_gazetteer"]
+
+SYNTHETIC_PUBLISHERS = ("Synthetic operator A", "Synthetic operator B")
+
+_STATIONS = {
+ 1: ("Alder", "Birch"), 2: ("Cedar",), 3: ("Elm", "Fir"), 4: ("Hazel",),
+ 5: ("Larch", "Maple"), 6: ("Oak",), 7: ("Pine", "Rowan"), 8: ("Spruce",),
+}
+_CLASSES = (("PV", 0.42), ("ODB", 0.28), ("BESS", 0.12), ("FW", 0.08), ("MIX", 0.06),
+ ("BIOGAZ", 0.04))
+_MEDIAN_DAYS = {"PV": 130, "ODB": 60, "BESS": 200, "FW": 210, "MIX": 125, "BIOGAZ": 150}
+_TOWNS = {"RIVERTON": (53.43, 14.55), "HILLSIDE": (53.56, 14.20), "LAKEVIEW": (53.30, 14.90)}
+_STREETS = ("Station Road", "Mill Lane", "Harbour Street", "Orchard Way", "Quarry Road")
+
+
+def _iso(d: date) -> str:
+ return d.isoformat()
+
+
+def _pick(rng: random.Random, weighted):
+ x, acc = rng.random(), 0.0
+ for v, w in weighted:
+ acc += w
+ if x <= acc:
+ return v
+ return weighted[-1][0]
+
+
+def synthetic_panel(n: int = 600, *, seed: int = 7, as_at: str = "2026-06-30",
+ with_nodes: bool = True) -> pd.DataFrame:
+ """A synthetic panel of ``n`` applications (plus node rows) in the 21-field schema."""
+ rng = random.Random(seed)
+ asat = date.fromisoformat(as_at)
+ rows = []
+ for i in range(n):
+ pub = SYNTHETIC_PUBLISHERS[0] if rng.random() < 0.6 else SYNTHETIC_PUBLISHERS[1]
+ klasa = _pick(rng, _CLASSES)
+ filed = asat - timedelta(days=rng.randint(20, 900))
+ wait = int(rng.expovariate(1 / _MEDIAN_DAYS[klasa]) * 1.1) + 14
+ cond = filed + timedelta(days=wait)
+ r = {"id_wniosku": f"SYN-{i:05d}", "publikujacy": pub, "klasa_zasobu": klasa,
+ "data_wniosku": _iso(filed), "data_publikacji": as_at, "_encja": "WNIOSEK"}
+ grp = rng.randint(1, len(_STATIONS))
+ stn = rng.choice(_STATIONS[grp])
+ town = rng.choice(list(_TOWNS))
+ if rng.random() < 0.55:
+ r["lokalizacja_tekst"] = stn
+ else:
+ r["lokalizacja_tekst"] = f"{town.title()}, {rng.choice(_STREETS)}"
+ r["poziom_napiecia"] = "SN" if rng.random() < 0.7 else "110 kV"
+ mw = round(rng.lognormvariate(0.4 if klasa != "ODB" else -0.3, 1.0), 3)
+ r["moc_pobierana" if klasa == "ODB" else "moc_wprowadzana"] = round(mw * 1000, 1)
+ if cond <= asat:
+ r["data_warunkow"] = _iso(cond)
+ u = rng.random()
+ if u < 0.35:
+ r["data_umowy"] = _iso(min(asat, cond + timedelta(days=rng.randint(20, 200))))
+ r["status_procesu"] = "UMOWA_OBOWIAZUJACA"
+ elif u < 0.42:
+ r["status_procesu"] = "WARUNKI_WYGASLE"
+ else:
+ r["status_procesu"] = "WARUNKI_WYDANE"
+ else:
+ u = rng.random()
+ if u < 0.08:
+ r["status_procesu"] = "ODMOWA"
+ r["data_odmowy"] = _iso(filed + timedelta(days=rng.randint(20, 120)))
+ r["powod_odmowy"] = "brak warunkow technicznych"
+ elif u < 0.14:
+ r["status_procesu"] = "WNIOSEK_NIEKOMPLETNY"
+ else:
+ r["status_procesu"] = "W_TRAKCIE_ANALIZY"
+ rows.append(r)
+ if with_nodes:
+ for grp, stations in _STATIONS.items():
+ for kier in ("wytworcza", "odbiorcza"):
+ free = 0.0 if grp == 8 else round(rng.uniform(2, 60) * 1000, 1)
+ rows.append({"_encja": "WEZEL", "id_wezla": f"N{grp}-{kier[0].upper()}",
+ "nazwa_wezla": stations[0], "_nr_grupy": grp, "_kierunek": kier,
+ "lokalizacja_tekst": ", ".join(stations), "moc_dostepna": free,
+ "publikujacy": SYNTHETIC_PUBLISHERS[0],
+ "data_publikacji": as_at})
+ df = pd.DataFrame(rows)
+ cols = list(empty_frame().columns) + [c for c in df.columns
+ if c not in empty_frame().columns]
+ return df.reindex(columns=cols)
+
+
+_FORWARD = {"W_TRAKCIE_ANALIZY": "WARUNKI_WYDANE", "WNIOSEK_NIEKOMPLETNY": "W_TRAKCIE_ANALIZY",
+ "WARUNKI_WYDANE": "UMOWA_OBOWIAZUJACA", "UMOWA_OBOWIAZUJACA": "PRZYLACZONY"}
+
+
+def synthetic_editions(n: int = 400, *, seed: int = 11) -> tuple[pd.DataFrame, pd.DataFrame]:
+ """Two consecutive synthetic editions of one register (applications only).
+
+ Between them about 6 % of rows disappear, 5 % are added, 8 % advance one stage, and a
+ handful move backwards, so the flows, the transition matrix and the linkage audit all
+ have something to show.
+ """
+ rng = random.Random(seed)
+ a = synthetic_panel(n, seed=seed, as_at="2026-05-31", with_nodes=False)
+ a["publikujacy"] = SYNTHETIC_PUBLISHERS[0]
+ b = a.copy()
+ b["data_publikacji"] = "2026-06-30"
+ keep = [rng.random() > 0.06 for _ in range(len(b))]
+ b = b[keep].copy()
+ back = 0
+ for idx in b.index:
+ s = b.at[idx, "status_procesu"]
+ u = rng.random()
+ if u < 0.08 and s in _FORWARD:
+ b.at[idx, "status_procesu"] = _FORWARD[s]
+ if s == "W_TRAKCIE_ANALIZY":
+ b.at[idx, "data_warunkow"] = "2026-06-15"
+ elif back < 2 and u > 0.97 and s == "WARUNKI_WYDANE":
+ # two deliberate backward moves: the transition matrix should show what a
+ # register that rewrites its own history looks like
+ b.at[idx, "status_procesu"] = "W_TRAKCIE_ANALIZY"
+ back += 1
+ new = synthetic_panel(int(n * 0.05) + 1, seed=seed + 1, as_at="2026-06-30",
+ with_nodes=False)
+ new["publikujacy"] = SYNTHETIC_PUBLISHERS[0]
+ new["id_wniosku"] = [f"SYN-NEW-{i:04d}" for i in range(len(new))]
+ new["data_wniosku"] = "2026-06-10"
+ new["data_warunkow"] = None
+ new["data_umowy"] = None
+ new["status_procesu"] = "WNIOSEK_ZLOZONY"
+ b = pd.concat([b, new], ignore_index=True)
+ return a.reset_index(drop=True), b
+
+
+def synthetic_gazetteer() -> dict:
+ """A small gazetteer in the layout read by :func:`~gridqueue.geoloc.resolver_from_gazeter_json`."""
+ out = {}
+ for k, (lat, lon) in _TOWNS.items():
+ ulice = {}
+ for j, st in enumerate(_STREETS):
+ if k == "LAKEVIEW" and j > 2:
+ continue # streets missing from the gazetteer, to show an honest "no candidate"
+ ulice[st] = {"lat": round(lat + 0.004 * (j - 2), 6),
+ "lon": round(lon + 0.006 * (j - 2), 6), "dlugosc_m": 400 + 150 * j}
+ out[k] = {"nazwa": k.title(), "lat": lat, "lon": lon, "ulice": ulice}
+ return {"miejscowosci": out}
diff --git a/src/gridqueue/geoloc.py b/src/gridqueue/geoloc.py
index 102e4e7..b122f24 100644
--- a/src/gridqueue/geoloc.py
+++ b/src/gridqueue/geoloc.py
@@ -56,7 +56,7 @@
__all__ = [
"Resolution", "LocationResolver", "MultiLocalityResolver",
"PEWNOSC", "DOKLADNOSC", "normalize", "tokens",
- "resolver_from_osm_json", "resolver_from_gazeter_json",
+ "resolver_from_osm_json", "resolver_from_gazeter_json", "geocode_frame",
]
PEWNOSC = ("EXACT", "UNIQUE_TOKEN", "TOKEN_SUBSET", "REVERSED", "SKRZYZOWANIE", "NONE")
@@ -677,3 +677,46 @@ def resolver_from_osm_json(path) -> LocationResolver:
lat, lon = item.get("lat"), item.get("lon")
gaz[nm] = (float(lat), float(lon)) if lat is not None and lon is not None else None
return LocationResolver(gaz)
+
+
+def geocode_frame(df, resolver, *, column: str = "lokalizacja_tekst",
+ overwrite: bool = False, station_code_column: Optional[str] = None):
+ """Resolve the place descriptions of a frame and write the result back to it.
+
+ Returns a copy of ``df`` with ``wspolrzedne`` filled as ``"lat,lon"`` (WGS84, the
+ format the schema validates) and three diagnostic columns that keep what the resolver
+ knows about each row: ``_geo_pewnosc`` (certainty of the name identification),
+ ``_geo_dokladnosc`` (precision of the returned point) and ``_geo_powod`` (reason, when
+ no point is returned). Coordinates disclosed by the publisher are kept unless
+ ``overwrite=True``; such rows get ``_geo_pewnosc = "ZRODLO"``.
+
+ ``resolver`` is a :class:`LocationResolver` or :class:`MultiLocalityResolver`, for
+ example from :func:`resolver_from_gazeter_json`. ``station_code_column`` names a boolean
+ column marking descriptions that are station codes rather than place names.
+ """
+ import pandas as pd
+
+ out = df.copy()
+ if "wspolrzedne" not in out.columns:
+ out["wspolrzedne"] = None
+ src = out["wspolrzedne"]
+ have = src.notna() & src.astype(str).str.strip().ne("") & ~overwrite
+ pew, dok, pow_, wsp = [], [], [], []
+ codes = out[station_code_column].fillna(False).astype(bool) \
+ if station_code_column and station_code_column in out.columns else None
+ for i, (idx, q) in enumerate(out[column].items()):
+ if bool(have.loc[idx]):
+ pew.append("ZRODLO"); dok.append("ZRODLO"); pow_.append(""); wsp.append(src.loc[idx])
+ continue
+ kw = {"is_station_code": bool(codes.loc[idx])} if codes is not None else {}
+ r = resolver.resolve(None if pd.isna(q) else str(q), **kw)
+ pew.append(r.pewnosc)
+ dok.append(getattr(r, "dokladnosc", "BRAK"))
+ has_pt = r.lat is not None and r.lon is not None
+ pow_.append(r.powod or ("" if has_pt else "brak_punktu"))
+ wsp.append(f"{r.lat:.6f},{r.lon:.6f}" if has_pt else None)
+ out["wspolrzedne"] = pd.Series(wsp, index=out.index, dtype=object)
+ out["_geo_pewnosc"] = pew
+ out["_geo_dokladnosc"] = dok
+ out["_geo_powod"] = pow_
+ return out
diff --git a/src/gridqueue/plots.py b/src/gridqueue/plots.py
new file mode 100644
index 0000000..b549401
--- /dev/null
+++ b/src/gridqueue/plots.py
@@ -0,0 +1,948 @@
+"""Charts drawn from the harmonised panel and from the results of the analyses.
+
+Every function takes the same objects the rest of the package returns (a frame in the
+21-field schema, an :class:`~gridqueue.longitudinal.EditionDelta`, a
+:class:`~gridqueue.longitudinal.LinkageAudit`, the frame returned by
+:func:`~gridqueue.analysis.node_loading`) and returns a :class:`matplotlib.figure.Figure`.
+Nothing is computed here that the analysis functions do not already compute: a chart is a
+view of a result, never a second implementation of it, so the numbers on a chart are the
+numbers in the report by construction.
+
+The same module draws the charts in the command line (``gridqueue plot``), in the Colab
+notebook and in the browser page, which loads this package into a WebAssembly interpreter.
+
+Design rules
+------------
+* **Figures are created without** ``pyplot``. A figure does not enter the global state of
+ the interpreter, so a loop over many publishers does not accumulate open windows, and the
+ module works unchanged on a server, in a notebook and in the browser. In Jupyter and
+ Colab a returned figure is displayed automatically; in a script call
+ ``fig.savefig(path)`` or :func:`save_figure`.
+* **What the register does not disclose is drawn as absence**, not as zero: a field a
+ publisher does not publish is a grey cell marked ``-``, a node with no declared free
+ capacity has an undefined loading and is drawn apart from the others.
+* ``matplotlib`` is an optional dependency (``pip install "gridqueue[plot]"``); the core of
+ the package does not need it.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Iterable, Optional, Sequence
+
+import pandas as pd
+
+from .analysis import kaplan_meier, kaplan_meier_by_group, processing_time
+from .layout import audit_column_shift
+from .longitudinal import STATUS_ORDER, TERMINAL_NEGATIVE
+from .quality import run_quality
+from .schema import STATUSY, schema_table
+
+__all__ = [
+ "PUBLISHER_SHORT", "PUBLISHER_COLORS", "STATUS_LABELS", "CLASS_LABELS",
+ "ACTIVE_STATUSES", "EVENT_LABELS", "publisher_label",
+ "plot_field_coverage", "plot_quality", "plot_column_shift",
+ "plot_status_distribution", "plot_capacity_by_class", "plot_processing_time",
+ "plot_node_loading", "plot_edition_flows", "plot_transitions", "plot_linkage_audit",
+ "plot_edition_comparison", "plot_location_resolution", "plot_overview",
+ "save_figure", "save_all",
+]
+
+#: Short publisher names used on axes. Unknown publishers are shown under their own name.
+PUBLISHER_SHORT: dict[str, str] = {
+ "TAURON Dystrybucja S.A.": "TAURON",
+ "ENERGA-OPERATOR S.A.": "ENERGA",
+ "Stoen Operator Sp. z o.o.": "Stoen",
+ "PSE S.A.": "PSE",
+ "National Grid Electricity Distribution": "National Grid",
+ "Boryszew Green Energy&Gas Sp. z o.o.": "Boryszew",
+ "e-Distribución Redes Digitales S.L.U.": "e-Distribución",
+ "ESB Networks DAC": "ESB Networks",
+ "E-REDES S.A.": "E-REDES",
+}
+
+#: Colours of the publishers. The first six are those of the article figures.
+PUBLISHER_COLORS: dict[str, str] = {
+ "TAURON": "#1b5e8c", "ENERGA": "#2e8b78", "Stoen": "#c77a1e", "PSE": "#7a4b9c",
+ "National Grid": "#6b7280", "Boryszew": "#a33b4e", "e-Distribución": "#d1495b",
+ "ESB Networks": "#e9b44c", "E-REDES": "#7fb7be",
+}
+_FALLBACK = ["#4e79a7", "#f28e2b", "#59a14f", "#b07aa1", "#9c755f", "#edc948", "#76b7b2"]
+
+#: English labels of the controlled status vocabulary.
+STATUS_LABELS: dict[str, str] = {
+ "WNIOSEK_ZLOZONY": "filed", "WNIOSEK_NIEKOMPLETNY": "incomplete",
+ "W_TRAKCIE_ANALIZY": "under review", "WARUNKI_WYDANE": "conditions issued",
+ "UMOWA_OBOWIAZUJACA": "agreement in force", "PRZYLACZONY": "connected",
+ "WARUNKI_WYGASLE": "conditions expired", "ODMOWA": "refused",
+ "UMOWA_ROZWIAZANA": "agreement terminated", "WNIOSEK_WYCOFANY": "withdrawn",
+ "NIEOKRESLONY": "unspecified",
+}
+
+#: English labels of the controlled resource-class vocabulary.
+CLASS_LABELS: dict[str, str] = {
+ "PV": "photovoltaics", "FW": "wind", "EW": "hydro", "BIOGAZ": "biogas", "GAZ": "gas",
+ "WEGIEL": "coal", "BESS": "energy storage", "ODB": "consumption", "MIX": "mixed",
+ "SIEC": "network", "INNE": "other", "NIEOKRESLONA": "unspecified",
+}
+
+#: Statuses of an application still in the queue (the default of ``node_loading``).
+ACTIVE_STATUSES: tuple[str, ...] = (
+ "WNIOSEK_ZLOZONY", "WNIOSEK_NIEKOMPLETNY", "W_TRAKCIE_ANALIZY", "WARUNKI_WYDANE",
+ "UMOWA_OBOWIAZUJACA")
+
+#: Readable names of the closing events of a proceeding, for chart titles.
+EVENT_LABELS: dict[str, str] = {
+ "data_warunkow": "issue of connection conditions", "data_umowy": "connection agreement",
+ "data_odmowy": "refusal", "data_energizacji": "energisation",
+ "data_wniosku": "application",
+}
+_KIERUNEK = {"wytworcza": "generation direction", "odbiorcza": "demand direction"}
+
+_GREY = "#6b7280"
+_INK = "#222222"
+_STATUS_COLORS = {
+ "WNIOSEK_ZLOZONY": "#c9dcea", "WNIOSEK_NIEKOMPLETNY": "#a3c4dc",
+ "W_TRAKCIE_ANALIZY": "#5b93b8", "WARUNKI_WYDANE": "#1b5e8c",
+ "UMOWA_OBOWIAZUJACA": "#2e8b78", "PRZYLACZONY": "#14532d",
+ "WARUNKI_WYGASLE": "#e0a96d", "ODMOWA": "#a33b4e", "UMOWA_ROZWIAZANA": "#c77a1e",
+ "WNIOSEK_WYCOFANY": "#d4b8a0", "NIEOKRESLONY": "#d1d5db",
+}
+
+
+# ---------------------------------------------------------------------------
+# helpers
+# ---------------------------------------------------------------------------
+
+def _mpl():
+ try:
+ import matplotlib # noqa: F401
+ from matplotlib.figure import Figure
+ except ImportError as e: # pragma: no cover - exercised only without matplotlib
+ raise ImportError(
+ "charts need matplotlib: pip install \"gridqueue[plot]\"") from e
+ return Figure
+
+
+def _figure(w: float, h: float, ax=None):
+ """Return (fig, ax): a new figure, or the figure of the axis supplied."""
+ if ax is not None:
+ return ax.figure, ax
+ Figure = _mpl()
+ fig = Figure(figsize=(w, h), dpi=110, layout="constrained")
+ return fig, fig.add_subplot(1, 1, 1)
+
+
+def _style(ax, grid_axis: str | None = "x") -> None:
+ for s in ("top", "right"):
+ ax.spines[s].set_visible(False)
+ for s in ("left", "bottom"):
+ ax.spines[s].set_color("#9ca3af")
+ ax.tick_params(colors="#374151", labelsize=7.5)
+ if grid_axis:
+ ax.grid(axis=grid_axis, color="#e5e7eb", lw=0.7, zorder=0)
+ ax.set_axisbelow(True)
+
+
+def _th(n) -> str:
+ return f"{int(round(float(n))):,}"
+
+
+def publisher_label(p) -> str:
+ """Short name of a publisher for an axis label."""
+ if p is None or (isinstance(p, float) and p != p) or str(p).strip() == "":
+ return "(publisher not set)"
+ return PUBLISHER_SHORT.get(str(p), str(p))
+
+
+def _color(short: str, i: int) -> str:
+ return PUBLISHER_COLORS.get(short, _FALLBACK[i % len(_FALLBACK)])
+
+
+def _groups(df: pd.DataFrame, by: Optional[str]) -> list[tuple[str, pd.DataFrame]]:
+ """Split by publisher (or another column), largest group first."""
+ if by is None or by not in df.columns or df[by].isna().all():
+ return [("panel", df)]
+ keys = df[by].fillna("").astype(str)
+ order = keys.value_counts().index.tolist()
+ return [(publisher_label(k) if by == "publikujacy" else (k or "(empty)"),
+ df[keys == k]) for k in order]
+
+
+def _filled(s: pd.Series) -> pd.Series:
+ txt = s.astype(str).str.strip()
+ return s.notna() & txt.ne("") & txt.str.lower().ne("nan") & txt.ne("")
+
+
+def _empty_note(ax, text: str) -> None:
+ ax.text(0.5, 0.5, text, ha="center", va="center", fontsize=8, color=_GREY,
+ transform=ax.transAxes, wrap=True)
+ ax.set_xticks([]); ax.set_yticks([])
+ for s in ax.spines.values():
+ s.set_visible(False)
+
+
+def _group_key(df: pd.DataFrame, by) -> pd.Series:
+ """The same group key as :func:`~gridqueue.analysis.kaplan_meier_by_group`."""
+ kol = [by] if isinstance(by, str) else list(by)
+ return df[kol].astype("string").fillna("(empty)").agg(" / ".join, axis=1)
+
+
+def _applications(df: pd.DataFrame) -> pd.DataFrame:
+ if "_encja" in df.columns:
+ return df[df["_encja"].fillna("WNIOSEK").astype(str).str.upper() != "WEZEL"]
+ return df
+
+
+# ---------------------------------------------------------------------------
+# what a disclosure contains
+# ---------------------------------------------------------------------------
+
+def plot_field_coverage(df: pd.DataFrame, by: str = "publikujacy", ax=None,
+ annotate: bool = True):
+ """Heat map: share of rows in which each of the 21 schema fields is populated.
+
+ One column per publisher (or per value of ``by``), with the row count under its name.
+ A field the publisher does not disclose at all is a grey ``-`` cell, distinct from a
+ sparsely populated one. Core fields are marked with a bullet.
+ """
+ import numpy as np
+ from matplotlib.colors import LinearSegmentedColormap
+
+ st = schema_table()
+ pola = st["pole"].tolist()
+ rdzen = dict(zip(st["pole"], st["rdzen"].astype(bool)))
+ grupy = _groups(df, by)
+ Z = np.zeros((len(pola), len(grupy)))
+ for j, (_, sub) in enumerate(grupy):
+ for i, f in enumerate(pola):
+ Z[i, j] = float(_filled(sub[f]).mean()) if f in sub.columns and len(sub) else 0.0
+
+ fig, ax = _figure(max(3.2, 1.6 + 0.62 * len(grupy)), 5.6, ax)
+ cmap = LinearSegmentedColormap.from_list("cov", ["#f7f7f7", "#c9dcea", "#5b93b8", "#1b5e8c"])
+ im = ax.imshow(Z, cmap=cmap, vmin=0, vmax=1, aspect="auto")
+ ax.set_xticks(range(len(grupy)))
+ ax.set_xticklabels([f"{k}\n{_th(len(s))}" for k, s in grupy], fontsize=6.5)
+ ax.set_yticks(range(len(pola)))
+ ax.set_yticklabels([("\u2022 " if rdzen[f] else "") + f for f in pola], fontsize=6.5)
+ for i, f in enumerate(pola):
+ if rdzen[f]:
+ ax.get_yticklabels()[i].set_fontweight("bold")
+ if annotate:
+ for i in range(len(pola)):
+ for j in range(len(grupy)):
+ v = Z[i, j]
+ txt = "-" if v == 0 else ("<1" if v < 0.005 else
+ ("100" if v >= 0.995 else f"{v * 100:.0f}"))
+ ax.text(j, i, txt, ha="center", va="center", fontsize=5.6,
+ color="#ffffff" if v > 0.62 else ("#9ca3af" if v == 0 else _INK))
+ ax.set_xticks(np.arange(-.5, len(grupy), 1), minor=True)
+ ax.set_yticks(np.arange(-.5, len(pola), 1), minor=True)
+ ax.grid(which="minor", color="#ffffff", lw=0.8)
+ ax.tick_params(which="both", length=0)
+ for s in ax.spines.values():
+ s.set_visible(False)
+ ax.xaxis.tick_top()
+ n_pokr = int((Z.max(axis=1) > 0).sum())
+ ax.set_title(f"Schema field coverage, % of rows: {n_pokr} of {len(pola)} fields populated"
+ "\n\u2022 core field - not disclosed", fontsize=8, loc="left", pad=6)
+ cb = fig.colorbar(im, ax=ax, shrink=0.35, pad=0.02, aspect=14)
+ cb.ax.tick_params(labelsize=6)
+ cb.set_ticks([0, 0.5, 1]); cb.set_ticklabels(["0", "50", "100 %"])
+ cb.outline.set_visible(False)
+ return fig
+
+
+def plot_quality(df: pd.DataFrame, by: str = "publikujacy", ax=None):
+ """Violations of the eight quality rules, stacked by publisher.
+
+ The rules are evaluated per publisher with :func:`~gridqueue.quality.run_quality`, as
+ in the article, so a rule that depends on the set of fields a publisher discloses is
+ judged against that publisher only. A rule with no violation is drawn as a dot.
+ """
+ import numpy as np
+
+ rows = []
+ for k, sub in _groups(df, by):
+ for x in run_quality(sub.reset_index(drop=True)).as_dict()["reguly"]:
+ rows.append({"kod": x["kod"], "nazwa": x["nazwa"], "pub": k,
+ "naruszenia": max(int(x["naruszenia"]), 0)})
+ jak = pd.DataFrame(rows)
+ piv = (jak.pivot_table(index=["kod", "nazwa"], columns="pub", values="naruszenia",
+ aggfunc="sum", sort=False).fillna(0).astype(int))
+ kol = [k for k, _ in _groups(df, by)]
+ piv = piv[[k for k in kol if k in piv.columns]]
+ tot = piv.sum(axis=1)
+ piv = piv.loc[tot.sort_values(kind="stable").index]
+ tot = tot.loc[piv.index]
+ n = len(df)
+
+ fig, ax = _figure(6.8, 3.0, ax)
+ y = np.arange(len(piv))
+ left = np.zeros(len(piv))
+ for i, k in enumerate(piv.columns):
+ v = piv[k].values.astype(float)
+ if v.sum() == 0:
+ continue
+ ax.barh(y, v, left=left, height=0.62, color=_color(k, i), label=k, zorder=2)
+ left += v
+ szer = max(int(tot.max()) if len(tot) else 0, 1)
+ for yy, t in zip(y, tot.values):
+ if t == 0:
+ ax.plot([0], [yy], "o", ms=3.2, color="#444444", zorder=3, clip_on=False)
+ ax.text(szer * 0.015, yy, "0 violations", va="center", fontsize=6.5, color=_GREY)
+ else:
+ ax.text(t + szer * 0.012, yy, f"{_th(t)} ({t / max(n, 1) * 100:.3f} %)",
+ va="center", fontsize=6.5)
+ ax.set_yticks(y)
+ ax.set_yticklabels([f"{k} {nm}" for k, nm in piv.index], fontsize=7)
+ ax.set_xlim(0, szer * 1.45)
+ from matplotlib.ticker import MaxNLocator
+ ax.xaxis.set_major_locator(MaxNLocator(integer=True))
+ ax.set_xlabel(f"rows violating the rule (of {_th(n)} rows)", fontsize=7.5)
+ ax.set_title("Quality rules: a count of disclosure defects, not software errors",
+ fontsize=8.5, loc="left")
+ if len(piv.columns) > 1 and left.sum() > 0:
+ ax.legend(loc="lower right", fontsize=6.5, ncol=2, frameon=False,
+ handlelength=1.1, columnspacing=1.0)
+ _style(ax)
+ ax.margins(y=0.06)
+ return fig
+
+
+def plot_column_shift(df: pd.DataFrame, ax=None):
+ """Rows in which a value stands in a field it cannot belong to, by field.
+
+ A view of :func:`~gridqueue.layout.audit_column_shift`. An empty chart (0 %) is the
+ expected result of a correct extraction.
+ """
+ a = audit_column_shift(df)
+ d = a.as_dict()
+ per = pd.Series(d.get("per_pole", {}), dtype=float).sort_values()
+ fig, ax = _figure(5.6, max(1.6, 0.9 + 0.28 * max(len(per), 1)), ax)
+ title = (f"Column shift audit: {_th(d['n_przesunietych'])} of {_th(d['n_wierszy'])} rows "
+ f"({d['udzial'] * 100:.3f} %)")
+ if not len(per):
+ _empty_note(ax, "no value stands in a field it cannot belong to")
+ ax.set_title(title, fontsize=8.5, loc="left")
+ return fig
+ ax.barh(range(len(per)), per.values, color="#c77a1e", height=0.6, zorder=2)
+ for i, v in enumerate(per.values):
+ ax.text(v, i, f" {_th(v)}", va="center", fontsize=6.5)
+ ax.set_yticks(range(len(per)))
+ ax.set_yticklabels(per.index, fontsize=7)
+ ax.set_xlabel("rows with a type-inconsistent value", fontsize=7.5)
+ ax.set_xlim(0, per.max() * 1.25)
+ ax.set_title(title, fontsize=8.5, loc="left")
+ _style(ax)
+ return fig
+
+
+# ---------------------------------------------------------------------------
+# the state of the queue
+# ---------------------------------------------------------------------------
+
+def _status_order(values: Iterable[str]) -> list[str]:
+ ordered = [s for s in STATUSY if s in set(values)]
+ ordered.sort(key=lambda s: (s in TERMINAL_NEGATIVE or s == "UMOWA_ROZWIAZANA",
+ STATUS_ORDER.get(s, 99), s))
+ rest = sorted(set(values) - set(ordered))
+ return ordered + rest
+
+
+def plot_status_distribution(df: pd.DataFrame, by: str = "publikujacy",
+ normalize: bool = False, codes: bool = False, ax=None):
+ """Applications by stage of the proceeding, one bar per publisher.
+
+ Stages are ordered as the proceeding runs (filed, under review, conditions issued,
+ agreement, connected), followed by the states that end it without a connection.
+ ``normalize=True`` shows shares instead of counts, which makes publishers of very
+ different size comparable.
+ """
+ import numpy as np
+
+ app = _applications(df)
+ if "status_procesu" not in app.columns or not _filled(app["status_procesu"]).any():
+ fig, ax = _figure(5.5, 2.0, ax)
+ _empty_note(ax, "the disclosure carries no process status")
+ return fig
+ grupy = [(k, s) for k, s in _groups(app, by) if len(s)]
+ st = _status_order(app["status_procesu"].dropna().astype(str).unique())
+ M = pd.DataFrame({k: s["status_procesu"].astype(str).value_counts() for k, s in grupy}
+ ).reindex(st).fillna(0).T
+ if normalize:
+ M = M.div(M.sum(axis=1).replace(0, 1), axis=0) * 100
+ fig, ax = _figure(6.8, max(1.8, 1.2 + 0.36 * len(grupy)), ax)
+ y = np.arange(len(M))[::-1]
+ left = np.zeros(len(M))
+ for s in st:
+ v = M[s].values.astype(float)
+ ax.barh(y, v, left=left, height=0.62, zorder=2, label=s if codes else
+ STATUS_LABELS.get(s, s), color=_STATUS_COLORS.get(s, "#d1d5db"))
+ left += v
+ for yy, t, (k, sub) in zip(y, left, grupy):
+ ax.text(t, yy, f" {_th(len(sub))}" if not normalize else f" n={_th(len(sub))}",
+ va="center", fontsize=6.5)
+ ax.set_yticks(y)
+ ax.set_yticklabels([k for k, _ in grupy], fontsize=7)
+ ax.set_xlim(0, left.max() * 1.12 if left.max() else 1)
+ ax.set_xlabel("share of applications, %" if normalize else "applications", fontsize=7.5)
+ ax.set_title("Applications by stage of the proceeding", fontsize=8.5, loc="left")
+ ax.legend(loc="center left", bbox_to_anchor=(1.0, 0.5), ncol=1 if len(st) <= 6 else 2,
+ fontsize=6.2, frameon=False, handlelength=1.0, columnspacing=0.8,
+ borderaxespad=0.3)
+ _style(ax)
+ return fig
+
+
+def plot_capacity_by_class(df: pd.DataFrame, power: str = "auto",
+ by: str = "klasa_zasobu", hue: Optional[str] = "poziom_napiecia",
+ statuses: Optional[Sequence[str]] | str = "active",
+ codes: bool = False, ax=None):
+ """Requested capacity in MW by resource class, split by voltage level.
+
+ ``power="auto"`` adds injected capacity for generating classes and consumed capacity for
+ consumption, which is how the schema core treats the capacity alternative; give a field
+ name to use one field only. ``statuses="active"`` keeps applications still in the
+ queue (the default of ``node_loading``); ``None`` keeps all rows.
+ """
+ import numpy as np
+
+ app = _applications(df).copy()
+ if statuses == "active":
+ statuses = ACTIVE_STATUSES
+ if statuses is not None and "status_procesu" in app.columns:
+ app = app[app["status_procesu"].isin(set(statuses))]
+ if by not in app.columns:
+ fig, ax = _figure(5.5, 2.0, ax)
+ _empty_note(ax, f"the disclosure carries no field {by}")
+ return fig
+ if power == "auto":
+ w = pd.to_numeric(app.get("moc_wprowadzana"), errors="coerce") \
+ if "moc_wprowadzana" in app else pd.Series(np.nan, index=app.index)
+ p = pd.to_numeric(app.get("moc_pobierana"), errors="coerce") \
+ if "moc_pobierana" in app else pd.Series(np.nan, index=app.index)
+ app["_mw"] = w.where(w.notna() & (w > 0), p) / 1000.0
+ opis = "requested capacity (injected, or consumed where no injection is declared)"
+ else:
+ app["_mw"] = pd.to_numeric(app.get(power), errors="coerce") / 1000.0
+ opis = power
+ app = app[app["_mw"].notna() & (app["_mw"] > 0)]
+ fig, ax = _figure(6.6, 3.0, ax)
+ if not len(app):
+ _empty_note(ax, "no row with a disclosed capacity in the selected statuses")
+ return fig
+ h = hue if hue and hue in app.columns else None
+ app["_k"] = app[by].astype(object).where(app[by].notna(), "(empty)").astype(str)
+ app["_h"] = app[h].astype(object).where(app[h].notna(), "(not disclosed)").astype(str) \
+ if h else "all"
+ if h:
+ # At most five levels by capacity, the rest pooled: a multi-voltage publisher writes
+ # combinations such as "110; 220 kV", and a legend of ten shades is not readable.
+ top_h = app.groupby("_h")["_mw"].sum().sort_values(ascending=False).index[:5]
+ app.loc[~app["_h"].isin(top_h), "_h"] = "other levels"
+ P = app.pivot_table(index="_k", columns="_h", values="_mw", aggfunc="sum").fillna(0)
+ P = P.loc[P.sum(axis=1).sort_values().index]
+ N = app.groupby("_k").size().reindex(P.index)
+ y = np.arange(len(P))
+ left = np.zeros(len(P))
+ pal = ["#1b5e8c", "#8fb4cc", "#2e8b78", "#c77a1e", "#7a4b9c", "#9ca3af"]
+ for i, c in enumerate(P.columns):
+ ax.barh(y, P[c].values, left=left, height=0.62, color=pal[i % len(pal)],
+ label=c, zorder=2)
+ left += P[c].values
+ for yy, t, n in zip(y, left, N.values):
+ ax.text(t, yy, f" {t:,.0f} MW, {_th(n)} appl.", va="center", fontsize=6.3)
+ lab = (lambda k: k) if codes or by != "klasa_zasobu" else \
+ (lambda k: CLASS_LABELS.get(k, k))
+ ax.set_yticks(y)
+ ax.set_yticklabels([lab(k) for k in P.index], fontsize=7)
+ ax.set_xlim(0, left.max() * 1.42)
+ ax.set_xlabel(f"MW, {opis}", fontsize=7)
+ which = "applications in the queue" if statuses is not None else "all applications"
+ ax.set_title(f"Requested capacity by {'resource class' if by == 'klasa_zasobu' else by}, "
+ f"{which}", fontsize=8.5, loc="left")
+ if h:
+ ax.legend(title=h, title_fontsize=6.5, fontsize=6.3, frameon=False,
+ loc="center left", bbox_to_anchor=(1.0, 0.5), borderaxespad=0.3)
+ _style(ax)
+ return fig
+
+
+def plot_processing_time(df: pd.DataFrame, by: Optional[str] = "klasa_zasobu",
+ od: str = "data_wniosku", do: str = "data_warunkow",
+ min_n: int = 30, max_groups: int = 6, ci: bool = True,
+ codes: bool = False, ax=None):
+ """Kaplan-Meier curves of the time from application to the event ``do``.
+
+ Applications without the closing date are **censored** at the as-at date of the
+ edition, not dropped, exactly as :func:`~gridqueue.analysis.processing_time` and
+ :func:`~gridqueue.analysis.kaplan_meier` compute them. Only groups with at least
+ ``min_n`` measurable applications are drawn (the inclusion criterion of the article);
+ the legend gives the group size, the share censored and the median with its interval.
+ """
+ tab = kaplan_meier_by_group(_applications(df), by=by, od=od, do=do, min_n=min_n)
+ fig, ax = _figure(6.4, 3.6, ax)
+ if not len(tab):
+ _empty_note(ax, f"no group with at least {min_n} applications carrying {od}")
+ ax.set_title("Time to " + EVENT_LABELS.get(do, do), fontsize=8.5, loc="left")
+ return fig
+ tab = tab.head(max_groups)
+ app = _applications(df)
+ cov = min(float(_filled(app[c]).mean()) for c in (od, do))
+ pt = processing_time(app, od=od, do=do)
+ pal = ["#1b5e8c", "#c77a1e", "#2e8b78", "#a33b4e", "#7a4b9c", "#6b7280", "#e9b44c"]
+ xmax = 0.0
+ for i, r in enumerate(tab.itertuples(index=False)):
+ sub = pt if by is None else pt[_group_key(pt, by) == r.grupa]
+ km = kaplan_meier(sub["czas_dni"].astype("float64"), sub["zdarzenie"].astype("float64"))
+ t = [0.0] + km.tabela["t"].tolist()
+ S = [1.0] + km.tabela["S"].tolist()
+ tmax = float(pd.to_numeric(sub["czas_dni"], errors="coerce").max())
+ t.append(tmax); S.append(S[-1])
+ below = [tt for tt, ss in zip(t, S) if ss <= 0.01]
+ xmax = max(xmax, below[0] if below else tmax)
+ c = pal[i % len(pal)]
+ name = r.grupa if (codes or by != "klasa_zasobu") else \
+ f"{CLASS_LABELS.get(r.grupa, r.grupa)} ({r.grupa})"
+ med = "not reached" if pd.isna(r.mediana) else f"{r.mediana:.0f} d"
+ ax.step(t, [s * 100 for s in S], where="post", color=c, lw=1.4, zorder=3,
+ label=f"{name}: n={_th(r.n)}, censored {r.udzial_cenzurowanych * 100:.1f} %, "
+ f"median {med}")
+ if ci and len(km.tabela):
+ lo = [100.0] + (km.tabela["S_dolne"] * 100).tolist() + [km.tabela["S_dolne"].iloc[-1] * 100]
+ hi = [100.0] + (km.tabela["S_gorne"] * 100).tolist() + [km.tabela["S_gorne"].iloc[-1] * 100]
+ ax.fill_between(t, lo, hi, step="post", color=c, alpha=0.13, lw=0, zorder=2)
+ if not pd.isna(r.mediana):
+ ax.plot([r.mediana], [50], "o", ms=3.5, color=c, zorder=4)
+ ax.axhline(50, color="#9ca3af", lw=0.8, ls="--", zorder=1)
+ ax.set_ylim(0, 102)
+ ax.set_xlim(0, xmax * 1.05 if xmax else 1)
+ ax.set_ylabel("% still waiting", fontsize=7.5)
+ ax.set_xlabel(f"days from {od} to {do} (axis ends where the curves fall below 1 % or end)",
+ fontsize=7.5)
+ ax.set_title(f"Time from {EVENT_LABELS.get(od, od)} to {EVENT_LABELS.get(do, do)}: "
+ "Kaplan-Meier, censored at the as-at date, 95 % band"
+ f"\nboth dates populated in {cov * 100:.1f} % of applications"
+ + ("" if cov >= 0.8 else ", below the 80 % inclusion criterion of the article"),
+ fontsize=8.5, loc="left")
+ ax.legend(fontsize=6.3, frameon=False, loc="upper right")
+ _style(ax, "both")
+ return fig
+
+
+def plot_node_loading(loading: pd.DataFrame, ax=None, label: str = "nazwa_wezla",
+ top: int = 12, title: Optional[str] = None):
+ """Node loading: requested capacity of the active queue over declared free capacity.
+
+ Takes the frame returned by :func:`~gridqueue.analysis.node_loading`. Bars are on a
+ logarithmic axis with the line at 1 (queue equal to the declared capacity). Nodes with
+ zero declared free capacity have an undefined ratio; they are counted in the title and
+ not drawn, rather than drawn as infinite.
+ """
+ import numpy as np
+
+ d = loading.copy()
+ zero = int(d.get("moc_dostepna_zero", pd.Series(dtype=bool)).fillna(False).sum())
+ d = d[pd.to_numeric(d["obciazenie"], errors="coerce").notna()].copy()
+ d["obciazenie"] = pd.to_numeric(d["obciazenie"], errors="coerce").astype(float)
+ fig, ax = _figure(6.4, 3.4, ax)
+ if not len(d):
+ _empty_note(ax, "no node with a defined loading")
+ return fig
+ d = d.sort_values("obciazenie", ascending=False).reset_index(drop=True)
+ nonzero = d["obciazenie"] > 0
+ n_zero_queue = int((~nonzero).sum())
+ floor = float(d.loc[nonzero, "obciazenie"].min()) * 0.4 if nonzero.any() else 0.1
+ pos = d["obciazenie"].where(nonzero, np.nan)
+ cols = np.where(d["obciazenie"] > 1, "#a33b4e", "#2e8b78")
+ ax.bar(np.arange(len(d))[nonzero.values], pos[nonzero].values, color=cols[nonzero.values],
+ width=0.8, zorder=2)
+ if n_zero_queue:
+ # loading exactly 0 (free capacity declared, no active application assigned) cannot be
+ # drawn on a log axis; it is marked at the floor instead of being given a fake height
+ ax.plot(np.arange(len(d))[~nonzero.values], [floor * 1.15] * n_zero_queue, marker="v",
+ ls="none", ms=4, color=_GREY, zorder=3,
+ label=f"\u03bb = 0: no active application assigned ({n_zero_queue})")
+ ax.set_yscale("log")
+ ax.axhline(1, color=_INK, lw=0.9, ls="--", zorder=3,
+ label="\u03bb = 1: queue equal to the declared free capacity")
+ ax.legend(loc="upper right", fontsize=6.5, frameon=False)
+ if label in d.columns and top:
+ for i in range(min(top, len(d))):
+ if not nonzero.iloc[i]:
+ continue
+ ax.text(i, pos.iloc[i] * 1.12, str(d.loc[i, label])[:14], rotation=90,
+ ha="center", va="bottom", fontsize=5.4, color="#374151")
+ ax.set_xticks([])
+ ax.set_xlim(-0.8, len(d) - 0.2)
+ ax.set_ylim(min(floor, 0.5), max(float(np.nanmax(pos.values)) if nonzero.any() else 1.0, 2.0) * 8)
+ ax.set_xlabel(f"node-directions, sorted ({_th(len(d))} with declared free capacity)",
+ fontsize=7.5)
+ ax.set_ylabel("loading \u03bb(n), log scale", fontsize=7.5)
+ over = int((d["obciazenie"] > 1).sum())
+ kier = d["kierunek"].dropna().iloc[0] if "kierunek" in d and d["kierunek"].notna().any() else None
+ ax.set_title(title or (
+ f"Node loading{f', {_KIERUNEK.get(kier, kier)}' if kier else ''}: median {d['obciazenie'].median():.3f}, "
+ f"{over} of {len(d)} above 1; {zero} {'node' if zero == 1 else 'nodes'} with zero free capacity not drawn"),
+ fontsize=8.5, loc="left")
+ _style(ax, "y")
+ return fig
+
+
+# ---------------------------------------------------------------------------
+# two editions
+# ---------------------------------------------------------------------------
+
+def plot_edition_flows(delta, ax=None):
+ """Flows between two editions: shared, absent from the later one, new in it.
+
+ The two hatched segments are the warning signs of a non-monotone register: rows that
+ disappear while still active, and rows that appear already closed.
+ """
+ d = delta.as_dict()
+ fig, ax = _figure(6.4, 2.4, ax)
+ ea, eb = d["edycja_a"], d["edycja_b"]
+ shared, gone, new = d["wspolnych"], d["ubylych"], d["nowych"]
+ gone_act, new_closed = d["ubyle_czynne"], d["nowe_zamkniete"]
+ rej_a = d["odrzuconych_jako_niejednoznaczne_a"]
+ rej_b = d["odrzuconych_jako_niejednoznaczne_b"]
+ y = [1, 0]
+ ax.barh(1, shared, color="#1b5e8c", height=0.55, zorder=2, label="shared (linked)")
+ ax.barh(1, gone - gone_act, left=shared, color="#c9dcea", height=0.55, zorder=2,
+ label="absent from the later edition")
+ ax.barh(1, gone_act, left=shared + gone - gone_act, color="#c9dcea", hatch="////",
+ edgecolor="#a33b4e", lw=0, height=0.55, zorder=2,
+ label="absent while still active")
+ ax.barh(1, rej_a, left=shared + gone, color="#e5e7eb", height=0.55, zorder=2,
+ label="key collision, not linked")
+ ax.barh(0, shared, color="#1b5e8c", height=0.55, zorder=2)
+ ax.barh(0, new - new_closed, left=shared, color="#e9c46a", height=0.55, zorder=2,
+ label="new in the later edition")
+ ax.barh(0, new_closed, left=shared + new - new_closed, color="#e9c46a", hatch="////",
+ edgecolor="#a33b4e", lw=0, height=0.55, zorder=2, label="new, already closed")
+ ax.barh(0, rej_b, left=shared + new, color="#e5e7eb", height=0.55, zorder=2)
+ for yy, tot, n in ((1, shared + gone + rej_a, d["wierszy_a"]),
+ (0, shared + new + rej_b, d["wierszy_b"])):
+ ax.text(tot, yy, f" {_th(n)} rows", va="center", fontsize=6.8)
+ ax.text(shared / 2, 0.5, f"{_th(shared)} linked", ha="center", va="center",
+ fontsize=6.8, color="#1b5e8c")
+ ax.set_yticks(y)
+ ax.set_yticklabels([str(ea), str(eb)], fontsize=7.5)
+ ax.set_xlim(0, max(d["wierszy_a"], d["wierszy_b"]) * 1.18)
+ ax.set_xlabel("rows", fontsize=7.5)
+ mono = "monotone" if d["migawka_monotoniczna"] else "NOT a monotone snapshot"
+ ax.set_title(f"Flows between editions: {_th(new)} new, {_th(gone)} absent; register is {mono}",
+ fontsize=8.5, loc="left")
+ ax.legend(loc="upper center", bbox_to_anchor=(0.5, -0.32), ncol=3, fontsize=6.2,
+ frameon=False, handlelength=1.2)
+ _style(ax)
+ return fig
+
+
+def plot_transitions(delta, ax=None, codes: bool = False):
+ """Status transition matrix among linked rows.
+
+ Off-diagonal cells are coloured by direction: forward in the order of the proceeding
+ (or to a terminal negative state) in green, backward in red. A high forward share is the
+ independent test of the linkage described in the article.
+ """
+ import numpy as np
+ from matplotlib.colors import LinearSegmentedColormap
+
+ tr = delta.przejscia
+ fig, ax = _figure(5.0, 4.4, ax)
+ if tr is None or not len(tr):
+ _empty_note(ax, "no linked rows")
+ return fig
+ st = _status_order(set(tr.index) | set(tr.columns))
+ M = tr.reindex(index=st, columns=st).fillna(0).astype(int)
+ off = M.values.copy()
+ np.fill_diagonal(off, 0)
+ vmax = max(int(off.max()), 1)
+ cmap = LinearSegmentedColormap.from_list("t", ["#ffffff", "#c9dcea", "#1b5e8c"])
+ ax.imshow(np.where(off > 0, off, 0), cmap=cmap, vmin=0, vmax=vmax, aspect="equal")
+ for i, a in enumerate(st):
+ for j, b in enumerate(st):
+ n = int(M.iloc[i, j])
+ if not n:
+ continue
+ if i == j:
+ ax.text(j, i, _th(n), ha="center", va="center", fontsize=6, color=_GREY)
+ continue
+ fwd = (a in STATUS_ORDER and b in STATUS_ORDER and STATUS_ORDER[b] > STATUS_ORDER[a]) \
+ or (a in STATUS_ORDER and b in TERMINAL_NEGATIVE)
+ back = a in STATUS_ORDER and b in STATUS_ORDER and STATUS_ORDER[b] < STATUS_ORDER[a]
+ col = "#14532d" if fwd else ("#a33b4e" if back else _INK)
+ ax.text(j, i, _th(n), ha="center", va="center", fontsize=7, fontweight="bold",
+ color="#ffffff" if n > vmax * 0.6 and not back else col)
+ if back:
+ ax.add_patch(__import__("matplotlib").patches.Rectangle(
+ (j - .5, i - .5), 1, 1, fill=False, ec="#a33b4e", lw=1.2))
+ lab = (lambda s: s) if codes else (lambda s: STATUS_LABELS.get(s, s))
+ ax.set_xticks(range(len(st)))
+ ax.set_xticklabels([lab(s) for s in st], rotation=40, ha="right", fontsize=6.5)
+ ax.set_yticks(range(len(st)))
+ ax.set_yticklabels([lab(s) for s in st], fontsize=6.5)
+ ax.set_xlabel(f"status in {delta.edycja_b}", fontsize=7.5)
+ ax.set_ylabel(f"status in {delta.edycja_a}", fontsize=7.5)
+ ax.tick_params(length=0)
+ sh = delta.udzial_przejsc_w_przod
+ ax.set_title("Status transitions among linked rows"
+ + (f": {sh * 100:.2f} % forward" if sh is not None else ""),
+ fontsize=8.5, loc="left")
+ for s in ax.spines.values():
+ s.set_color("#e5e7eb")
+ return fig
+
+
+def plot_linkage_audit(audit, recall=None, ax=None):
+ """Confirmed, uncertain and false pairs on the fields excluded from the key.
+
+ With a :class:`~gridqueue.longitudinal.LinkageRecall` the pairs the key missed are
+ added as a fourth segment, so precision and sensitivity are read from one bar.
+ """
+ a = audit.as_dict()
+ fig, ax = _figure(6.4, 1.7, ax)
+ parts = [("confirmed", a["potwierdzonych"], "#2e8b78"),
+ ("uncertain", a["niepewnych"], "#e9c46a"),
+ ("false", a["falszywych"], "#a33b4e")]
+ if recall is not None:
+ parts.append(("missed by the key, recovered", recall.as_dict()["odzyskanych"], "#9ca3af"))
+ left = 0
+ for name, v, c in parts:
+ ax.barh(0, v, left=left, color=c, height=0.5, label=f"{name} ({_th(v)})", zorder=2)
+ left += v
+ ax.set_yticks([])
+ ax.set_xlim(0, max(left, 1) * 1.02)
+ ax.set_xlabel("linked pairs", fontsize=7.5)
+ ttl = (f"Linkage precision {a['precyzja_dolna'] * 100:.2f} to "
+ f"{a['precyzja_gorna'] * 100:.2f} % over {_th(a['par'])} pairs")
+ if recall is not None and recall.as_dict()["czulosc_klucza"] == recall.as_dict()["czulosc_klucza"]:
+ ttl += f"; key sensitivity {recall.as_dict()['czulosc_klucza']:.4f}"
+ ax.set_title(ttl, fontsize=8.5, loc="left")
+ ax.legend(loc="upper center", bbox_to_anchor=(0.5, -0.55), ncol=4, fontsize=6.3,
+ frameon=False, handlelength=1.1)
+ _style(ax)
+ return fig
+
+
+def plot_edition_comparison(delta, audit=None, recall=None):
+ """Flows, transition matrix and linkage audit of two editions in one figure."""
+ Figure = _mpl()
+ fig = Figure(figsize=(10.5, 5.2), dpi=110, layout="constrained")
+ gs = fig.add_gridspec(2, 2, width_ratios=[1.45, 1.0], height_ratios=[1.2, 0.8])
+ plot_edition_flows(delta, ax=fig.add_subplot(gs[0, 0]))
+ if audit is not None:
+ plot_linkage_audit(audit, recall, ax=fig.add_subplot(gs[1, 0]))
+ plot_transitions(delta, ax=fig.add_subplot(gs[:, 1]))
+ return fig
+
+
+# ---------------------------------------------------------------------------
+# locations
+# ---------------------------------------------------------------------------
+
+_GEO_KAT = {
+ "source coordinates": "#14532d", "placed by the gazetteer": "#2e8b78",
+ "locality centroid only": "#7fb7be", "name resolved, no geometry": "#b9d7c9",
+ "station code with no dictionary": "#1b5e8c", "no locality declared": "#8fb4cc",
+ "empty description": "#cfd8dd", "locality outside the gazetteer": "#c77a1e",
+ "several candidates": "#7a4b9c", "no candidate in the gazetteer": "#c9a0dc",
+ "not resolved, other reason": "#aaaaaa",
+ "resolver not run": "#e5e7eb",
+}
+_WITH_POINT = ("source coordinates", "placed by the gazetteer", "locality centroid only")
+_PEWNOSC = {"EXACT": "exact", "UNIQUE_TOKEN": "unique member", "SKRZYZOWANIE": "intersection",
+ "REVERSED": "reversed order", "TOKEN_SUBSET": "subset of members"}
+
+
+def _geo_category(wsp, pewnosc, powod, dokladnosc=None) -> str:
+ p = "" if pewnosc is None or (isinstance(pewnosc, float) and pewnosc != pewnosc) \
+ else str(pewnosc)
+ if _filled(pd.Series([wsp])).iloc[0]:
+ if not p or p == "ZRODLO":
+ return "source coordinates"
+ if p == "NONE" or str(dokladnosc) == "CENTROID_MIEJSCOWOSCI":
+ return "locality centroid only"
+ return "placed by the gazetteer"
+ s = "" if powod is None or (isinstance(powod, float) and powod != powod) else str(powod)
+ if not s and not p:
+ return "resolver not run"
+ if "bez_geometrii" in s or (p and p != "NONE"):
+ return "name resolved, no geometry"
+ if "kod_stacji" in s:
+ return "station code with no dictionary"
+ if "brak_deklaracji_miejscowosci" in s:
+ return "no locality declared"
+ if "pusty_opis" in s:
+ return "empty description"
+ if "poza_gazeterem" in s:
+ return "locality outside the gazetteer"
+ if "wiele_kandydatow" in s or "niejednoznaczna" in s:
+ return "several candidates"
+ if "brak_kandydatow" in s or "brak_tokenow" in s or "opis_slowny" in s:
+ return "no candidate in the gazetteer"
+ return "not resolved, other reason"
+
+
+def plot_location_resolution(df: pd.DataFrame, by: str = "publikujacy"):
+ """Where a coordinate comes from, and where it does not, by publisher.
+
+ Left: share of rows with source coordinates, coordinates placed by the gazetteer, and
+ each stated reason for absence. Right: confidence levels of the gazetteer placements.
+ Reads the resolver diagnostics ``_geo_pewnosc`` and ``_geo_powod`` when present.
+ """
+ import numpy as np
+
+ Figure = _mpl()
+ d = df.copy()
+ wsp = d["wspolrzedne"] if "wspolrzedne" in d else pd.Series(None, index=d.index)
+ pew = d["_geo_pewnosc"] if "_geo_pewnosc" in d else pd.Series(None, index=d.index)
+ pow_ = d["_geo_powod"] if "_geo_powod" in d else pd.Series(None, index=d.index)
+ dok = d["_geo_dokladnosc"] if "_geo_dokladnosc" in d else pd.Series(None, index=d.index)
+ d["_kat"] = [_geo_category(w, p, r, k) for w, p, r, k in zip(wsp, pew, pow_, dok)]
+ grupy = _groups(d, by)
+ kat = [k for k in _GEO_KAT if (d["_kat"] == k).any()]
+ CT = pd.DataFrame({k: s["_kat"].value_counts() for k, s in grupy}).reindex(kat).fillna(0).T
+ placed = d.loc[d["_kat"].isin(["placed by the gazetteer", "name resolved, no geometry"]),
+ "_geo_pewnosc"] if "_geo_pewnosc" in d else pd.Series(dtype=object)
+ two = len(placed) > 0
+ fig = Figure(figsize=(7.2 if two else 5.4, max(2.3, 1.4 + 0.3 * len(grupy))), dpi=110,
+ layout="constrained")
+ gs = fig.add_gridspec(1, 2 if two else 1, width_ratios=[1.6, 1.0] if two else [1])
+ ax = fig.add_subplot(gs[0])
+ y = np.arange(len(CT))[::-1]
+ frac = CT.div(CT.sum(axis=1).replace(0, 1), axis=0) * 100
+ left = np.zeros(len(CT))
+ for k in CT.columns:
+ ax.barh(y, frac[k].values, left=left, height=0.62, color=_GEO_KAT[k], label=k, zorder=2)
+ left += frac[k].values
+ for yy, k in zip(y, CT.index):
+ have = int(CT.loc[k, [c for c in CT.columns if c in _WITH_POINT]].sum())
+ ax.text(101.5, yy, f"{_th(have)} / {_th(CT.loc[k].sum())}", va="center", fontsize=6.3)
+ ax.set_yticks(y)
+ ax.set_yticklabels(CT.index, fontsize=7)
+ ax.set_xlim(0, 120)
+ ax.set_xticks([0, 25, 50, 75, 100])
+ ax.set_xlabel("share of the publisher's rows, %", fontsize=7.5)
+ n_have = int((d["_kat"].isin(_WITH_POINT)).sum())
+ ax.set_title(f"Coordinates in {_th(n_have)} of {_th(len(d))} rows;\n"
+ "absence is returned with a reason, not guessed", fontsize=8.5, loc="left")
+ ax.legend(loc="upper center", bbox_to_anchor=(0.5, -0.25), ncol=2, fontsize=6,
+ frameon=False, handlelength=1.0, columnspacing=0.9)
+ _style(ax)
+ if two:
+ ax2 = fig.add_subplot(gs[1])
+ pw = placed.astype(str).value_counts()
+ keys = [k for k in _PEWNOSC if k in pw.index] + [k for k in pw.index if k not in _PEWNOSC]
+ vv = [int(pw[k]) for k in keys]
+ yb = np.arange(len(keys))[::-1]
+ ax2.barh(yb, vv, height=0.6, color="#2e8b78", zorder=2)
+ for yy, v in zip(yb, vv):
+ ax2.text(v + max(vv) * 0.02, yy, f"{v} ({v / sum(vv) * 100:.1f} %)",
+ va="center", fontsize=6.3)
+ ax2.set_yticks(yb)
+ ax2.set_yticklabels([_PEWNOSC.get(k, k) for k in keys], fontsize=6.5)
+ ax2.set_xlim(0, max(vv) * 1.45)
+ ax2.set_xlabel("rows", fontsize=7.5)
+ ax2.set_title(f"Certainty of {_th(sum(vv))} identified place names", fontsize=8.5,
+ loc="left")
+ _style(ax2)
+ return fig
+
+
+# ---------------------------------------------------------------------------
+# one-call views
+# ---------------------------------------------------------------------------
+
+def plot_overview(df: pd.DataFrame, by: str = "publikujacy"):
+ """Four panels for a first look: coverage, quality, status and capacity."""
+ Figure = _mpl()
+ n_pub = len(_groups(df, by))
+ fig = Figure(figsize=(13, 9.6), dpi=100, layout="constrained")
+ gs = fig.add_gridspec(3, 2, width_ratios=[0.22 + 0.06 * min(n_pub, 9), 1.0],
+ height_ratios=[1.0, 0.75, 1.15])
+ plot_field_coverage(df, by=by, ax=fig.add_subplot(gs[:, 0]))
+ plot_quality(df, by=by, ax=fig.add_subplot(gs[0, 1]))
+ plot_status_distribution(df, by=by, normalize=True, ax=fig.add_subplot(gs[1, 1]))
+ plot_capacity_by_class(df, ax=fig.add_subplot(gs[2, 1]))
+ return fig
+
+
+def save_figure(fig, path, dpi: int = 300) -> Path:
+ """Save a figure (format from the suffix: .png, .svg, .pdf) and return the path."""
+ p = Path(path)
+ p.parent.mkdir(parents=True, exist_ok=True)
+ fig.savefig(p, dpi=dpi, bbox_inches="tight", facecolor="white")
+ return p
+
+
+def save_all(df: pd.DataFrame, out_dir, *, edition_b: Optional[pd.DataFrame] = None,
+ labels: tuple[str, str] = ("A", "B"), formats: Sequence[str] = ("png",),
+ dpi: int = 200, min_n: int = 30,
+ min_coverage: float = 0.8) -> dict[str, list[Path]]:
+ """Draw every chart the data supports and save it to ``out_dir``.
+
+ Charts that need data the disclosure does not carry are skipped and listed under the
+ key ``"_skipped"`` with the reason, rather than drawn empty. With ``edition_b`` the
+ edition-comparison charts are added, ``df`` being the earlier edition.
+
+ The processing-time curves follow the inclusion criterion of the article, fixed before
+ the result is computed: both dates populated in at least ``min_coverage`` of the
+ applications and at least ``min_n`` applications in a group.
+ """
+ from .analysis import assign_to_nodes, node_loading, station_node_map
+ from .longitudinal import compare_editions, linkage_audit, linkage_recall
+
+ out = Path(out_dir)
+ saved: dict[str, list[Path]] = {}
+ skipped: list[str] = []
+
+ def put(name, fig):
+ saved[name] = [save_figure(fig, out / f"{name}.{fmt}", dpi=dpi) for fmt in formats]
+
+ put("field_coverage", plot_field_coverage(df))
+ put("quality_rules", plot_quality(df))
+ put("column_shift", plot_column_shift(df))
+ if "status_procesu" in df and _filled(_applications(df)["status_procesu"]).any():
+ put("status_distribution", plot_status_distribution(df))
+ else:
+ skipped.append("status_distribution: no process status disclosed")
+ if "klasa_zasobu" in df and _filled(_applications(df)["klasa_zasobu"]).any():
+ put("capacity_by_class", plot_capacity_by_class(df))
+ else:
+ skipped.append("capacity_by_class: no resource class disclosed")
+ app = _applications(df)
+ cov = min((float(_filled(app[c]).mean()) if c in app and len(app) else 0.0)
+ for c in ("data_wniosku", "data_warunkow"))
+ km = kaplan_meier_by_group(app, min_n=min_n) if cov >= min_coverage else []
+ if cov < min_coverage:
+ skipped.append(f"processing_time: date coverage {cov * 100:.1f} % is below the "
+ f"{min_coverage * 100:.0f} % inclusion criterion")
+ elif len(km):
+ put("processing_time", plot_processing_time(df, min_n=min_n))
+ else:
+ skipped.append(f"processing_time: no group of {min_n} applications with data_wniosku")
+ if "wspolrzedne" in df:
+ put("location_resolution", plot_location_resolution(df))
+ if "moc_dostepna" in df and "_encja" in df and (df["_encja"] == "WEZEL").any() \
+ and (df["_encja"] != "WEZEL").any():
+ mapa = station_node_map(df)
+ if len(mapa):
+ przyp = assign_to_nodes(df, mapa)
+ for pole, nazwa in (("moc_wprowadzana", "generation"), ("moc_pobierana", "demand")):
+ L = node_loading(przyp, pole_mocy=pole)
+ if pd.to_numeric(L["obciazenie"], errors="coerce").notna().any():
+ put(f"node_loading_{nazwa}", plot_node_loading(L))
+ else:
+ skipped.append("node_loading: node rows carry no station list")
+ else:
+ skipped.append("node_loading: needs both node rows and application rows")
+ if edition_b is not None:
+ delta = compare_editions(df, edition_b, labels[0], labels[1])
+ aud = linkage_audit(df, edition_b)
+ rec = linkage_recall(df, edition_b)
+ put("edition_comparison", plot_edition_comparison(delta, aud, rec))
+ saved["_skipped"] = skipped # type: ignore[assignment]
+ return saved
diff --git a/src/gridqueue/registry.py b/src/gridqueue/registry.py
index 15a49e3..1956f93 100644
--- a/src/gridqueue/registry.py
+++ b/src/gridqueue/registry.py
@@ -17,7 +17,7 @@
from .adapters.pt_eredes import EredesAdapter
from .adapters.uk_nationalgrid import NationalGridAdapter
-__all__ = ["REGISTRY", "get_adapter", "detect_publisher", "declarations"]
+__all__ = ["REGISTRY", "get_adapter", "detect_publisher", "declarations", "parse_sources"]
REGISTRY: dict[str, Type[Adapter]] = {
"pl_tauron": TauronAdapter,
@@ -68,3 +68,77 @@ def detect_publisher(source, *, sniff_bytes: int = 4000) -> Optional[str]:
def declarations() -> list[dict]:
"""Declarations of optional fields of all registered publishers."""
return [cls().declaration() for cls in REGISTRY.values()]
+
+
+
+def parse_sources(sources, adapter: Optional[str] = None):
+ """Parse any number of documents, of one or several publishers, into one panel.
+
+ Each file is assigned to a publisher by :func:`detect_publisher` (or to ``adapter`` when
+ given) and parsed **on its own, by a fresh adapter instance**, so that every document keeps
+ the as-at date it states. This matters twice: a publisher that splits one edition into
+ several documents (ENERGA-OPERATOR's applications as at 30 June and node capacities as at
+ 31 August 2026) keeps both dates, and two editions of one register dropped together are
+ never stamped with a single date. The node and application rows of one publisher still end
+ up in one panel, which is all that :func:`~gridqueue.analysis.station_node_map` needs.
+
+ Returns ``(frame, reports)``: the concatenated frame in the schema and one
+ ``{"adapter", "files", "rows", "data_publikacji", "report"}`` entry per file. A file no
+ adapter recognises raises ``ValueError`` naming it, rather than being skipped silently.
+ """
+ import pandas as pd
+
+ paths = [sources] if isinstance(sources, (str, Path)) else list(sources)
+ keys = []
+ for p in paths:
+ key = adapter or detect_publisher(p)
+ if key is None:
+ raise ValueError(f"no adapter recognised {Path(str(p)).name!r}; keep the file "
+ f"name the publisher gave it, or choose the adapter explicitly")
+ keys.append(key)
+ frames, reports = [], []
+ for key, p in zip(keys, paths):
+ res = get_adapter(key).parse(p)
+ frames.append(res.frame)
+ dates = res.frame["data_publikacji"].dropna().astype(str).unique().tolist() \
+ if "data_publikacji" in res.frame else []
+ reports.append({"adapter": key, "files": [Path(str(p)).name], "rows": int(len(res.frame)),
+ "data_publikacji": dates[0] if len(dates) == 1 else dates,
+ "report": dict(res.report)})
+ frame = pd.concat(frames, ignore_index=True) if frames else pd.DataFrame()
+ return frame, reports
+
+
+def split_editions(df):
+ """Split a panel into editions of one register by the as-at date of its application rows.
+
+ Returns ``[(date, frame), ...]`` ordered by date when the panel holds exactly one
+ publisher whose application rows carry two or more publication dates; otherwise a
+ single ``(None, df)``. Node rows (``_encja == "WEZEL"``) go to the edition whose
+ application date is nearest to their own, later on a tie, because a publisher may date its
+ node and application documents of one edition differently.
+ """
+ import pandas as pd
+
+ if "data_publikacji" not in df or df.empty:
+ return [(None, df)]
+ is_node = df["_encja"].astype("string").fillna("WNIOSEK").str.upper().eq("WEZEL") \
+ if "_encja" in df else pd.Series(False, index=df.index)
+ pubs = df["publikujacy"].dropna().unique() if "publikujacy" in df else []
+ dates = sorted(df.loc[~is_node, "data_publikacji"].dropna().astype(str).unique())
+ if len(pubs) != 1 or len(dates) < 2:
+ return [(None, df)]
+ d = pd.to_datetime(pd.Series(dates))
+ own = df["data_publikacji"].astype(str)
+
+ def edition_of(v):
+ if v in dates:
+ return v
+ t = pd.to_datetime(v, errors="coerce")
+ if pd.isna(t):
+ return dates[-1]
+ gap = (d - t).abs()
+ return dates[max(i for i in range(len(dates)) if gap[i] == gap.min())]
+
+ lab = own.map(edition_of)
+ return [(x, df[lab == x].reset_index(drop=True)) for x in dates]
diff --git a/tests/test_analysis_v16.py b/tests/test_analysis_v16.py
new file mode 100644
index 0000000..de81e1c
--- /dev/null
+++ b/tests/test_analysis_v16.py
@@ -0,0 +1,83 @@
+"""Regressions fixed in 1.6.0: censoring date per row, voltage filter of node loading."""
+
+import pandas as pd
+
+import gridqueue as G
+
+
+def test_censoring_uses_the_rows_own_publication_date():
+ # Node rows published later must not lengthen the censored times of applications.
+ df = pd.DataFrame([
+ {"_encja": "WNIOSEK", "data_wniosku": "2026-01-01", "data_publikacji": "2026-06-30"},
+ {"_encja": "WNIOSEK", "data_wniosku": "2026-01-01", "data_warunkow": "2026-03-01",
+ "data_publikacji": "2026-06-30"},
+ {"_encja": "WEZEL", "data_publikacji": "2026-08-31"},
+ ])
+ pt = G.processing_time(df)
+ assert pt.loc[0, "czas_dni"] == (pd.Timestamp("2026-06-30") - pd.Timestamp("2026-01-01")).days
+ assert pt.loc[0, "zdarzenie"] == 0 and pt.loc[1, "zdarzenie"] == 1
+
+
+def test_censoring_per_publisher_in_a_panel():
+ df = pd.DataFrame([
+ {"publikujacy": "A", "data_wniosku": "2026-01-01", "data_publikacji": "2026-03-31"},
+ {"publikujacy": "B", "data_wniosku": "2026-01-01", "data_publikacji": "2026-06-30"},
+ ]).assign(data_warunkow=None)
+ pt = G.processing_time(df)
+ assert list(pt["czas_dni"]) == [89, 180]
+
+
+def test_explicit_as_at_date_applies_to_every_row():
+ df = pd.DataFrame([{"data_wniosku": "2026-01-01", "data_publikacji": "2026-03-31",
+ "data_warunkow": None}])
+ assert G.processing_time(df, na_dzien="2026-01-11")["czas_dni"].iloc[0] == 10
+
+
+def test_node_loading_voltage_filter():
+ rows = [
+ {"_encja": "WEZEL", "id_wezla": "G1", "nazwa_wezla": "Alfa", "_nr_grupy": 1,
+ "lokalizacja_tekst": "Alfa", "moc_dostepna": 1000.0, "_kierunek": "wytworcza"},
+ {"_encja": "WNIOSEK", "lokalizacja_tekst": "Alfa", "moc_wprowadzana": 3000.0,
+ "status_procesu": "WARUNKI_WYDANE", "poziom_napiecia": "110 kV"},
+ {"_encja": "WNIOSEK", "lokalizacja_tekst": "Alfa", "moc_wprowadzana": 500.0,
+ "status_procesu": "WARUNKI_WYDANE", "poziom_napiecia": "SN"},
+ ]
+ df = pd.DataFrame(rows)
+ przyp = G.assign_to_nodes(df, G.station_node_map(df))
+ assert G.node_loading(przyp, napiecie="110")["obciazenie"].iloc[0] == 3.0
+ assert G.node_loading(przyp)["obciazenie"].iloc[0] == 3.5
+
+
+def test_split_editions_by_application_date():
+ from gridqueue import examples as X
+ a, b = X.synthetic_editions(120)
+ eds = G.split_editions(pd.concat([b, a], ignore_index=True))
+ assert [x for x, _ in eds] == ["2026-05-31", "2026-06-30"]
+ assert [len(f) for _, f in eds] == [len(a), len(b)]
+
+
+def test_split_editions_keeps_one_edition_with_node_rows_of_another_date():
+ df = pd.DataFrame([
+ {"_encja": "WNIOSEK", "publikujacy": "P", "data_publikacji": "2026-06-30"},
+ {"_encja": "WEZEL", "publikujacy": "P", "data_publikacji": "2026-08-31"},
+ ])
+ eds = G.split_editions(df)
+ assert len(eds) == 1 and eds[0][0] is None
+
+
+def test_split_editions_assigns_node_rows_to_the_nearest_edition():
+ df = pd.DataFrame([
+ {"_encja": "WNIOSEK", "publikujacy": "P", "data_publikacji": "2026-03-31"},
+ {"_encja": "WNIOSEK", "publikujacy": "P", "data_publikacji": "2026-06-30"},
+ {"_encja": "WEZEL", "publikujacy": "P", "data_publikacji": "2026-07-15"},
+ ])
+ eds = dict(G.split_editions(df))
+ assert len(eds["2026-03-31"]) == 1 and len(eds["2026-06-30"]) == 2
+
+
+def test_parse_sources_names_an_unrecognised_file(tmp_path):
+ p = tmp_path / "unknown.txt"
+ p.write_text("nothing here", encoding="utf-8")
+ import pytest
+ with pytest.raises(ValueError, match="unknown.txt"):
+ G.parse_sources([p])
diff --git a/tests/test_examples_geocode.py b/tests/test_examples_geocode.py
new file mode 100644
index 0000000..b468a01
--- /dev/null
+++ b/tests/test_examples_geocode.py
@@ -0,0 +1,45 @@
+"""Synthetic examples and geocode_frame."""
+
+import json
+
+import gridqueue as G
+from gridqueue import examples as X
+
+
+def test_synthetic_panel_is_clean_and_complete():
+ df = X.synthetic_panel(300, seed=3)
+ assert set(df["publikujacy"].dropna()) <= set(X.SYNTHETIC_PUBLISHERS)
+ assert G.run_quality(df).as_dict()["naruszen_lacznie"] == 0
+ assert (df["_encja"] == "WEZEL").sum() == 16
+ assert len(G.kaplan_meier_by_group(df, min_n=20)) >= 2
+
+
+def test_synthetic_panel_is_reproducible():
+ a, b = X.synthetic_panel(50, seed=1), X.synthetic_panel(50, seed=1)
+ assert a.equals(b)
+
+
+def test_synthetic_editions_have_every_kind_of_flow():
+ a, b = X.synthetic_editions(300)
+ d = G.compare_editions(a, b, "A", "B")
+ assert d.nowych > 0 and d.ubylych > 0 and d.zmian_statusu > 0
+ assert d.przejscia_wstecz >= 1
+
+
+def test_geocode_frame_writes_schema_points_and_reasons(tmp_path):
+ p = tmp_path / "gaz.json"
+ p.write_text(json.dumps(X.synthetic_gazetteer()), encoding="utf-8")
+ res = G.resolver_from_gazeter_json(p)
+ df = X.synthetic_panel(200, seed=5, with_nodes=False)
+ df["wspolrzedne"] = df["wspolrzedne"].astype(object)
+ df.loc[df.index[0], "wspolrzedne"] = "50.0,20.0"
+ out = G.geocode_frame(df, res)
+ assert out.loc[out.index[0], "_geo_pewnosc"] == "ZRODLO"
+ assert out.loc[out.index[0], "wspolrzedne"] == "50.0,20.0"
+ placed = out["wspolrzedne"].notna()
+ assert placed.sum() > 1
+ assert (out.loc[~placed, "_geo_powod"].str.len() > 0).all()
+ # every point is in the format the schema validates
+ assert G.validate_frame(out)["n_wierszy"] == len(out)
+ lat = out.loc[placed, "wspolrzedne"].str.split(",").str[0].astype(float)
+ assert lat.between(-90, 90).all()
diff --git a/tests/test_plots.py b/tests/test_plots.py
new file mode 100644
index 0000000..0199ed2
--- /dev/null
+++ b/tests/test_plots.py
@@ -0,0 +1,173 @@
+"""Charts: every function draws from constructed frames without raising, and the numbers
+it puts on the chart are the numbers of the analysis it shows."""
+
+import json
+
+import pandas as pd
+import pytest
+
+pytest.importorskip("matplotlib")
+
+from gridqueue import (compare_editions, kaplan_meier, kaplan_meier_by_group, # noqa: E402
+ linkage_audit, linkage_recall)
+from gridqueue import cli, plots # noqa: E402
+from gridqueue.analysis import assign_to_nodes, node_loading, station_node_map # noqa: E402
+from gridqueue.schema import empty_frame # noqa: E402
+
+
+def _frame(rows):
+ return pd.DataFrame(rows).reindex(columns=list(empty_frame().columns) + [
+ c for c in pd.DataFrame(rows).columns if c not in empty_frame().columns])
+
+
+def _panel(n=80):
+ rows = []
+ for i in range(n):
+ klasa = "PV" if i % 2 else "ODB"
+ rows.append({
+ "id_wniosku": f"A{i}", "lokalizacja_tekst": f"Stacja {i % 4}",
+ "klasa_zasobu": klasa, "poziom_napiecia": "SN" if i % 3 else "110 kV",
+ "moc_wprowadzana": 1000.0 + i if klasa == "PV" else None,
+ "moc_pobierana": 500.0 + i if klasa == "ODB" else None,
+ "status_procesu": "WARUNKI_WYDANE" if i % 5 else "WNIOSEK_ZLOZONY",
+ "data_wniosku": f"2025-01-{1 + i % 28:02d}",
+ "data_warunkow": f"2025-0{3 + i % 6}-15" if i % 5 else None,
+ "data_publikacji": "2025-12-31",
+ "publikujacy": "PSE S.A." if i < n // 2 else "ENERGA-OPERATOR S.A.",
+ })
+ return _frame(rows)
+
+
+def _texts(fig):
+ out = []
+ for ax in fig.axes:
+ out.append(ax.get_title(loc="left"))
+ out += [t.get_text() for t in ax.texts]
+ return " ".join(out)
+
+
+@pytest.fixture(scope="module")
+def panel():
+ return _panel()
+
+
+def test_every_panel_chart_draws(panel):
+ for f in (plots.plot_field_coverage, plots.plot_quality, plots.plot_column_shift,
+ plots.plot_status_distribution, plots.plot_capacity_by_class,
+ plots.plot_processing_time, plots.plot_location_resolution,
+ plots.plot_overview):
+ fig = f(panel)
+ assert fig.axes, f.__name__
+
+
+def test_charts_tolerate_a_minimal_frame():
+ df = _frame([{"id_wniosku": "X", "lokalizacja_tekst": "a", "klasa_zasobu": "ODB",
+ "status_procesu": "WNIOSEK_ZLOZONY", "moc_pobierana": 10.0}])
+ for f in (plots.plot_field_coverage, plots.plot_quality, plots.plot_status_distribution,
+ plots.plot_capacity_by_class, plots.plot_processing_time,
+ plots.plot_location_resolution, plots.plot_overview):
+ assert f(df).axes
+
+
+def test_coverage_counts_fields_like_the_schema(panel):
+ txt = plots.plot_field_coverage(panel).axes[0].get_title(loc="left")
+ populated = sum(1 for c in plots.schema_table()["pole"] if panel[c].notna().any())
+ assert f"{populated} of 21 fields populated" in txt
+
+
+def test_quality_chart_total_matches_run_quality(panel):
+ from gridqueue import run_quality
+
+ fig = plots.plot_quality(panel, by=None)
+ expected = {r["kod"]: r["naruszenia"] for r in run_quality(panel).as_dict()["reguly"]}
+ shown = _texts(fig)
+ for kod, n in expected.items():
+ if n > 0:
+ assert f"{n:,} (" in shown, kod
+
+
+def test_km_chart_legend_uses_the_estimator(panel):
+ tab = kaplan_meier_by_group(panel, by="klasa_zasobu", min_n=10)
+ fig = plots.plot_processing_time(panel, min_n=10)
+ legend = " ".join(t.get_text() for t in fig.axes[0].get_legend().get_texts())
+ for r in tab.itertuples():
+ assert f"n={r.n:,}" in legend
+ assert f"median {r.mediana:.0f} d" in legend
+
+
+def test_km_median_interval_is_ordered():
+ # Regression: up to 1.5.2 the two bounds of the median interval were swapped.
+ t = list(range(1, 201))
+ e = [1] * 200
+ km = kaplan_meier(t, e)
+ assert km.mediana_dolna <= km.mediana <= km.mediana_gorna
+
+
+def test_km_all_censored_group_does_not_raise():
+ # Regression: a group with no event produced a frame without columns (KeyError 'S').
+ km = kaplan_meier([10, 20, 30], [0, 0, 0])
+ assert km.n == 3 and km.zdarzen == 0 and km.mediana != km.mediana
+
+
+def test_km_by_group_respects_min_n(panel):
+ assert kaplan_meier_by_group(panel, min_n=10_000).empty
+ tab = kaplan_meier_by_group(panel, by=["publikujacy", "klasa_zasobu"], min_n=5)
+ assert tab["grupa"].str.contains(" / ").all()
+
+
+def _editions():
+ a = _panel(40)
+ b = a.copy()
+ b.loc[b.index[:5], "status_procesu"] = "UMOWA_OBOWIAZUJACA"
+ b = b.iloc[2:]
+ return a, b
+
+
+def test_edition_charts_show_the_delta():
+ a, b = _editions()
+ d = compare_editions(a, b, "A", "B")
+ fig = plots.plot_edition_comparison(d, linkage_audit(a, b), linkage_recall(a, b))
+ assert len(fig.axes) == 3
+ txt = _texts(fig)
+ assert f"{d.wspolnych:,} linked" in txt
+
+
+def test_node_loading_chart_skips_zero_capacity():
+ rows = [
+ {"_encja": "WEZEL", "id_wezla": "G1", "nazwa_wezla": "Alfa", "_nr_grupy": 1,
+ "lokalizacja_tekst": "Alfa, Beta", "moc_dostepna": 1000.0, "_kierunek": "wytworcza"},
+ {"_encja": "WEZEL", "id_wezla": "G2", "nazwa_wezla": "Gamma", "_nr_grupy": 2,
+ "lokalizacja_tekst": "Gamma", "moc_dostepna": 0.0, "_kierunek": "wytworcza"},
+ {"_encja": "WNIOSEK", "lokalizacja_tekst": "Alfa", "moc_wprowadzana": 2500.0,
+ "status_procesu": "WARUNKI_WYDANE"},
+ {"_encja": "WNIOSEK", "lokalizacja_tekst": "Gamma", "moc_wprowadzana": 500.0,
+ "status_procesu": "WARUNKI_WYDANE"},
+ ]
+ df = pd.DataFrame(rows)
+ L = node_loading(assign_to_nodes(df, station_node_map(df)), pole_mocy="moc_wprowadzana")
+ fig = plots.plot_node_loading(L)
+ title = fig.axes[0].get_title(loc="left")
+ assert "median 2.500" in title and "1 node with zero free capacity" in title
+
+
+def test_save_all_and_cli_plot(tmp_path, panel, capsys):
+ p = tmp_path / "panel.parquet"
+ panel.to_parquet(p, index=False)
+ assert cli.main(["plot", str(p), "--out", str(tmp_path / "figs"), "--format", "png,svg",
+ "--min-n", "10"]) == 0
+ out = json.loads(capsys.readouterr().out)
+ assert "field_coverage" in out["zapisane"]
+ assert (tmp_path / "figs" / "quality_rules.svg").exists()
+ assert any("node_loading" in s for s in out["pominiete"])
+
+
+def test_cli_compare(tmp_path, capsys):
+ a, b = _editions()
+ pa, pb = tmp_path / "a.parquet", tmp_path / "b.parquet"
+ a.to_parquet(pa, index=False)
+ b.to_parquet(pb, index=False)
+ assert cli.main(["compare", str(pa), str(pb), "--transitions",
+ str(tmp_path / "tr.csv")]) == 0
+ out = json.loads(capsys.readouterr().out)
+ assert out["porownanie"]["wspolnych"] == compare_editions(a, b).wspolnych
+ assert (tmp_path / "tr.csv").exists()