Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
331 changes: 27 additions & 304 deletions CHANGELOG.md

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion CITATION.cff
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ abstract: >-
and the corresponding registers from other jurisdictions into a single set of rows in a 21-field
schema derived from the structure of the legal obligation.
type: software
version: "1.7.0"
version: "1.0.0"
date-released: "2026-10-24"
license: Apache-2.0
repository-code: "https://github.com/s-matysik/gridqueue"
Expand Down
30 changes: 15 additions & 15 deletions CODE_METADATA.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,9 @@ listed again in the final section.

| Nr | Item | Value |
|----|------|-------|
| C1 | Current code version | `1.7.0` (declared identically in `pyproject.toml` and in `gridqueue.__version__`; CI asserts the two agree on every push) |
| C2 | Permanent link to code / repository used for this code version | https://github.com/s-matysik/gridqueue (tag `v1.7.0`) |
| C3 | Permanent link to Reproducible Capsule | ⟦TEAM DECISION⟧, archive DOI. The repository is ready for a Zenodo deposit of tag `v1.6.0`; enabling the integration requires the team's account |
| C1 | Current code version | `1.0.0` (declared identically in `pyproject.toml` and in `gridqueue.__version__`; CI asserts the two agree on every push) |
| C2 | Permanent link to code / repository used for this code version | https://github.com/s-matysik/gridqueue (tag `v1.0.0`) |
| C3 | Permanent link to Reproducible Capsule | ⟦TEAM DECISION⟧, archive DOI. The repository is ready for a Zenodo deposit of tag `v1.0.0`; enabling the integration requires the team's account |
| C4 | Legal Code License | Apache-2.0, `LICENSE.txt` at the distribution root. Input data licences are fixed and separate: OpenStreetMap gazetteer under ODbL 1.0; operators' statutory publications under the terms each publisher applies to its own site |
| C5 | Code versioning system used | git |
| C6 | Software code languages, tools and services used | Python, `requires-python >= 3.10`; build backend `setuptools.build_meta` (`setuptools>=68`); test runner `pytest` |
Expand All @@ -22,8 +22,8 @@ listed again in the final section.

| Nr | Item | Value |
|----|------|-------|
| S1 | Current software version | `1.7.0`, matching C1 |
| S2 | Permanent link to executables of this version | Source distribution only, `gridqueue-1.7.0.tar.gz`, pure Python, no binaries, no compiled extensions |
| S1 | Current software version | `1.0.0`, matching C1 |
| S2 | Permanent link to executables of this version | Source distribution only, `gridqueue-1.0.0.tar.gz`, pure Python, no binaries, no compiled extensions |
| S3 | Legal Software License | Apache-2.0, same licence as C4 |
| S4 | Computing platforms / Operating Systems | Linux, macOS, Windows × Python 3.10, 3.11, 3.12, 3.13, a twelve-cell matrix exercised by GitHub Actions (`.github/workflows/ci.yml`) on every push and pull request. The measured figures reported in the paper were produced on macOS/arm64 with Python 3.11.15 |
| S5 | Installation requirements & dependencies | As C7. No GPU, no network access required at run time once the source documents and the gazetteer are local |
Expand Down Expand Up @@ -56,28 +56,28 @@ not.
11 process statuses. Verified by measured coverage across the six packaged row-level sources
(`gridqueue_coverage_matrix_6publ.csv`): ENERGA-OPERATOR 17/21, PSE 17/21, TAURON 15/21,
Stoen 13/21, National Grid (GB) 10/21, Boryszew 8/21. **All 21 schema fields are populated
somewhere in the 19,220-row panel.** `moc_dostepna` is closed from sources by 96 ENERGA node
somewhere in the 19,220-row panel.** `capacity_available` is closed from sources by 96 ENERGA node
rows (0-175,000 kW); PSE discharges the same obligation point as a **count of free connection
slots rather than a power figure** (69 switchgears), mapped to `ograniczenie_flaga` and an
extension field, leaving `moc_dostepna` empty rather than fabricating a unit. `wspolrzedne` is
slots rather than a power figure** (69 switchgears), mapped to `constraint_flag` and an
extension field, leaving `capacity_available` empty rather than fabricating a unit. `coordinates` is
populated by derivation from the place name, not by disclosure: no publisher reports coordinates.
Extensions across publishers are complementary, not conflicting.
**Resolved in 1.0.** The core no longer requires `moc_pobierana` unconditionally. It requires the
**Resolved in 1.0.** The core no longer requires `capacity_withdrawn` unconditionally. It requires the
row's OWN entity core: for a APPLICATION row four unconditional fields plus **at least one** of
(`moc_pobierana`, `moc_wprowadzana`); for a WEZEL row `id_wezla` plus at least one of
(`moc_dostepna`, `ograniczenie_flaga`). Rule R7 therefore fell from 5,390 violations (28.044%)
(`capacity_withdrawn`, `capacity_injected`); for a NODE row `node_id` plus at least one of
(`capacity_available`, `constraint_flag`). Rule R7 therefore fell from 5,390 violations (28.044%)
to 771 (4.011%), and rows with a complete core rose from 13,830 to 18,449 of 19,220. What remains
is a genuine disclosure gap, not a schema artefact: 570 applications with no power figure at all,
202 with no location text, 2 nodes with neither available capacity nor a constraint flag.
Superseded text follows for the record: the core required `moc_pobierana`, which purely generating applications do
Superseded text follows for the record: the core required `capacity_withdrawn`, which purely generating applications do
not carry, 5,390 of 19,220 rows (28.044%) in 0.3.x, the whole of rule R7 at that time. The core should require
either capacity, consumed or injected.
2. Every location resolution carries an explicit confidence level on a six-point scale (`EXACT`,
`UNIQUE_TOKEN`, `TOKEN_SUBSET`, `REVERSED`, `SKRZYZOWANIE`, `NONE` with a reason), plus a second,
`UNIQUE_TOKEN`, `TOKEN_SUBSET`, `REVERSED`, `INTERSECTION`, `NONE` with a reason), plus a second,
independent placement-accuracy level (address point, intersection point, point on the street
axis, locality centroid, no geometry, none). A failure is returned as missing-with-reason, never
as a guess. Street identification, Stoen Warsaw subset (1,179 positions): `EXACT` 884,
`UNIQUE_TOKEN` 152, `SKRZYZOWANIE` 70, `REVERSED` 54, `TOKEN_SUBSET` 5, `NONE` 14-1,165 of
`UNIQUE_TOKEN` 152, `INTERSECTION` 70, `REVERSED` 54, `TOKEN_SUBSET` 5, `NONE` 14-1,165 of
1,179 resolved, 98.81%; without the intersection rule 1,095 of 1,179 = 92.88%; without both the
inverted-order and the intersection rule 1,041 of 1,179 = 88.30%.
Coordinate coverage after the gazetteer was rebuilt with geometry (13,149 street names in 25
Expand Down Expand Up @@ -113,7 +113,7 @@ blocker is server-side client filtering, the site returns a rejection page to th
control the operator put in place deliberately, so it was not attempted. Electricity North West:
rows behind the same access control as UK Power Networks; metadata only.

**Size.** Read from the built source distribution `gridqueue-1.7.0.tar.gz`: 7 core modules under `src/gridqueue`, 2,125 lines, `geoloc.py` 672, `schema.py` 480, `quality.py` 370, `layout.py` 353, `cli.py` 148, `registry.py` 64, `__init__.py` 38, and 9 adapter modules, 1,777 lines (`pl_pse.py` 303, `pl_energa.py` 282, `_ptpiree.py` 278, `pl_boryszew.py` 232, `pl_stoen.py` 194, `uk_nationalgrid.py` 166, `pl_tauron.py` 161, `base.py` 156, `__init__.py` 5); 3,902 lines in total under `src`. `_ptpiree.py` is shared by the PSE and ENERGA adapters because both publishers fill the same agreed industry template, one as a spreadsheet and one as a print document. Tests: 7 files, 1,040 lines (`test_geoloc_coordinates.py` 280, `test_schema.py` 164, `test_pse_energa.py` 147, `test_adapters.py` 144, `test_quality.py` 124, `test_geoloc.py` 95, `test_layout.py` 86). Runnable workflows shipped alongside the package: `examples/research_use_case.py` 168 lines, and `reproducibility/` 382 lines (`run_figures.py` 248, `run_validation.py` 134). `python -m pytest -q`: **153 passed, 2 skipped (155 collected), 0 failed, 0 errors**, the two skips are the PSE and ENERGA integration tests, which require the source documents and are skipped when those are absent from the working tree. The twelve-cell OS × Python matrix in CI runs the same suite.
**Size.** Read from the built source distribution `gridqueue-1.0.0.tar.gz`: 7 core modules under `src/gridqueue`, 2,125 lines, `geoloc.py` 672, `schema.py` 480, `quality.py` 370, `layout.py` 353, `cli.py` 148, `registry.py` 64, `__init__.py` 38, and 9 adapter modules, 1,777 lines (`pl_pse.py` 303, `pl_energa.py` 282, `_ptpiree.py` 278, `pl_boryszew.py` 232, `pl_stoen.py` 194, `uk_nationalgrid.py` 166, `pl_tauron.py` 161, `base.py` 156, `__init__.py` 5); 3,902 lines in total under `src`. `_ptpiree.py` is shared by the PSE and ENERGA adapters because both publishers fill the same agreed industry template, one as a spreadsheet and one as a print document. Tests: 7 files, 1,040 lines (`test_geoloc_coordinates.py` 280, `test_schema.py` 164, `test_pse_energa.py` 147, `test_adapters.py` 144, `test_quality.py` 124, `test_geoloc.py` 95, `test_layout.py` 86). Runnable workflows shipped alongside the package: `examples/research_use_case.py` 168 lines, and `reproducibility/` 382 lines (`run_figures.py` 248, `run_validation.py` 134). `python -m pytest -q`: **153 passed, 2 skipped (155 collected), 0 failed, 0 errors**, the two skips are the PSE and ENERGA integration tests, which require the source documents and are skipped when those are absent from the working tree. The twelve-cell OS × Python matrix in CI runs the same suite.

**Functionalities.** Document ingest with per-row provenance; layout-based extraction; unit
normalisation to the canonical kW (MW→kW, with the source unit retained in an extension column); resource-class and process-status vocabulary mapping; composite-date
Expand Down
6 changes: 3 additions & 3 deletions DATA_LICENSE.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,10 +42,10 @@ with coordinates attached, made available onwards, does trigger share-alike.
## 4. The resulting panel

The panel produced by the package is a work derived from both of the layers above.
To the extent that it contains **values derived from the gazetteer** (the `wspolrzedne` field
To the extent that it contains **values derived from the gazetteer** (the `coordinates` field
and the resolver's diagnostic columns), it is subject to ODbL. To the extent that it contains
only content transcribed from official documents, it is subject to the status of those documents.
The `_geo_pewnosc` column makes it possible to separate those two layers: rows with no resolved
The `_geo_confidence` column makes it possible to separate those two layers: rows with no resolved
location contain no content originating from OpenStreetMap.

## 5. Personal data
Expand Down Expand Up @@ -85,7 +85,7 @@ more current it looked.

The consequence has to be stated openly, because it is inconvenient: **byte-for-byte
reproduction ceases to be possible once a publisher replaces or moves a document.**
While release 1.3.0 was being prepared, the link to the disclosure of one of the operators
While the package was being prepared, the link to the disclosure of one of the operators
stopped leading to the file and began returning a page of the service; the download script
recognizes that case and reports it separately instead of saving the page as
a document. That is why the checksums are in the manifest: a mismatch is a signal that
Expand Down
23 changes: 12 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,14 +17,15 @@ public, but it is not processable.

`gridqueue` turns those publications into one set of rows with **measured accuracy**.

Release 1.7.0: **nine adapters in five jurisdictions** (Poland, Spain, Ireland, Portugal, Great
Release 1.0.0: **nine adapters in five jurisdictions** (Poland, Spain, Ireland, Portugal, Great
Britain), all 21 schema fields populated by at least one publisher, quality rules, a longitudinal
module with a linkage audit, processing times with censoring, node loading, a location resolver and
**charts for every result**. Documentation: **https://s-matysik.github.io/gridqueue/**

The code and documentation are in English. Schema field names and codes (`id_wniosku`,
`WARUNKI_WYDANE`, ...) keep the wording of the Polish statute, because they are the data format of
the panel; their English meaning is in the [glossary](https://s-matysik.github.io/gridqueue/schema.html#glossary-of-field-and-code-names).
The code, the schema and the documentation are in English. The Supplementary Material of the
article names the fields in the Polish wording of the statute (`id_wniosku`, `moc_wprowadzana`, ...);
the [correspondence table](https://s-matysik.github.io/gridqueue/schema.html#names-in-the-wording-of-the-statute)
gives the English names, and `gridqueue.from_legacy_names(df)` converts such a table.

## Three ways to run it

Expand Down Expand Up @@ -79,7 +80,7 @@ resolver is shown on a synthetic gazetteer, with a one-line change to use your o
One command, with no compilation and no external services:

```bash
pip install "gridqueue[plot] @ git+https://github.com/s-matysik/gridqueue@v1.7.0"
pip install "gridqueue[plot] @ git+https://github.com/s-matysik/gridqueue@v1.0.0"
```

The `plot` extra adds matplotlib for the charts; without it everything else works.
Expand Down Expand Up @@ -114,7 +115,7 @@ from gridqueue import resolver_from_osm_json
r = resolver_from_osm_json("streets_warsaw.json")
res = r.resolve("WARSZAWA, WINCENTEGO ŚW.")
print(res.matched, res.confidence) # Świętego Wincentego REVERSED
print(r.resolve("SKB3").reason) # kod_stacji_bez_slownika, we do not guess
print(r.resolve("SKB3").reason) # station_code_without_dictionary, we do not guess
```

## CLI
Expand Down Expand Up @@ -156,7 +157,7 @@ skipped node with the reason), never as zero.
from gridqueue import parse_sources, plots

panel, reports = parse_sources(["applications.pdf", "demand_capacity.pdf", "generation_capacity.pdf"])
plots.plot_processing_time(panel, by="klasa_zasobu") # displays in Jupyter / Colab
plots.plot_processing_time(panel, by="resource_class") # displays in Jupyter / Colab
plots.save_all(panel, "charts/", formats=("png", "svg"), dpi=300)
```

Expand All @@ -166,8 +167,8 @@ as synthetic, for trying every function without a real disclosure.
## Schema

21 fields in three groups reflecting the structure of the duty (IDENTITY / CAPACITY / PROCESS),
a mandatory core of 5 fields (`id_wniosku`, `lokalizacja_tekst`, `klasa_zasobu`, `moc_pobierana`,
`status_procesu`), the rest optional and **declared per publisher**. Canonical units:
a mandatory core of 5 fields (`application_id`, `location_text`, `resource_class`, `capacity_withdrawn`,
`process_status`), the rest optional and **declared per publisher**. Canonical units:
capacity in kW, ISO 8601 dates, WGS84 coordinates. Fields beyond the schema, which a given operator
supplies beyond the required minimum, go into columns with the `_` prefix and are declared explicitly
by the adapter (`adapter.declaration()`).
Expand Down Expand Up @@ -290,9 +291,9 @@ detected by comparing the output with the reference set read from the document.
raw agreement 0,9725, Cohen's kappa 0,9690, zero perceptual discrepancies. The PSE
and ENERGA reference sets remain single-annotator.
* The location at TAURON is an internal substation code; the operator does not publish a dictionary of
the codes, so the resolver returns missing with the reason `kod_stacji_bez_slownika`. It does not
the codes, so the resolver returns missing with the reason `station_code_without_dictionary`. It does not
guess.
* **None of the publishers reports coordinates.** The `wspolrzedne` field is populated by
* **None of the publishers reports coordinates.** The `coordinates` field is populated by
derivation from the place name against a gazetteer with geometry (13,149 names in 25 localities):
1,225 of 19,220 panel rows. The evidential value of this field is therefore different from that of
the fields transcribed from the document, and it is described as such in the result.
Expand Down
22 changes: 11 additions & 11 deletions docs/adapters.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,8 @@ from gridqueue.adapters.base import Adapter, ParseResult

class MojAdapter(Adapter):
key = "pl_moj_operator"
publikujacy = "Nazwa Operatora S.A."
jurysdykcja = "PL"
publisher = "Nazwa Operatora S.A."
jurisdiction = "PL"
detekcja = [r"moj_operator", r"nazwa_operatora"] # file name patterns

def parse(self, source) -> ParseResult:
Expand Down Expand Up @@ -73,23 +73,23 @@ apparent power. The canonical unit of the schema is **kW of active power**. Thes
physical quantities, and converting one into the other requires a power factor,
which the publisher does not report.

That is why `moc_dostepna` remains **empty**, and the values in MVA go into the
That is why `capacity_available` remains **empty**, and the values in MVA go into the
extension fields together with the source unit recorded explicitly:

| extension field | meaning |
|---|---|
| `_jednostka_mocy_zrodlowa` | `MVA` |
| `_moc_dostepna_mva` | MV+HV reception capability |
| `_moc_przylaczona_mva` | capacity already connected |
| `_moc_zakontraktowana_mva` | contracted capacity |
| `_moc_w_potwierdzaniu_mva` | capacity under confirmation |
| `_source_capacity_unit` | `MVA` |
| `_capacity_available_mva` | MV+HV reception capability |
| `_capacity_connected_mva` | capacity already connected |
| `_capacity_contracted_mva` | contracted capacity |
| `_capacity_being_confirmed_mva` | capacity under confirmation |

This is the same principle that was applied to the publisher reporting a count of
free connection slots instead of a capacity: entering a count of slots into a field with a different
unit would be fabrication, not harmonization.

The core of the NODE entity is nevertheless satisfied, because it is discharged by the second member of
the alternative, `ograniczenie_flaga`, set on the basis of a zero reception capability. Zero is
the alternative, `constraint_flag`, set on the basis of a zero reception capability. Zero is
a **disclosed** value and means a constraint; an empty cell means non-disclosure
and does not set the flag. This concerns 222 of 469 nodes.

Expand All @@ -101,7 +101,7 @@ a Polish category error, turned out to be the condition for absorbing a foreign
```python
from gridqueue import get_adapter
r = get_adapter("pt_eredes").parse("pt_eredes_capacidade_rececao_rnd_2T2026.csv")
r.frame[["id_wezla", "nazwa_wezla", "lokalizacja_tekst", "ograniczenie_flaga"]]
r.frame[["node_id", "node_name", "location_text", "constraint_flag"]]
r.report["note_units"]
```

Expand Down Expand Up @@ -150,7 +150,7 @@ transformers at a substation. Coordinates are present in 46,525 rows.
### Two units in one file

The publisher reports the **offtake side in MVA** (apparent power) and the **generating side in MW**
(active power). The `moc_dostepna` field therefore carries the available **firm generating**
(active power). The `capacity_available` field therefore carries the available **firm generating**
capacity, converted from MW to kW, while the offtake side stays in the extensions with the unit in the
field name. Summing MW with MVA would give a figure with no physical sense, and choosing one side
without saying so openly would be hiding a decision.
Expand Down
4 changes: 2 additions & 2 deletions docs/analysis.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,8 @@ from gridqueue import station_node_map, assign_to_nodes, node_loading

mapping = station_node_map(panel) # station -> group of stations
panel = assign_to_nodes(panel, mapping) # assignment of applications to groups
L = node_loading(panel, capacity_field="moc_wprowadzana")
L[["nazwa_wezla", "applications", "capacity_requested", "moc_dostepna", "loading"]]
L = node_loading(panel, capacity_field="capacity_injected")
L[["node_name", "applications", "capacity_requested", "capacity_available", "loading"]]
```

**The key is the group of stations, not the node row.** The publisher lists the same
Expand Down
Binary file added docs/app/gridqueue-1.0.0-py3-none-any.whl
Binary file not shown.
Binary file removed docs/app/gridqueue-1.7.0-py3-none-any.whl
Binary file not shown.
Loading
Loading