diff --git a/CHANGELOG.md b/CHANGELOG.md index e8f2ee9..4a4bd63 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,307 +4,30 @@ Format based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); semantic versioning. The version of the data schema is separate from the version of the package and is reported as `SCHEMA_VERSION`. -## 1.7.0 (2026-10-24) - -### Changed -- **The whole package is in English.** Function arguments, the keys of result objects and the - columns of result tables, local names, comments, docstrings, error messages, the CI workflow, - the Colab notebook and the documentation file names (`schemat.md` is now `schema.md`, and so on). - The full list of renamed public names is in [docs/renamed.md](docs/renamed.md). -- The three renamed top-level names (`PRZEDROSTKI_STACJI`, `normalizuj_nazwe`, - `resolver_from_gazeter_json`) keep working with a `DeprecationWarning` until 2.0. Renamed keyword - arguments and result keys have no aliases. -- Legal citations follow the style of the article: `Art. 7(8l), item 1/3/4` instead of - `pkt 1/3/4`. Numbers in the documentation use English formatting (`1,838`, `98.88%`). -- Chart and report text: percentages without a space before `%`, readable labels for every code. - -### Not changed -- Schema field names, extension columns, controlled vocabularies and geolocation codes are the - data format of the panel and stay as they are; their English meaning is in the new glossary of - `docs/schema.md`. -- Every number reported in the article and in the Supplement is reproduced unchanged; the - expected-value files under `reproducibility/expected/` carry the same values under English keys. - -## 1.6.0 (2026-10-24) - -### Added -- **Charts** (`gridqueue.plots`, optional extra `pip install "gridqueue[plot]"`): field coverage, - quality rules, column-shift audit, stage of the proceeding, requested capacity by class and - voltage, Kaplan-Meier processing-time curves with censoring and a 95% band, node loading, - flows / transitions / linkage audit of two editions, location resolution, a four-panel - overview, and `save_all`, which draws every chart the data supports and lists the skipped ones - with the reason. A chart is a view of a result, never a second computation of it. Figures are - created without `pyplot`, so the module runs unchanged in scripts, notebooks and the browser. -- **CLI**: `gridqueue compare` (flows, transitions, linkage precision and sensitivity of two - editions) and `gridqueue plot` (every chart, PNG/SVG/PDF). -- `parse_sources`: any number of files of one or several publishers into one panel; every file is - parsed by a fresh adapter instance, so each document keeps the as-at date it states. - `split_editions` separates two or more editions of one register dropped together. -- `kaplan_meier_by_group`: Kaplan-Meier summary per group (or per combination of columns) with the - inclusion criterion of the article. -- `geocode_frame`: writes the resolver's result back to the panel (`wspolrzedne` as `"lat,lon"`, - `_geo_pewnosc`, `_geo_dokladnosc`, `_geo_powod`); source coordinates are kept. -- `node_loading(..., napiecie=...)`: voltage filter of the queue. `napiecie="110"` is the base - variant of the article (median 6.500 / 4.688, 29 of 36 / 33 of 37 for ENERGA-OPERATOR, - 31 August 2026); the default is the all-voltage upper variant (8.611 / 7.638). -- `gridqueue.examples`: a synthetic panel with node rows, two synthetic editions and a small - gazetteer, labelled as synthetic, for trying every function without a real disclosure. -- **Browser page, simple view** (the default): drop the documents and get the main charts, a few - sentences with the key numbers, and one ZIP with charts, data and report. Two editions of one - register dropped together are recognised by their as-at dates and compared. -- **Browser page, full view**: several files at once, manual choice of the adapter, a later edition - (flows, transitions, linkage audit), an optional gazetteer, schema validation and the column-shift - audit, processing times and node loading, all charts (PNG 300 dpi and SVG), and export to CSV, - XLSX, JSON and one ZIP archive; two synthetic demonstration modes; a table of what runs where. -- **Colab notebook**: installs 1.6.0 with charts; adds schema validation, the column-shift audit, - key sensitivity (checked against the article), ENERGA-OPERATOR processing times and node loading - (both checked against the Supplementary Material), the resolver on a synthetic gazetteer, all - charts and a downloadable results archive. - -### Fixed -- **Censoring date of `processing_time`.** The default censored every open application at the - single latest publication date of the frame, node rows included. ENERGA-OPERATOR publishes its - applications as at 30 June and its node capacities as at 31 August 2026, so every censored time - was two months too long (median for consumption at 110 kV 256 d instead of 240 d). Each row is - now censored at its own publication date, which is also correct for panels of several publishers. -- **Interval of the Kaplan-Meier median**: the lower and upper bounds were swapped - (`mediana_dolna > mediana > mediana_gorna`). -- `kaplan_meier` raised `KeyError` for a group in which every observation is censored. - -## 1.5.2 (2026-10-17) - -### Added -- **Running in the browser** (`docs/app/`): a page that loads this same package - into a WebAssembly interpreter and runs it locally. The document is not - sent anywhere, because there is no server that could receive it. The result is by - construction identical with the result of the command line, because it is the same code and the same - version. The page detects the publisher, extracts the schema rows, runs the eight - quality rules and allows the table to be downloaded as CSV and the validation report as JSON. - -### Fixed -- **Rule R3 returned -1 instead of a count of violations** when the `publikujacy` column - was present but empty: `groupby` omits a group with an empty value, so a row - without that annotation had no entry in the dictionary of disclosed fields and a lookup by - key raised `KeyError`. The error was caught, the rule reported -1, - and the total number of violations in the whole report became negative. A silent defect, detected - by running the package on a frame built by hand while the browser - page was being built. A regression test was added. - -### Changed -- The em dash and en dash were removed from the prose of the repository. The only remaining occurrence - is in the regular expression of the TAURON adapter and is load-bearing: it matches a character - occurring in the source document. - -## 1.5.1 (2026-10-08) - -### Corrected -- **The arithmetic identity of the Spanish document described correctly.** Version 1.5.0 - stated that the columns "with permission" and "in progress" reconstruct the occupied capacity in 395 of 767 - rows. That was **a description of an incomplete identity**: both groups are COMPONENTS, not - alternative breakdowns. Eleven subcolumns, nine substation items plus those - two, sum to the occupied capacity **in exactly 1,838 of 1,838 rows, in both - editions checked**. The previous figure was true, but it measured a fragment. - -### Added -- `total_occupied_capacity_components` and the field `identity_sum_occupied_capacity` in the report of the - Spanish adapter. For this publisher **there is no reference set** read - independently of the parser, so the extraction accuracy cannot be measured against a reference set. - Arithmetic closure is evidence of **consistency of reading**: for the sum of eleven cells - to agree with the twelfth, all twelve have to be read correctly. Across two - editions that gives 3,676 independent checks, all of them closed. This is - evidence weaker than a reference set and it is described as such. -- `NODE_KEY_FIELDS`, a content key for rows of the NODE entity. The default key is - built for the APPLICATION entity and on node rows it **falls apart**, because the application fields - are empty there. Measured on a pair of Spanish issues: with the default key 916 of 1,838 - rows fall into a collision and 660 are rejected as ambiguous; with the node - key the collisions number **zero**, all 1,838 pairs are linked unambiguously, - at a precision and recall of **1,000**. The more general conclusion: a content key has to be - matched to the entity just as the schema core is, and a mismatch **does not report - an error**, it quietly produces collisions that look like events in the register. - -## 1.5.0 (2026-10-08) - -### Added -- The `es_edistribucion` adapter, **the first foreign source issued as a document - meant for printing**, which closes the strongest missing test of transferability. A monthly issue, - 28 pages, 1,838 rows of the NODE entity, all declared fields populated - at 100 %, including the geographic coordinates of every substation. Capacities in MW of active power. -- The `ie_esbnetworks` adapter, the largest source in the panel, 46,527 rows, the only one - going below high voltage substations. Coordinates in 46,525 rows. -- Reference panel: **68,054 rows, nine publishers, five jurisdictions** - (PL, GB, PT, ES, IE). The NODE entity grows from 634 to 48,999 rows, and the coverage of the - coordinate field from 6.2 % to 72.9 % of the panel. - -### Settled -- **Units, for the second and third time.** The Irish source mixes, in one file, apparent power - on the offtake side with active power on the generating side; the capacity field carries the generating side - converted from MW, and the offtake side stays in the extensions with the unit in the name. The Spanish - source reports MW of active power, so it converts without assumptions. Three jurisdictions - show three different unit practices in one schema field. -- **The semantics of the Spanish columns established by arithmetic, not by the header.** A header - with merged cells suggested that two columns belonged to the unsettled capacity; - the sums showed that they belong to the occupied capacity. Taking the header on trust would have given a silent - transposition of two capacity fields. **Made precise in 1.5.1**, see below. -- **An empty constraint cell in the Irish source** is read as no constraint. - The publisher does not describe this convention, so it is an interpretation, not the content of the source: - the raw phrase stays in an extension, and the decision is pinned by a test. - -### Changed -- The registry test covers nine adapters in five jurisdictions. -- `reproducibility/manifest.csv` counts 16 entries. - -## 1.4.0 (2026-10-07) - -### Added -- The `pt_eredes` adapter, the reception capability of the national distribution network of Portugal, - 469 rows of the NODE entity, published quarterly. **The third jurisdiction in the panel - and the third mode of issue**: not a document meant for printing and not a file on a page, but a dataset - on an open data portal with a REST interface. -- The first foreign source of the NODE entity. That entity grows in the panel from 165 rows - to 634, and both jurisdictions satisfy its core **through different members of the same - alternative**: Poland with available capacity (58% of the node rows), Portugal with the - constraint flag (100%). The conditional core introduced in version 1.0 for a Polish - category error turned out to be the condition for absorbing a foreign jurisdiction. - -### Settled -- **Units.** The Portuguese publisher reports capacities in MVA, that is in apparent power, - whereas the canonical unit of the schema is kW of active power. Conversion would require - an undisclosed power factor, so `moc_dostepna` remains empty, and the source - values go into the extension fields with the unit recorded explicitly. This is the same - principle that was applied to the publisher reporting a count of free connection - slots instead of a capacity. - -### Fixed -- **The adapter declaration test carried an out-of-date core contract.** It required - the core of the APPLICATION entity from every adapter, that is it repeated the category error fixed - in rule R7 in version 1.0: a publisher discharging only point 2 of the duty - discloses nodes rather than applications, so it has neither an application identifier nor an application capacity. - The defect was revealed only by the Portuguese adapter. The test now checks the core **appropriate - to the adapter's entity**. - -### Changed -- `reproducibility/manifest.csv` covers the Portuguese export with a checksum. -- The reference panel counts 19,689 rows from seven publishers in three jurisdictions. - - -## 1.3.0 (2026-10-07) - -### Added -- The `analysis` module with a research layer above the schema: `station_node_map`, - `assign_to_nodes`, `node_loading`, `processing_time`, `kaplan_meier`. - It makes it possible to compute **node loading**, the ratio of the active capacity of the queue to - the disclosed available capacity, that is the criterion which literature reviews - describe as unmeasurable because of the absence of the data. -- `linkage_recall`, the **recall** of the longitudinal linkage, alongside the precision - measured so far. On a pair of editions of the transmission operator the recall of the key is - 0.9652: of the 29 missed pairs, 10 are broken by the notation of the location, 15 by a change of capacity, - 4 by a change of voltage. -- `surrogate_key(..., zero_as_missing=True)`, an option equating zero with a missing - value. It recovers 12 of the 51 pairs lost by the key. **Disabled - by default**: zero is a disclosed value and a gap is a non-disclosure of it, so - equating them is a decision of the analyst and is meant to be visible in the code. -- `schema_table()` returns the columns `entity` and `core_status`; the core is - conditional per entity, which the two-valued `core` column did not convey. -- `reproducibility/fetch_sources.py`, downloads the source documents from the manifest - and verifies the SHA-256 checksums. It supports a custom bundle of trusted issuers through - `GRIDQUEUE_CA_BUNDLE`, without disabling verification. - -### Fixed -- **Name normalization split words containing the letter `ł`.** The canonical Unicode - decomposition does not separate the diacritic from it, because it is a separate letter, - so cleaning the characters turned it into a space and "Białogard" became - "bia ogard". This also concerns `đ`, `ø`, `ß`, `æ`, `œ`. It is the second occurrence - of the same trap in the project; the first concerned the harmonization of statuses. -- **Objects planned on a line were arbitrarily assigned to a node.** The entry - "planowany RS w linii relacji A, B" describes an object lying between nodes; - matching by name assigned its full capacity to whichever end happened - to appear in the list of substations. Such applications now have their own status `LINE` - and are not assigned. On the data of one publisher this concerns 63 applications. - -### Changed -- `reproducibility/manifest.csv` covers the July edition of the disclosure of the transmission - operator, used in the longitudinal analysis. -- `DATA_LICENSE.md` explains why the documents are not redistributed, - and records that the link of one of the publishers stopped leading to the file, - which the download script recognizes by the signature of the file and reports separately. - - -## 1.2.1 (2026-10-04) - -### Added -- `notebooks/gridqueue_colab.ipynb`, a full research run in the browser, with no installation: - downloading two editions directly from the publisher, extraction, quality control, queue flows - with the linkage precision, and **an assertion of agreement with the figures published in the paper**. - Documentation: `docs/colab.md`. -- `QualityReport.as_frame()`, a method **promised by the documentation but absent from the code**. - The defect came to light only when the notebook was run, since it used the public API - the way the documentation describes it. -- Two continuous integration jobs: a run on `pandas>=3.0` and a check of the notebook - (syntactic correctness of the cells and the existence in the API of every name imported from it). - -### Fixed -- **Incompatibility with pandas 3.** The longitudinal key was built using `.astype(str)`, which in pandas 2 - turned a gap into the string `"nan"`, whereas in pandas 3 it preserves the missing value; concatenating - the components raised `TypeError` and the whole longitudinal analysis was impossible on that version. - The key is now built by an explicit conversion, identically in both versions; **all the published - figures remain unchanged**, which was checked by comparison. The same fix makes the rule - "a gap is a value" a property of the code rather than a side effect of a type cast. - An analogous assumption was removed from the adapter of the transmission operator. - -### Changed -- The CI matrix tested Python versions but not library versions, which is how the defect got through. - The `pandas3` job closes that gap. - -## 1.2.0 (2026-10-04) - -### Added -- `linkage_audit`, an assessment of the precision of the longitudinal linkage on nine fields - **excluded from the key** (entity name, object name, application date, six fields of the - capacity breakdown). The key does not enforce the agreement of these fields, so their agreement is - independent evidence. On the pair of PSE editions the precision is 98.88-99.63% - across all 804 pairs. -- `EditionDelta` now reports key collisions and the number of rows rejected - from the linkage as ambiguous. -- `DATA_LICENSE.md`, separating the licence of the code from the status of the input - and output data, including the duties following from ODbL for the gazetteer. -- `CHANGELOG.md`. -- Status badges in `README.md`. -- `reproducibility/expected/`, the expected results together with assertions in the reproducing - scripts; a divergence ends the run with a non-zero code. - -### Changed -- The documentation of the longitudinal analysis gives the full specification of the linkage algorithm: - behaviour on a missing value, the uniqueness requirement, the handling of collisions, the absence of tolerance - on the capacities. -- The longitudinal example prints the linkage precision and the collision counts. - -## 1.1.0 (2026-10-03) - -### Added -- The `longitudinal` module: `compare_editions`, `surrogate_key`, `EditionDelta`. - The register does not publish a persistent application identifier, so linkage between - editions goes by a content key. -- `examples/longitudinal_use_case.py` and `docs/longitudinal.md`. - -### Fixed -- **Multiple capacity cells.** When an application concerns several connection points, - the publisher enters several values into one cell, separated by a line break. - Whitespace normalization turned the line break into a space, and the capacity conversion - treats a space as a thousands separator, so "129⏎240" (129 MW + 240 MW) became - 129,240 MW. In the edition with semicolons the same cell did not convert at all. - The capacity is now split on the line break and on the semicolon, and deliberately not on the space; - the components are summed and the split is preserved in the extensions. - -## 1.0.0 (2026-10-03) - -### Changed -- **The schema core is conditional and separate for each entity.** The core completeness - rule required the `moc_pobierana` field unconditionally and applied the requirements of the - APPLICATION entity to rows of the NODE entity as well. Both decisions turned a property of the schema - into an apparent violation of the disclosure. Violations of rule R7 number 771 (4.011%) instead of - 5,390 (28.044%). -- `SCHEMA_VERSION` was introduced, exposed in the validation report and in the public API. - -### Added -- `examples/research_use_case.py`, the `reproducibility/` directory, `CITATION.cff`, - a continuous integration workflow on three systems and four Python versions. +## 1.0.0 (2026-10-24) + +First public release. + +### Added +- **Schema** of 21 fields derived from the structure of the duty in Art. 7(8l) of the Polish Energy + Law, with two entities (`APPLICATION`, `NODE`), controlled vocabularies of process status and + resource class, canonical units (kW, ISO dates) and validation (`validate_frame`). +- **Nine adapters in five jurisdictions**: PSE, ENERGA-OPERATOR, TAURON Dystrybucja, Stoen + Operator, Boryszew (Poland), E-REDES (Portugal), e-distribución (Spain), ESB Networks (Ireland) + and National Grid (Great Britain); automatic publisher detection and `parse_sources` for several + documents and editions at once. +- **Quality layer**: eight rules, a column-shift audit and per-publisher reports. +- **Longitudinal module**: comparison of two editions (flows, status transitions, monotone-snapshot + test), a linkage audit with precision bounds and the sensitivity of the linkage key. +- **Analytical layer**: processing times with censoring (Kaplan-Meier, per-group medians and + intervals) and node loading, with a voltage filter. +- **Location resolver**: gazetteer-based resolution of place descriptions with declared confidence, + accuracy and reason, written back to the frame by `geocode_frame`. +- **Charts** for every result (`gridqueue.plots`, extra `pip install "gridqueue[plot]"`). +- **Three access routes**: the Python package with a command-line interface, a Google Colab notebook + that reproduces the results of the article and stops on any discrepancy, and a browser application + (Pyodide/WebAssembly) with a simple and a full view, which does not upload the document. +- **Names in the wording of the statute**: `from_legacy_names` and the correspondence table in + `docs/schema.md` convert tables that use the Polish field names of the Supplementary Material. +- Reproducibility material: source manifest with checksums, download and validation scripts, and + the expected values of the article. diff --git a/CITATION.cff b/CITATION.cff index 56719f5..51101fe 100644 --- a/CITATION.cff +++ b/CITATION.cff @@ -7,7 +7,7 @@ abstract: >- and the corresponding registers from other jurisdictions into a single set of rows in a 21-field schema derived from the structure of the legal obligation. type: software -version: "1.7.0" +version: "1.0.0" date-released: "2026-10-24" license: Apache-2.0 repository-code: "https://github.com/s-matysik/gridqueue" diff --git a/CODE_METADATA.md b/CODE_METADATA.md index cc1d9d8..67e51c9 100644 --- a/CODE_METADATA.md +++ b/CODE_METADATA.md @@ -8,9 +8,9 @@ listed again in the final section. | Nr | Item | Value | |----|------|-------| -| C1 | Current code version | `1.7.0` (declared identically in `pyproject.toml` and in `gridqueue.__version__`; CI asserts the two agree on every push) | -| C2 | Permanent link to code / repository used for this code version | https://github.com/s-matysik/gridqueue (tag `v1.7.0`) | -| C3 | Permanent link to Reproducible Capsule | ⟦TEAM DECISION⟧, archive DOI. The repository is ready for a Zenodo deposit of tag `v1.6.0`; enabling the integration requires the team's account | +| C1 | Current code version | `1.0.0` (declared identically in `pyproject.toml` and in `gridqueue.__version__`; CI asserts the two agree on every push) | +| C2 | Permanent link to code / repository used for this code version | https://github.com/s-matysik/gridqueue (tag `v1.0.0`) | +| C3 | Permanent link to Reproducible Capsule | ⟦TEAM DECISION⟧, archive DOI. The repository is ready for a Zenodo deposit of tag `v1.0.0`; enabling the integration requires the team's account | | C4 | Legal Code License | Apache-2.0, `LICENSE.txt` at the distribution root. Input data licences are fixed and separate: OpenStreetMap gazetteer under ODbL 1.0; operators' statutory publications under the terms each publisher applies to its own site | | C5 | Code versioning system used | git | | C6 | Software code languages, tools and services used | Python, `requires-python >= 3.10`; build backend `setuptools.build_meta` (`setuptools>=68`); test runner `pytest` | @@ -22,8 +22,8 @@ listed again in the final section. | Nr | Item | Value | |----|------|-------| -| S1 | Current software version | `1.7.0`, matching C1 | -| S2 | Permanent link to executables of this version | Source distribution only, `gridqueue-1.7.0.tar.gz`, pure Python, no binaries, no compiled extensions | +| S1 | Current software version | `1.0.0`, matching C1 | +| S2 | Permanent link to executables of this version | Source distribution only, `gridqueue-1.0.0.tar.gz`, pure Python, no binaries, no compiled extensions | | S3 | Legal Software License | Apache-2.0, same licence as C4 | | S4 | Computing platforms / Operating Systems | Linux, macOS, Windows × Python 3.10, 3.11, 3.12, 3.13, a twelve-cell matrix exercised by GitHub Actions (`.github/workflows/ci.yml`) on every push and pull request. The measured figures reported in the paper were produced on macOS/arm64 with Python 3.11.15 | | S5 | Installation requirements & dependencies | As C7. No GPU, no network access required at run time once the source documents and the gazetteer are local | @@ -56,28 +56,28 @@ not. 11 process statuses. Verified by measured coverage across the six packaged row-level sources (`gridqueue_coverage_matrix_6publ.csv`): ENERGA-OPERATOR 17/21, PSE 17/21, TAURON 15/21, Stoen 13/21, National Grid (GB) 10/21, Boryszew 8/21. **All 21 schema fields are populated - somewhere in the 19,220-row panel.** `moc_dostepna` is closed from sources by 96 ENERGA node + somewhere in the 19,220-row panel.** `capacity_available` is closed from sources by 96 ENERGA node rows (0-175,000 kW); PSE discharges the same obligation point as a **count of free connection - slots rather than a power figure** (69 switchgears), mapped to `ograniczenie_flaga` and an - extension field, leaving `moc_dostepna` empty rather than fabricating a unit. `wspolrzedne` is + slots rather than a power figure** (69 switchgears), mapped to `constraint_flag` and an + extension field, leaving `capacity_available` empty rather than fabricating a unit. `coordinates` is populated by derivation from the place name, not by disclosure: no publisher reports coordinates. Extensions across publishers are complementary, not conflicting. - **Resolved in 1.0.** The core no longer requires `moc_pobierana` unconditionally. It requires the + **Resolved in 1.0.** The core no longer requires `capacity_withdrawn` unconditionally. It requires the row's OWN entity core: for a APPLICATION row four unconditional fields plus **at least one** of - (`moc_pobierana`, `moc_wprowadzana`); for a WEZEL row `id_wezla` plus at least one of - (`moc_dostepna`, `ograniczenie_flaga`). Rule R7 therefore fell from 5,390 violations (28.044%) + (`capacity_withdrawn`, `capacity_injected`); for a NODE row `node_id` plus at least one of + (`capacity_available`, `constraint_flag`). Rule R7 therefore fell from 5,390 violations (28.044%) to 771 (4.011%), and rows with a complete core rose from 13,830 to 18,449 of 19,220. What remains is a genuine disclosure gap, not a schema artefact: 570 applications with no power figure at all, 202 with no location text, 2 nodes with neither available capacity nor a constraint flag. - Superseded text follows for the record: the core required `moc_pobierana`, which purely generating applications do + Superseded text follows for the record: the core required `capacity_withdrawn`, which purely generating applications do not carry, 5,390 of 19,220 rows (28.044%) in 0.3.x, the whole of rule R7 at that time. The core should require either capacity, consumed or injected. 2. Every location resolution carries an explicit confidence level on a six-point scale (`EXACT`, - `UNIQUE_TOKEN`, `TOKEN_SUBSET`, `REVERSED`, `SKRZYZOWANIE`, `NONE` with a reason), plus a second, + `UNIQUE_TOKEN`, `TOKEN_SUBSET`, `REVERSED`, `INTERSECTION`, `NONE` with a reason), plus a second, independent placement-accuracy level (address point, intersection point, point on the street axis, locality centroid, no geometry, none). A failure is returned as missing-with-reason, never as a guess. Street identification, Stoen Warsaw subset (1,179 positions): `EXACT` 884, - `UNIQUE_TOKEN` 152, `SKRZYZOWANIE` 70, `REVERSED` 54, `TOKEN_SUBSET` 5, `NONE` 14-1,165 of + `UNIQUE_TOKEN` 152, `INTERSECTION` 70, `REVERSED` 54, `TOKEN_SUBSET` 5, `NONE` 14-1,165 of 1,179 resolved, 98.81%; without the intersection rule 1,095 of 1,179 = 92.88%; without both the inverted-order and the intersection rule 1,041 of 1,179 = 88.30%. Coordinate coverage after the gazetteer was rebuilt with geometry (13,149 street names in 25 @@ -113,7 +113,7 @@ blocker is server-side client filtering, the site returns a rejection page to th control the operator put in place deliberately, so it was not attempted. Electricity North West: rows behind the same access control as UK Power Networks; metadata only. -**Size.** Read from the built source distribution `gridqueue-1.7.0.tar.gz`: 7 core modules under `src/gridqueue`, 2,125 lines, `geoloc.py` 672, `schema.py` 480, `quality.py` 370, `layout.py` 353, `cli.py` 148, `registry.py` 64, `__init__.py` 38, and 9 adapter modules, 1,777 lines (`pl_pse.py` 303, `pl_energa.py` 282, `_ptpiree.py` 278, `pl_boryszew.py` 232, `pl_stoen.py` 194, `uk_nationalgrid.py` 166, `pl_tauron.py` 161, `base.py` 156, `__init__.py` 5); 3,902 lines in total under `src`. `_ptpiree.py` is shared by the PSE and ENERGA adapters because both publishers fill the same agreed industry template, one as a spreadsheet and one as a print document. Tests: 7 files, 1,040 lines (`test_geoloc_coordinates.py` 280, `test_schema.py` 164, `test_pse_energa.py` 147, `test_adapters.py` 144, `test_quality.py` 124, `test_geoloc.py` 95, `test_layout.py` 86). Runnable workflows shipped alongside the package: `examples/research_use_case.py` 168 lines, and `reproducibility/` 382 lines (`run_figures.py` 248, `run_validation.py` 134). `python -m pytest -q`: **153 passed, 2 skipped (155 collected), 0 failed, 0 errors**, the two skips are the PSE and ENERGA integration tests, which require the source documents and are skipped when those are absent from the working tree. The twelve-cell OS × Python matrix in CI runs the same suite. +**Size.** Read from the built source distribution `gridqueue-1.0.0.tar.gz`: 7 core modules under `src/gridqueue`, 2,125 lines, `geoloc.py` 672, `schema.py` 480, `quality.py` 370, `layout.py` 353, `cli.py` 148, `registry.py` 64, `__init__.py` 38, and 9 adapter modules, 1,777 lines (`pl_pse.py` 303, `pl_energa.py` 282, `_ptpiree.py` 278, `pl_boryszew.py` 232, `pl_stoen.py` 194, `uk_nationalgrid.py` 166, `pl_tauron.py` 161, `base.py` 156, `__init__.py` 5); 3,902 lines in total under `src`. `_ptpiree.py` is shared by the PSE and ENERGA adapters because both publishers fill the same agreed industry template, one as a spreadsheet and one as a print document. Tests: 7 files, 1,040 lines (`test_geoloc_coordinates.py` 280, `test_schema.py` 164, `test_pse_energa.py` 147, `test_adapters.py` 144, `test_quality.py` 124, `test_geoloc.py` 95, `test_layout.py` 86). Runnable workflows shipped alongside the package: `examples/research_use_case.py` 168 lines, and `reproducibility/` 382 lines (`run_figures.py` 248, `run_validation.py` 134). `python -m pytest -q`: **153 passed, 2 skipped (155 collected), 0 failed, 0 errors**, the two skips are the PSE and ENERGA integration tests, which require the source documents and are skipped when those are absent from the working tree. The twelve-cell OS × Python matrix in CI runs the same suite. **Functionalities.** Document ingest with per-row provenance; layout-based extraction; unit normalisation to the canonical kW (MW→kW, with the source unit retained in an extension column); resource-class and process-status vocabulary mapping; composite-date diff --git a/DATA_LICENSE.md b/DATA_LICENSE.md index 35d212c..522f611 100644 --- a/DATA_LICENSE.md +++ b/DATA_LICENSE.md @@ -42,10 +42,10 @@ with coordinates attached, made available onwards, does trigger share-alike. ## 4. The resulting panel The panel produced by the package is a work derived from both of the layers above. -To the extent that it contains **values derived from the gazetteer** (the `wspolrzedne` field +To the extent that it contains **values derived from the gazetteer** (the `coordinates` field and the resolver's diagnostic columns), it is subject to ODbL. To the extent that it contains only content transcribed from official documents, it is subject to the status of those documents. -The `_geo_pewnosc` column makes it possible to separate those two layers: rows with no resolved +The `_geo_confidence` column makes it possible to separate those two layers: rows with no resolved location contain no content originating from OpenStreetMap. ## 5. Personal data @@ -85,7 +85,7 @@ more current it looked. The consequence has to be stated openly, because it is inconvenient: **byte-for-byte reproduction ceases to be possible once a publisher replaces or moves a document.** -While release 1.3.0 was being prepared, the link to the disclosure of one of the operators +While the package was being prepared, the link to the disclosure of one of the operators stopped leading to the file and began returning a page of the service; the download script recognizes that case and reports it separately instead of saving the page as a document. That is why the checksums are in the manifest: a mismatch is a signal that diff --git a/README.md b/README.md index b82d99c..5736304 100644 --- a/README.md +++ b/README.md @@ -17,14 +17,15 @@ public, but it is not processable. `gridqueue` turns those publications into one set of rows with **measured accuracy**. -Release 1.7.0: **nine adapters in five jurisdictions** (Poland, Spain, Ireland, Portugal, Great +Release 1.0.0: **nine adapters in five jurisdictions** (Poland, Spain, Ireland, Portugal, Great Britain), all 21 schema fields populated by at least one publisher, quality rules, a longitudinal module with a linkage audit, processing times with censoring, node loading, a location resolver and **charts for every result**. Documentation: **https://s-matysik.github.io/gridqueue/** -The code and documentation are in English. Schema field names and codes (`id_wniosku`, -`WARUNKI_WYDANE`, ...) keep the wording of the Polish statute, because they are the data format of -the panel; their English meaning is in the [glossary](https://s-matysik.github.io/gridqueue/schema.html#glossary-of-field-and-code-names). +The code, the schema and the documentation are in English. The Supplementary Material of the +article names the fields in the Polish wording of the statute (`id_wniosku`, `moc_wprowadzana`, ...); +the [correspondence table](https://s-matysik.github.io/gridqueue/schema.html#names-in-the-wording-of-the-statute) +gives the English names, and `gridqueue.from_legacy_names(df)` converts such a table. ## Three ways to run it @@ -79,7 +80,7 @@ resolver is shown on a synthetic gazetteer, with a one-line change to use your o One command, with no compilation and no external services: ```bash -pip install "gridqueue[plot] @ git+https://github.com/s-matysik/gridqueue@v1.7.0" +pip install "gridqueue[plot] @ git+https://github.com/s-matysik/gridqueue@v1.0.0" ``` The `plot` extra adds matplotlib for the charts; without it everything else works. @@ -114,7 +115,7 @@ from gridqueue import resolver_from_osm_json r = resolver_from_osm_json("streets_warsaw.json") res = r.resolve("WARSZAWA, WINCENTEGO ŚW.") print(res.matched, res.confidence) # Świętego Wincentego REVERSED -print(r.resolve("SKB3").reason) # kod_stacji_bez_slownika, we do not guess +print(r.resolve("SKB3").reason) # station_code_without_dictionary, we do not guess ``` ## CLI @@ -156,7 +157,7 @@ skipped node with the reason), never as zero. from gridqueue import parse_sources, plots panel, reports = parse_sources(["applications.pdf", "demand_capacity.pdf", "generation_capacity.pdf"]) -plots.plot_processing_time(panel, by="klasa_zasobu") # displays in Jupyter / Colab +plots.plot_processing_time(panel, by="resource_class") # displays in Jupyter / Colab plots.save_all(panel, "charts/", formats=("png", "svg"), dpi=300) ``` @@ -166,8 +167,8 @@ as synthetic, for trying every function without a real disclosure. ## Schema 21 fields in three groups reflecting the structure of the duty (IDENTITY / CAPACITY / PROCESS), -a mandatory core of 5 fields (`id_wniosku`, `lokalizacja_tekst`, `klasa_zasobu`, `moc_pobierana`, -`status_procesu`), the rest optional and **declared per publisher**. Canonical units: +a mandatory core of 5 fields (`application_id`, `location_text`, `resource_class`, `capacity_withdrawn`, +`process_status`), the rest optional and **declared per publisher**. Canonical units: capacity in kW, ISO 8601 dates, WGS84 coordinates. Fields beyond the schema, which a given operator supplies beyond the required minimum, go into columns with the `_` prefix and are declared explicitly by the adapter (`adapter.declaration()`). @@ -290,9 +291,9 @@ detected by comparing the output with the reference set read from the document. raw agreement 0,9725, Cohen's kappa 0,9690, zero perceptual discrepancies. The PSE and ENERGA reference sets remain single-annotator. * The location at TAURON is an internal substation code; the operator does not publish a dictionary of - the codes, so the resolver returns missing with the reason `kod_stacji_bez_slownika`. It does not + the codes, so the resolver returns missing with the reason `station_code_without_dictionary`. It does not guess. -* **None of the publishers reports coordinates.** The `wspolrzedne` field is populated by +* **None of the publishers reports coordinates.** The `coordinates` field is populated by derivation from the place name against a gazetteer with geometry (13,149 names in 25 localities): 1,225 of 19,220 panel rows. The evidential value of this field is therefore different from that of the fields transcribed from the document, and it is described as such in the result. diff --git a/docs/adapters.md b/docs/adapters.md index a8160c8..d02d8b6 100644 --- a/docs/adapters.md +++ b/docs/adapters.md @@ -29,8 +29,8 @@ from gridqueue.adapters.base import Adapter, ParseResult class MojAdapter(Adapter): key = "pl_moj_operator" - publikujacy = "Nazwa Operatora S.A." - jurysdykcja = "PL" + publisher = "Nazwa Operatora S.A." + jurisdiction = "PL" detekcja = [r"moj_operator", r"nazwa_operatora"] # file name patterns def parse(self, source) -> ParseResult: @@ -73,23 +73,23 @@ apparent power. The canonical unit of the schema is **kW of active power**. Thes physical quantities, and converting one into the other requires a power factor, which the publisher does not report. -That is why `moc_dostepna` remains **empty**, and the values in MVA go into the +That is why `capacity_available` remains **empty**, and the values in MVA go into the extension fields together with the source unit recorded explicitly: | extension field | meaning | |---|---| -| `_jednostka_mocy_zrodlowa` | `MVA` | -| `_moc_dostepna_mva` | MV+HV reception capability | -| `_moc_przylaczona_mva` | capacity already connected | -| `_moc_zakontraktowana_mva` | contracted capacity | -| `_moc_w_potwierdzaniu_mva` | capacity under confirmation | +| `_source_capacity_unit` | `MVA` | +| `_capacity_available_mva` | MV+HV reception capability | +| `_capacity_connected_mva` | capacity already connected | +| `_capacity_contracted_mva` | contracted capacity | +| `_capacity_being_confirmed_mva` | capacity under confirmation | This is the same principle that was applied to the publisher reporting a count of free connection slots instead of a capacity: entering a count of slots into a field with a different unit would be fabrication, not harmonization. The core of the NODE entity is nevertheless satisfied, because it is discharged by the second member of -the alternative, `ograniczenie_flaga`, set on the basis of a zero reception capability. Zero is +the alternative, `constraint_flag`, set on the basis of a zero reception capability. Zero is a **disclosed** value and means a constraint; an empty cell means non-disclosure and does not set the flag. This concerns 222 of 469 nodes. @@ -101,7 +101,7 @@ a Polish category error, turned out to be the condition for absorbing a foreign ```python from gridqueue import get_adapter r = get_adapter("pt_eredes").parse("pt_eredes_capacidade_rececao_rnd_2T2026.csv") -r.frame[["id_wezla", "nazwa_wezla", "lokalizacja_tekst", "ograniczenie_flaga"]] +r.frame[["node_id", "node_name", "location_text", "constraint_flag"]] r.report["note_units"] ``` @@ -150,7 +150,7 @@ transformers at a substation. Coordinates are present in 46,525 rows. ### Two units in one file The publisher reports the **offtake side in MVA** (apparent power) and the **generating side in MW** -(active power). The `moc_dostepna` field therefore carries the available **firm generating** +(active power). The `capacity_available` field therefore carries the available **firm generating** capacity, converted from MW to kW, while the offtake side stays in the extensions with the unit in the field name. Summing MW with MVA would give a figure with no physical sense, and choosing one side without saying so openly would be hiding a decision. diff --git a/docs/analysis.md b/docs/analysis.md index b0fcb12..1aadfee 100644 --- a/docs/analysis.md +++ b/docs/analysis.md @@ -21,8 +21,8 @@ from gridqueue import station_node_map, assign_to_nodes, node_loading mapping = station_node_map(panel) # station -> group of stations panel = assign_to_nodes(panel, mapping) # assignment of applications to groups -L = node_loading(panel, capacity_field="moc_wprowadzana") -L[["nazwa_wezla", "applications", "capacity_requested", "moc_dostepna", "loading"]] +L = node_loading(panel, capacity_field="capacity_injected") +L[["node_name", "applications", "capacity_requested", "capacity_available", "loading"]] ``` **The key is the group of stations, not the node row.** The publisher lists the same diff --git a/docs/app/gridqueue-1.0.0-py3-none-any.whl b/docs/app/gridqueue-1.0.0-py3-none-any.whl new file mode 100644 index 0000000..a637c26 Binary files /dev/null and b/docs/app/gridqueue-1.0.0-py3-none-any.whl differ diff --git a/docs/app/gridqueue-1.7.0-py3-none-any.whl b/docs/app/gridqueue-1.7.0-py3-none-any.whl deleted file mode 100644 index 3276751..0000000 Binary files a/docs/app/gridqueue-1.7.0-py3-none-any.whl and /dev/null differ diff --git a/docs/app/index.html b/docs/app/index.html index 6162987..e461848 100644 --- a/docs/app/index.html +++ b/docs/app/index.html @@ -569,7 +569,7 @@

What runs where

3 The Parquet engine has no WebAssembly build; use the CSV or XLSX export, or the notebook.

For everything at once, open the Colab notebook, - or install the package with pip install "gridqueue[plot] @ git+https://github.com/s-matysik/gridqueue@v1.7.0".

+ or install the package with pip install "gridqueue[plot] @ git+https://github.com/s-matysik/gridqueue@v1.0.0".