license: derive a license from the free-form License field - #40
Merged
Conversation
Types now consults the legacy free-form License field when neither License-Expression nor a License:: classifier says anything. That population previously derived to Unknown. The tier is strict by design: one unrecognized token rejects the whole field. It resolves only when the field is itself a published, non-deprecated SPDX identifier, or an expression whose every identifier is one. That is what makes it impossible to invent a license here -- "3-Clause BSD License" tokenizes to 3-Clause / BSD / License, and accepting the plausible part would attribute BSD-3-Clause to a package on the strength of a word. The same rule rejects bare GPL, GPLv3 and BSD, and defers the alias tier (Apache 2.0, MIT License) without anyone having to adjudicate it. Because the tier runs last it can only move a package out of Unknown, never overrule a license declared through a structured field. Identifiers are now checked against the published SPDX License List rather than an ad hoc set, with deprecated ids excluded. Measured against the live PyPI corpus of 1,745,000 free-form-only versions captured 2026-08-18: 48.35% resolve. The 250 highest-volume strings are committed as a regression fixture. Types takes a third argument and is a breaking change for callers.
Review found a wrong-license attribution reachable from real PyPI metadata: a package whose only structured declaration was "License :: Other/Proprietary License" derived to MIT if its free-form License field said MIT. StandardizeLicensePyPIClassifiers collapses the 17 classifiers SPDX has no identifier for to Unknown, so an unmappable classifier arrived at the free-form tier looking identical to a package that declared nothing. But such a classifier still says something, and "proprietary" in particular says the opposite of what the free-form string claimed. Suppress the tier on the presence of any License:: classifier rather than on the value it mapped to. The bare "License :: OSI Approved" umbrella still does not count, matching how the classifier tier already treats it. This costs nothing on the measured population: the corpus selects versions whose only license signal is the free-form field, so it contains no License:: classifiers at all, and the accept rate is unchanged at 48.35%. Also correct two overclaims the review flagged. The strict token rule stops an UNRECOGNIZED token from producing a license; it does not guarantee a recognized token was meant as an identifier, since SPDX publishes ~120 single-word ids including company names. Measured, that residual is 5 of 843,695 resolving versions. And the deprecated-id check is scoped to the free-form tier; License-Expression still passes through unvalidated. Rekey the corpus fixture on what RESOLVES rather than what is frequent. All 278 resolving strings are now covered, plus the 60 highest-volume rejections as anchors. Sampling by volume omitted the entire tail, which is exactly where the single-word risk lives.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Derives a license from the legacy free-form
Licensefield when neitherLicense-Expressionnor aLicense ::classifier says anything. That population previously derived toUnknown.This is PR 1 of the chain for rstudio/package-manager#19138. The producer call in
rstudio/pypi-manifestand the PPM adapter follow separately; nothing here is customer-visible until the corpus is re-encoded and republished.The design property that matters
One unrecognized token rejects the whole field.
The field resolves only when it is itself a published, non-deprecated SPDX identifier, or an SPDX expression whose every identifier is one. That single rule is what makes it impossible to invent a license here.
ParseSPDXExpressiondoes not validate, it strips operators and hands back whatever tokens remain, so3-Clause BSD Licensetokenizes to3-Clause/BSD/License. Accepting the recognizable-looking part of that would attributeBSD-3-Clauseto a package on the strength of a word.The rule also settles the ambiguous spellings without anyone adjudicating them:
GPLandGPLv3are not identifiers (the identifiers areGPL-3.0-only,GPL-3.0-or-later), so they are rejected rather than resolved to a guessed versionBSDnames a family with several incompatible membersApache 2.0andMIT License, are rejected for the same reason. Recognizing them is a separate decision requiring per-entry human review, and the strict gate defers that tier automaticallyMatching is case-insensitive, so
mitresolves toMIT. An exception is recognized but never returned, since the identifiers here are licenses:Apache-2.0 WITH LLVM-exceptionyieldsApache-2.0, whileMIT with restrictionsis rejected, because reporting plainMITfor it would state the opposite of what the package said.Because the tier runs last, it can only move a package out of
Unknown. It can never overrule a license the package declared through a structured field.Measured against real data, not invented strings
Scored against the live PyPI corpus captured 2026-08-18: 1,745,000 free-form-only (name, version) pairs over 17,846 distinct strings.
48.35% resolve (843,695 versions) from 278 distinct strings.
Largest accepted:
MIT600,772 ·Apache-2.0167,432 ·BSD-3-Clause18,546 ·mit6,922 ·GPL-3.0-or-later6,845.Largest still rejected, all correctly:
UNKNOWN69,824 ·Apache 2.063,176 ·Apache License 2.057,107 ·BSD44,585 ·MIT License40,663 ·GPL31,081 ·GPLv323,697.The 250 highest-volume strings are committed as
license/testdata/freeform_corpus.json, a regression fixture covering 87.9% of the corpus by volume. It is explicitly not a specification: every non-Unknownrow is a claim about a real package's license, so a row that moves needs a human to decide whether the new answer is right, not a re-record.SPDX identifiers now come from the published list
license/data/spdx_licenses.json, generated bylicense/data/gen_spdx_licenses.py, replaces an ad hoc set. Deprecated identifiers are excluded, soGPL-3.0(9,769 versions) andAGPL-3.0(8,223) do not resolve, being ambiguous between the-onlyand-or-laterforms that replaced them.The
Unknownsentinel is barred from the lookup table in two independent places, the generator andbuildSPDXIDDB, so a free-form field spellingUNKNOWNcan never be laundered into a type indistinguishable from a real derivation.Breaking
Typestakes a third argument, the raw free-formLicensefield. Pass""to keep the previous behaviour exactly. The only consumer is PPM.Verification
go test -race ./...green across every packagegolangci-lint run ./...0 issues,gofmtclean