[ANierbeck]

Draft, for discussion on dev@. Not meant to be merged before the list has agreed to the scope.

What

A ProvenanceRecord describes a dataset. This adds a second record type, TrainingProvenanceRecord, for the model side: this model was trained on these datasets. It is asserted by whoever produced the model, so dataset records stay untouched and never point back at models.

{
  "@context": "https://sourcelume.apache.org/context/0.0.1/sourcelume.jsonld",
  "id": "https://sourcelume.apache.org/records/dfm-mimir-v1/training",
  "type": "TrainingProvenanceRecord",
  "identifier": "https://huggingface.co/danish-foundation-models/DFM-Mimir",
  "name": "DFM Mímir v1",
  "creator": [{ "type": "schema:Organization", "name": "Danish Foundation Models", "role": "originator" }],
  "created": "2026-10-13T00:00:00Z",
  "disclosure": "partial",
  "trainingData": [
    {
      "dataset": "https://sourcelume.apache.org/records/dfm-mimir-v1/laerebogen",
      "phase": "pretraining",
      "tokens": 8320000000,
      "share": 0.1181
    }
  ]
}

• The model is described with the terms a dataset record already uses (identifier, name, version, creator).
• trainingData[].dataset is an IRI, normally the id of a ProvenanceRecord. phase, tokens and share are optional.
• disclosure is complete, partial or undisclosed. trainingData may only be left out for undisclosed.
• Whether a referenced record exists, and in which state, is looked up when the model record is read. It is not stored in either record, and there is no approved flag.

Why a separate record rather than a field on the dataset record: the claim is the model producer's, not the dataset producer's, and a back-reference would let third parties write into someone else's record.

Example

examples/dfm-mimir-v1-training.jsonld refers to 12 DFM Mímir v1 dataset records (first commit, taken over from the unmerged stress-tests branch with the canonical @context). It is partial: the technical report lists 161 datasets and only these 12 have a record, about 43 percent of the sampled tokens. It is illustrative, authored from the public report, not attested by the producer.

Commits

1. The 12 Mímir dataset records as examples.
2. The record type: context, schema, shapes, validator, tests, spec text, changelog, SPDX crosswalk.
3. The two training record examples.

Each commit passes tools/validate.py, the unit tests and mvn verify.

Checks

• tools/validate.py: 14 examples pass; it now picks the schema by type.
• 14 new unit tests that check the JSON Schema and the SHACL shapes against each other.
• The shapes were also run with Jena SHACL 6.2.0, which the registry uses: 16 valid documents conform and 9 invalid variants are rejected, the same result as pyshacl. (Scratch runner outside the repo: a Graph parsed with RDFParser + ShaclValidator.get().validate(shapes, graph); easy to re-run against schema/0.0.1/sourcelume.shacl.ttl with the context inlined.)
• share is xsd:double: rdflib creates an xsd:decimal literal with a float value for a JSON number, which pyshacl then rejects.

Not in this PR

• ProvenanceRecord and its schema are unchanged.
• Registry: ShaclRecordValidator only accepts graphs with a sl:ProvenanceRecord node and the JSON Schema plugin loads one schema, so ingesting the new type needs a follow-up there.
• Audit results (e.g. memorisation audits) and what happens to a model's weights after training (merging, editing) stay out of scope. The non-goals say so.
• Signing, as before.

Open questions

1. Should this be in the spec, and before a first release?
2. Name: TrainingProvenanceRecord.
3. Is disclosure worth having? It lets a producer state that the data is not disclosed.
4. Phases: pretraining, fine-tuning, alignment, evaluation. Closed list or open?
5. One record per model version, or per training run?

Apache-ai: Yes
Generated-by: pi (claude-sonnet-5-5)
Reviewed-by: anierbeck

[ANierbeck]

No further discussion on this, therefore ready for review