Skip to content

Add 0.0.1 ProvenanceRecord spec (common, model-agnostic parts) - #5

Merged
jgoodyear merged 2 commits into
mainfrom
common-spec-to-main
Sep 12, 2026
Merged

jgoodyear merged 2 commits into
mainfrom
common-spec-to-main

Conversation

@ANierbeck

Copy link
Copy Markdown
Contributor

What this does

Fills in the 0.0.1 skeleton on main with the concrete ProvenanceRecord shape developed and stress-tested across four open-model corpora. Only the common, model-agnostic solution parts land here; the model-specific example records and stress-test notes stay on dedicated branches for dev-list discussion.

What lands (10 files)

  • schema/0.0.1/sourcelume.schema.json — ProvenanceRecord JSON Schema. Required: id, type, identifier, name, license, creator[], created, added, contentCreated, origin, custodyChain[]. Optional: version, licenseScope, licenseNote, licenseCategory, licenseHistory.
  • schema/0.0.1/sourcelume.shacl.ttl — matching SHACL shapes.
  • context/0.0.1/sourcelume.jsonld — JSON-LD context. Reuses dct:/schema: where equivalents exist; keeps the sl: namespace minimal. Uses main's sl: prefix and version-inclusive namespace IRI.
  • spec/0.0.1/index.md — normative prose (field table + License IRI conventions subsection).
  • tools/validate.py — Python validator (JSON Schema + SHACL). Exits 0 with "nothing to validate" when examples/ is empty.
  • mappings/ — draft crosswalks to Croissant, the SPDX AI Profile, and OTDI.
  • pyproject.toml, uv.lock — Python tooling.

What stays on branches (not in this PR)

  • All 30 examples/*.jsonld model-specific records + 3 examples/prototypes/
  • All notes/stress-test-*.md and notes/mimir-dynaword-sources.md

These remain on common-corpus-stress-test (and future dedicated branches) for dev-list discussion of the still-open gaps (B, C, D, E).

The four additive license fields

All optional, non-breaking; records that omit them validate.

Field Purpose
licenseScope Qualifies jurisdiction-scoped licenses (esp. public-domain claims whose PD status is not worldwide).
licenseNote Flags when a license IRI is materially misleading (e.g. reflects a processing artifact rather than intended selection).
licenseCategory Classifies the licensing model (open-source, open-data, public-domain, agreement-supplied, use-restricted, risk-based, other) so it is checkable at a glance rather than only by reading the IRI.
licenseHistory Records prior licenses when a dataset's own license changed over time; current license stays in license.

Conventions followed

  • Namespace: main's sl: prefix + version-inclusive IRI (https://sourcelume.apache.org/ns/0.0.1#), preserved.
  • Vocabulary reuse: dct:/schema: terms where equivalents exist; sl: kept minimal.
  • Generic descriptions: no model-specific names appear in the common files (descriptive text uses generic examples).
  • No examples/ or notes/ dependency: validator is tolerant of an empty examples/ dir; spec has no dangling links to example files.

Validation

  • tools/validate.py exits 0 on main + this PR (no examples present).
  • On the stress-test branch (with examples), 30/30 records PASS against this schema + SHACL.

Apache-ai: Yes
Generated-by: pi (anthropic/claude-opus-4-1)
Reviewed-by: anierbeck

…ator, crosswalks

Fills in the 0.0.1 skeleton on main with the concrete ProvenanceRecord shape
developed and stress-tested across four open-model corpora (DFM Mimir, Pleias
Common Corpus, BigScience BLOOM/ROOTS, AI2 OLMo/Dolma). The model-specific
example records and stress-test notes stay on dedicated branches for dev-list
discussion; only the common, model-agnostic solution parts land here.

What lands:
- schema/0.0.1/sourcelume.schema.json: the ProvenanceRecord JSON Schema. Required
  fields: id, type, identifier, name, license, creator[], created, added,
  contentCreated, origin, custodyChain[]. Optional: version, licenseScope,
  licenseNote, licenseCategory, licenseHistory.
- schema/0.0.1/sourcelume.shacl.ttl: matching SHACL shapes.
- context/0.0.1/sourcelume.jsonld: JSON-LD context. Reuses dct:/schema: terms
  where equivalents exist; keeps the sl: namespace minimal. Uses main's
  sl: prefix and version-inclusive namespace IRI (https://sourcelume.apache.org/ns/0.0.1#).
- spec/0.0.1/index.md: normative prose. Field table + License IRI conventions
  subsection (CC Public Domain Mark for copyright-expiration PD; sl:agreement-supplied
  for agreement-gated data). No stress-test or example references.
- tools/validate.py: Python validator (JSON Schema + SHACL). Exits 0 with
  "nothing to validate" when examples/ is empty, so it is useful on main
  before any example records land.
- mappings/: draft crosswalks to Croissant, the SPDX AI Profile, and OTDI.
- pyproject.toml, uv.lock: Python tooling for the validator.

The four additive license fields (licenseScope, licenseNote, licenseCategory,
licenseHistory) are optional and non-breaking; records that omit them validate.
licenseScope qualifies jurisdiction-scoped licenses (especially public-domain
claims). licenseNote flags when a license IRI is materially misleading (e.g.
reflects a processing artifact). licenseCategory classifies the licensing model
(open-source, open-data, public-domain, agreement-supplied, use-restricted,
risk-based, other) so it is checkable at a glance. licenseHistory records prior
licenses when a dataset's own license changed over time.

Descriptive text in schema and spec is generic; no model-specific names appear
in the common files.

Apache-ai: Yes
Generated-by: pi (anthropic/claude-opus-4-1)
Reviewed-by: anierbeck
ANierbeck added a commit that referenced this pull request Sep 12, 2026
The 0.0.1 spec prose rewrite dropped the ASF license header comment that
RAT (apache-rat-plugin) enforces on non-excluded files. spec/0.0.1/index.md
is not in pom.xml's RAT exclude list (unlike context/**/*.jsonld,
schema/**/*.{json,ttl}, mappings/**/*.md, uv.lock), so it must carry the
standard ASF header. CI failed on PR #5 with "Files with unapproved licenses:
spec/0.0.1/index.md".

No content change beyond re-prepending the header comment block.

Apache-ai: Yes
Generated-by: pi (anthropic/claude-opus-4-1)
Reviewed-by: anierbeck
The 0.0.1 spec prose rewrite dropped the ASF license header comment that
RAT (apache-rat-plugin) enforces on non-excluded files. spec/0.0.1/index.md
is not in pom.xml's RAT exclude list (unlike context/**/*.jsonld,
schema/**/*.{json,ttl}, mappings/**/*.md, uv.lock), so it must carry the
standard ASF header. CI failed on PR #5 with "Files with unapproved licenses:
spec/0.0.1/index.md".

No content change beyond re-prepending the header comment block.

Apache-ai: Yes
Generated-by: pi (anthropic/claude-opus-4-1)
Reviewed-by: anierbeck
@ANierbeck
ANierbeck requested a review from jgoodyear September 12, 2026 16:50
@jgoodyear
jgoodyear merged commit da90d59 into main Sep 12, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants