Source-level LilyPond API and datasets for curated corpora; release 0.4.0

Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "lytk"
-version = "0.3.0"
+version = "0.4.0"
 edition = "2021"
 # Floor set by our oldest dependency (clap 4.6). Matters because the PyPI sdist
 # is compiled on the user's own toolchain.

README.md

@@ -28,8 +28,9 @@ from the command line.
   release to the next.
 - **Built for machine learning.** Encoders for note arrays, piano rolls and
   Performance-RNN event sequences, objective metrics from the muspy family,
-  folder datasets with deterministic splits and caching, and padded data
-  loaders for PyTorch and TensorFlow.
+  datasets over folders or JSON Lines records with deterministic (or the
+  records' own) splits and caching, and padded data loaders for PyTorch and
+  TensorFlow.
 - **Fast.** A Rust core with prebuilt wheels, and a batch converter that uses
   every core. It transposes LilyPond about 50× faster than python-ly.
 
@@ -108,6 +109,19 @@ Scores differ in length, so each batch is padded. The true lengths come back
 alongside it because `0` is a valid event, pitch and velocity, so padding alone
 can't tell you where a score ends. `to_tensorflow_dataloader` works the same way.
 
+A corpus kept as JSON Lines records (id, text, metadata) keeps its ids and its
+own splits:
+
+```python
+from lytk.datasets import RecordsDataset
+
+data = RecordsDataset.from_jsonl("scores.jsonl", split_field="split", on_error="skip")
+splits = data.split()                  # {"train": …, "valid": …, "test": …}, as recorded
+loader = splits["train"].to_pytorch_dataloader("note_array", batch_size=32, return_ids=True)
+for arrays, lengths, ids in loader:
+    ...
+```
+
 The package also installs a `lytk` command:
 
 ```bash
@@ -154,6 +168,7 @@ lyrics); slurs and ornaments are not performed.
 
 ## Documentation
 
+- [Python API reference](docs/python-api.md)
 - [Command line](docs/cli.md)
 - [What each format reads and writes](docs/import-export.md)
 - [Design and internals](docs/design.md)

SECURITY.md

@@ -19,10 +19,13 @@ reachable from a malicious input file are in scope, in particular:
 
 - panics or unbounded memory/CPU use while parsing (a reachable
   denial-of-service for any service that converts user uploads)
-- path traversal or unintended file reads. The LilyPond reader does not
-  follow `\include` (it reports it); `lytk flatten` / `lytk.flatten` does, by
-  design, so callers that flatten untrusted `.ly` files should run it in a
-  sandbox and pass only trusted `-I` search paths
+- path traversal or unintended file reads. The LilyPond readers do not
+  follow `\include` (they report it) unless given `include_paths`;
+  `lytk flatten`, `lytk.flatten` and `lytk.flatten_string` do, by design. An
+  include may name any path, relative (`../../x`) or absolute, and the
+  file's text becomes part of the output, so callers that follow the
+  includes of untrusted `.ly` text should run in a sandbox and pass only
+  trusted search paths
 - decompression bombs in `.mxl` archives (these are read with a size cap)
 
 Release builds keep `overflow-checks` on, so an integer overflow panics rather

docs/changelog.md

@@ -7,6 +7,106 @@ engineering notes are in the [development log](devlog.md).
 
 ## [Unreleased]
 
+## [0.4.0] - 2026-09-28
+
+LilyPond source you can inspect and edit, and datasets for curated
+corpora. `\version` is read, compared and edited; LilyPond text is
+tokenized (embedded Scheme as Scheme) and counted; includes are found on
+the syntax tree, `flatten_string` flattens text, and every reader follows
+includes when given `include_paths=`, with diagnostics pointing into the
+caller's text. `lytk.info` gives a score's statistics. Datasets read JSON
+Lines records as well as folders, keep ids and a corpus's own splits, skip
+what they cannot read when asked, and cache by content. The Python API
+reference is generated from the stubs.
+
+### Added
+
+- LilyPond `\version`: `lytk.lilypond_version(text)` reads the version the
+  first `\version` statement states, from the parse tree (a commented-out
+  one does not count), as a `lytk.LilyPondVersion`, which compares
+  numerically (`2.24` equals `2.24.0`) and accepts what LilyPond 2.24
+  accepts. `lytk.set_lilypond_version(text, version)` and
+  `lytk.strip_lilypond_version(text)` edit the statements (a version given
+  as a string is written as given, once checked).
+  `Score.lilypond_version` and `MusicDocument.lilypond_version` give the
+  version a LilyPond source stated. `to_lilypond(…, version=)` and
+  `to_lilypond_music(…, version=)` write another `\version` than 2.24.0.
+  An invalid `\version` is an `invalid-version` error. Rust:
+  `ly_to_ir::{LilyPondVersion, lilypond_version, set_lilypond_version,
+  strip_lilypond_version}` and `ScoreMetadata::lilypond_version`.
+- `lytk.tokenize(text)`: the tokens of LilyPond text from the parse tree
+  (`lytk.Token`: kind, text, character span, line, column, and whether it
+  is Scheme). The kinds are comment, string, command, symbol, number,
+  fraction, punctuation, and error for text the grammar cannot tokenize, so
+  every character but whitespace is in exactly one token, broken input
+  included. Embedded Scheme is tokenized as Scheme: `scheme` is the `#` or
+  `$` that starts it, then its symbols, numbers, strings, comments,
+  brackets and quotes, with `boolean`, `character` and `keyword` for
+  Scheme's own; LilyPond inside it (`#{ … #}`) is tokenized as LilyPond.
+  `lytk.strip_comments(text)` removes LilyPond's and Scheme's comments.
+  Rust: `ly_to_ir::{tokenize, strip_comments, Token, TokenKind}`.
+- `lytk.info(score)`: a score's metadata and counts as a dict, the library
+  home of `lytk info --json`, which now calls it. Besides the parts and notes
+  it counts voices, bars, the length in quarter notes, lyric syllables, chord
+  symbols and grace notes, computed in Rust. `lytk.source_stats(text)`:
+  bytes, lines, tokens, comments, Scheme expressions and error tokens of
+  LilyPond text. Rust: `ir::stats::score_info`, `ly_to_ir::source_stats`.
+- `lytk.from_lilypond_music_movements(path)`: every movement of a LilyPond
+  file as a `MusicDocument`, as `from_lilypond_movements` gives them as
+  scores; `lytk.from_lilypond_movements_string(text)`, every movement of
+  LilyPond text.
+- `lytk.flatten_string(text, *, base_dir=None, include_paths=None,
+  add_markers=True)`: `flatten` for text. `lytk.from_abc_tunes_string(text)`:
+  every tune of ABC text.
+- `lytk.datasets.RecordsDataset`: a dataset over records holding music as
+  text, `from_jsonl(path, …)` or `from_records(iterable, …)`, with
+  `text_field`, `id_field`, `format` (LilyPond, MusicXML, ABC or `**kern`)
+  and `split_field`. Items keep their ids and records (`record(i)`); with
+  `split_field`, `split()` returns the records' own splits as
+  `{value: Subset}` instead of re-shuffling them.
+- Datasets: `on_error="raise" | "skip" | "warn"` (iteration leaves out an
+  item that cannot be read and records it in `dataset.errors`; indexing
+  still raises); `movements="all"` (each movement of a file or record is an
+  item); `language`, `include_paths`, `strict` and MIDI `quantize` passed to
+  the readers; `ids` on every dataset (a folder's are paths relative to it);
+  `return_ids=True` on the torch and TensorFlow datasets and data loaders;
+  `split(…, groups=…)` keeps the items of a group in one subset.
+- `docs/python-api.md`, the Python API reference, generated from the stubs
+  (`scripts/python_api.py`); a test fails when it is stale.
+- `include_paths=` on every LilyPond reader and on `check_lilypond`: given
+  (possibly empty), `\include`s are followed, relative to the file's
+  directory, then the paths. A diagnostic in an included file is reported at
+  its `\include`, naming the file; an include not found is an
+  `ignored-include` warning. Without it, includes are still not followed.
+  Rust: `LyToIrAdapter::with_include_paths`, `read_text` and `check_str`.
+
+### Changed
+
+- `flatten` finds includes on the parse tree: anywhere in a line, never in a
+  comment or a string. It used to miss an include not at the start of a
+  line, and after a one-line `%{ … %}` it took the rest of the file for a
+  comment and followed no include in it.
+- `flatten` keeps an include of LilyPond's own files (`english.ly`,
+  `gregorian.ly`, …) as it is when no such file is found; it was a "not
+  found" error. Rust: `flatten_str` takes `Option<&Path>` for its base
+  directory, and `FlattenOpts::keep_missing` keeps any missing include.
+- The dataset cache is keyed by the item's content, how it is read, lytk's
+  version and the converter's arguments with their defaults filled in, and
+  written atomically: an edited file is no longer served stale, and
+  `to_note_arrays()` and `iter_representation("note_array")` share entries.
+  Entries of earlier versions are not read again. A `Subset` uses its
+  parent's cache (it bypassed it).
+
+### Fixed
+
+- `flatten` kept only the last `\language` line, so music written before it
+  was read in the wrong language; every `\language` now stays.
+- `flatten` refused a file with more than one `\header` block, a top-level
+  one with a score's own included (`FlattenError::MultipleHeaders`, now
+  gone). LilyPond merges top-level headers, and so does lytk's reader.
+- The `ignored-include` warning for one of LilyPond's own files (other than a
+  language file) no longer advises flattening, which would not follow it.
+
 ## [0.3.0] - 2026-09-27
 
 LilyPond input you can trust. Readers raise `lytk.ParseError` instead of
@@ -477,7 +577,8 @@ First public release.
   newer.
 - MIT licence.
 
-[Unreleased]: https://github.com/CSCPadova/lytk/compare/v0.3.0...HEAD
+[Unreleased]: https://github.com/CSCPadova/lytk/compare/v0.4.0...HEAD
+[0.4.0]: https://github.com/CSCPadova/lytk/compare/v0.3.0...v0.4.0
 [0.3.0]: https://github.com/CSCPadova/lytk/compare/v0.2.0...v0.3.0
 [0.2.0]: https://github.com/CSCPadova/lytk/compare/v0.1.0...v0.2.0
 [0.1.0]: https://github.com/CSCPadova/lytk/releases/tag/v0.1.0

docs/cli.md

@@ -152,9 +152,12 @@ Parts:    1
   - MusicXML Part (28 measures)
 ```
 
-`--json` prints title, subtitle, composer, arranger, lyricist, language,
-`part_count`, `note_count`, and for each part its id, name, abbreviation,
-number of measures, staves, MIDI program and MIDI instrument.
+`--json` prints `lytk.info(score)`: title, subtitle, composer, arranger,
+lyricist, language, `lilypond_version`, `part_count`, `note_count`,
+`voice_count`, `bar_count`, `duration_quarters`, `lyric_count`,
+`chord_symbol_count`, `grace_note_count`, and for each part its id, name,
+abbreviation, number of measures, staves, MIDI program, MIDI instrument,
+voices and notes.
 
 ## positions
 
@@ -226,13 +229,17 @@ lytk flatten INPUT.ly [-o OUTPUT] [-I DIR ...] [--no-markers]
 ```
 
 Inlines every `\include`, recursively, into one self-contained file (stdout
-without `-o`). Includes are looked up next to the file that includes them, then
-with `.ly` and `.ily` appended, then in each `-I`/`--include-path` directory.
-A missing file and an include cycle are errors; the cycle is shown
-(`a.ly -> b.ly -> a.ly`).
-
-Repeated `\version` and `\language` lines are merged, keeping the last one and
-warning on stderr. More than one `\header` block is an error. Each inlined file
+without `-o`). Includes are found on the parse tree, anywhere in a line and
+never in a comment or string. They are looked up next to the file that
+includes them, then with `.ly` and `.ily` appended, then in each
+`-I`/`--include-path` directory. A missing file and an include cycle are
+errors; the cycle is shown (`a.ly -> b.ly -> a.ly`). An include of one of
+LilyPond's own files (`english.ly`, `gregorian.ly`) stays as it is when no
+such file is found: LilyPond finds it in its installation.
+
+Repeated `\version` lines are merged, keeping the last one and warning on
+stderr. `\language` lines and `\header` blocks all stay: each `\language`
+applies from where it stands, and LilyPond merges headers. Each inlined file
 is wrapped in `% === BEGIN INCLUDE: … ===` / `% === END INCLUDE: … ===`
 comments; `--no-markers` leaves them out.