Repository navigation
Trustworthy LilyPond reading; release 0.3.0 - #11
Merged
Merged
Conversation
Three releases that make lytk a dependable reader of LilyPond, from lilycorpus's request list checked against 0.2.0: - J (0.3.0): no panic reaches Python (six known families, a firewall, LilyPond-shaped fuzz), an exception hierarchy, diagnostics with a strict mode and check_lilypond(), strings read whole (one fix for fifteen callers) with a header accessor, pitch-language files, release hygiene. - K (0.4.0): \version API, includes from strings, tokens, statistics. - L (0.5.0): robust FolderDataset, a records dataset, ids, API reference. The devlog records the grammar measurement behind J: 2 of 2,741 valid LilyPond files produce ERROR nodes, every structural error tried is caught, lexical and semantic ones are left to the walk. Co-Authored-By: Claude Opus 5.5 <[email protected]>
… reading J0 — tests/ly_corpus.rs: the grammar and the reader against LilyPond's own regression tests and snippets (2 of 2,626 valid files flagged, none refused; CI job lilypond-corpus on v2.26.0) and a structural-mutation board on the committed fixtures. J1 — no panic reaches Python: - every binding runs in guard(): a Rust panic becomes lytk.InternalError (base lytk.LytkError) and prints nothing; - the six known panic families are fixed at their source, the shared sinks are total, from_dict/from_json validate hand-supplied IR and the transforms bound their arguments; - the LilyPond reader has bounds (500,000 elements, 100,000 whole notes and bars, multipliers up to 2^24, nesting 2,000, octaves -128..127) and walks on a 64 MB thread, so the hunt's hangs, OOM aborts and stack overflows are refusals (ValueError) instead; - \repeat unfold inside \relative repeats the same pitches, as LilyPond does; - MusicXML divisions fall back to 10080 instead of overflowing or being truncated to u16. LilyPond-shaped fuzzing and the hunt's inputs are in tests/fuzz_inputs.rs (LYTK_FUZZ_CASES for more cases; 1,024 in CI). Co-Authored-By: Claude Opus 5.5 <[email protected]>
Plain \include inlines every time, so a file including the next one twice, level after level, doubled the output per level until memory ran out. Flattening now stops with FlattenError::TooLarge (a ValueError in Python) after 10,000 includes or 64 MiB of output. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…ct mode
J2: every reader raises lytk.ParseError (LytkError and ValueError) on input
it cannot read, and LilyPondSyntaxError (a ParseError) carries .diagnostics.
A non-UTF-8 .ly is a ParseError; I/O stays OSError.
J3: Diagnostic (src/diagnostics.rs). Syntax checks from the tree alone
(ly_to_ir/syntax.rs): ERROR and MISSING nodes, unclosed brackets, a >>
without <<, the leftovers of a Scheme expression missing its "(", a markup
\override without a pair. Semantic ones at the walk's drop sites: invalid
durations and ratios, plain text, unknown commands (LilyPond 2.26's
commands come from scripts/ly_builtins.py), includes, unknown languages,
dropped music. check_lilypond, strict=False on the LilyPond readers,
Score/MusicDocument.diagnostics, `lytk check`. Markup is read as text, not
music; a byte-order mark is whitespace anywhere.
Boards on LilyPond 2.26.0: 1 of 2,626 valid files with an error (a grammar
gap), 499 of 502 broken fixtures caught, check median 0.08 ms.
Also: a tremolo past :1024 no longer overflows the LilyPond writer (found
by the fuzzer).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
J4: ly_to_ir/text.rs decodes strings as LilyPond's lexer does (\n, \t, \\, \", \'; other backslashes kept) in every context; a string used to end at its first escape. Header values given as \markup become their plain text and #"…" their string; each \score's \header stays with its movement and the top-level one fills every movement. Score.header and MusicDocument.header give every field; lytk.header_fields(text) locates each one (character span, score index) from the tree alone. The writers write every header field (sorted, quoted keys) and the text at notes, which the Score path lost. The 389 snippets' texidoc and categories equal an independent decoder (snippet_headers). J5: \include "english.ly" and LilyPond's other language files set the pitch language (arabic.ly: Italian); makam, persian, bagpipe, hel-arabic and turkish-makam raise unknown-language. \language works inside \score and music, and an unknown name keeps the language in force. Co-Authored-By: Claude Opus 5.5 <[email protected]>
One version source: Cargo.toml (pyproject declares it dynamic; uv.lock records it so), exported as lytk.__version__ and printed by lytk --version. The release workflow fails when the tag is not the version; the Python CI job runs on 3.10–3.13. Stale docs corrected: the reader does not follow \include (SECURITY.md, the 0.1.0 changelog), the pre-0.2.0 MIDI limitations, design.md's test counts, development.md's Rust CLI, T11.2. The vendored grammar's upstream commit (b3b38a6) is recorded. Co-Authored-By: Claude Opus 5.5 <[email protected]>
The five areas the first hunt left (commands, values, modes, resources, other readers): 308 cases through every reader, writer, transform and representation, one process at a time under a memory limit. - \ottava past three octaves panicked (n * 8, abs); sizes are MusicXML's 8/15/22 everywhere now (the writers wrote \ottava #1 for a 15ma, or #8 for an 8va, and lost an 8vb's direction). - A **kern note with 256+ dots panicked. - A MusicXML note of 25 million whole notes made the ABC writer tie it over every bar line until an allocation aborted: the MusicXML, ABC and kern readers refuse music past 100,000 whole notes (as LilyPond and MIDI do), and the ABC writer, fallible now, refuses more than 100,000 bars. - Quadratic time: meter changes in Grid::build (5.8 s → 0.6 s), tuplet runs in the ABC writer, grace runs in the Music-path LilyPond writer, and diagnostics on one long line (columns are counted on along the line). repeated_constructs_scale_linearly pins them with a time ratio. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Each top-level music expression is a movement, in order with the \score
blocks, as LilyPond makes a score of each; music beside \score and a music
variable at the top level are read, no longer dropped. Notes outside braces
at the top level are a syntax error. \book and \bookpart are walked like
the top level.
Drum mode reads LilyPond's 128 drum names as General MIDI keys on a
percussion staff (table generated from drumpitch-init.ly). \fixed reads
absolute pitches from its reference. \chords, \figures and \lyrics no
longer read their contents as notes.
Context bodies go through walk_body, so \new Staff \new Voice { } and a
mode prefix stay in their context; part ids no longer skip P1 after a
variable definition.
Corpus: dropped-music 77 -> 0, unrecognized-token 437 -> 247; board (a)
unchanged.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
After writing the figures due at a beat, a grace note (which takes no time) asked for the figures of an empty range, and BTreeMap::range panicked. Found by CI's fuzz job on \figuremode inside music, which J7 now reads as figures; \new FiguredBass hit it before. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Epic J (J0–J7, trustworthy LilyPond reading): the changelog's [Unreleased] becomes [0.3.0] - 2026-09-27; roadmap and development log updated. The version is 0.3.0 in Cargo.toml since J6. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
LilyPond input you can trust (Epic J, J0–J7). Released as 0.3.0. See
docs/changelog.mdfor the list of changes anddocs/devlog.mdfor the notes.What changes
lytk.ParseError(aValueError); a Rust panic anywhere islytk.InternalError, with nothing printed. The reader is bounded (500,000 notes, 100,000 whole notes or bars, 2,000 nesting levels), andflattenstops a diamond of includes.lytk.check_lilypond(text, semantic=False)andlytk check FILE…reportDiagnostics: syntax errors from the tree (0.08 ms median per file), and withsemantic=Truewhat the walk rejects or does not read;strict=Trueon every LilyPond reader raisesLilyPondSyntaxError;Score.diagnostics/MusicDocument.diagnostics.\markupand#"…"header values are text, headers are scoped per movement, and there areScore.headerandlytk.header_fields(with spans).\include "english.ly"and the other language files set the language;\languageworks inside\scoreand music.\book/\bookpart;\fixed;\chords/\figures/\lyrics;lytk.__version__, version fromCargo.tomlonly, Python CI on 3.10–3.13, and the release workflow refuses a tag that is not the version.Checks
LilyPond corpusCI job on LilyPond 2.26's 2,626 regression files and snippets:🤖 Generated with Claude Code