diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 23f0aab5..10e991bf 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -926,7 +926,7 @@ jobs: set -euo pipefail sudo apt-get update sudo apt-get install -y --no-install-recommends \ - p7zip-full ffmpeg poppler-utils inkscape python3-pil + p7zip-full ffmpeg poppler-utils inkscape python3-pil python3-yaml # The distribution's Pillow cannot open an AVIF - measured on this # runner, PIL.UnidentifiedImageError on a file the structural checker diff --git a/CHANGELOG.md b/CHANGELOG.md index dea16f1a..f0470ae3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,7 +21,7 @@ because it turns other people's test suites red. used to start on whichever preset came first alphabetically - so what you saw when you opened the tab changed whenever a preset was added. A preset says now whether it is the one to start on, and `empty-and-minimal` is it: - pressing Generate without touching anything writes 32 214 B rather than the + pressing Generate without touching anything writes 32 667 B rather than the 73 MB the size-boundaries defaults come to, and its set means something without a number from you first. - **A recipe this program writes for you reads like one written by hand.** @@ -195,6 +195,33 @@ because it turns other people's test suites red. ### Added +- **Two more formats, both for configuration: `yaml` and `toml`.** A config + file at an exact size is what a size-limited upload is for a picture, and + neither format had a way to ask for one. Run `tfg formats` for all twenty + six, or `tfg formats yaml` for what one takes. + + Both write records: a `records` block of entries carrying an id, a name, an + email, an amount, a flag, a list of tags, a nested address and a note. YAML + comes out in block style rather than flow style, so it reads like a YAML + file rather than like JSON with a different extension. TOML comes out as an + array of `[[records]]` tables. + + **The smallest yaml is 241 B and the smallest toml is 212 B**, each one + whole record. An empty file is legal in both - and it stays something you + ask for by shape rather than by byte count, the same answer `json` gives + about an empty array. + + **The label rides in a comment**, which is new: these are the first record + formats whose label sits inside the file without touching the data being + tested. `csv` and `json` label from the outside because an extra field + changes the very structure under test, and a comment is not data. Ask for a + file at exactly the minimum and there is no room for it beside a whole + record, so it is left out and the run says so. + + **TOML takes no `encoding` or `bom` setting and says why rather than + calling it an unknown option.** TOML is UTF-8 by its own specification, and + both readers tested refuse a TOML file that opens with a byte order mark. + - **Three more presets: `upload-validation`, `text-encoding` and `tabular-import`.** Run `tfg preset list` for all five, or `tfg preset show ` for what one takes and what it would produce before @@ -246,8 +273,8 @@ because it turns other people's test suites red. - **A second preset: `empty-and-minimal`.** It answers "does a file that is valid and as small as the format allows get through?" and builds the smallest legal file of every format this build has, plus a file of nought - bytes for every format that has a legal empty form. The whole set is 26 - files and 32 214 B, so it checks twenty-four paths through your reader for + bytes for every format that has a legal empty form. The whole set is 28 + files and 32 667 B, so it checks twenty-six paths through your reader for the price of thirty-two kilobytes. Run it with `tfg generate --preset empty-and-minimal`, or pick it on the Presets screen. diff --git a/README.md b/README.md index d574c255..7a9dca3a 100644 --- a/README.md +++ b/README.md @@ -76,7 +76,7 @@ reference is below it. ## 📁 Formats it generates -Twenty four, and every one is a **real file of that format** - it opens in the +Twenty six, and every one is a **real file of that format** - it opens in the software that owns it, at the exact size you asked for: | group | formats | @@ -84,6 +84,7 @@ software that owns it, at the exact size you asked for: | 📄 **Documents** | `pdf`, `docx` (Word), `xlsx` (Excel), `pptx` (PowerPoint) | | 🖼️ **Images** | `png`, `jpg`, `bmp`, `gif`, `ico`, `svg`, `tiff`, `webp`, `avif`, `jxl` | | 📝 **Text and markup** | `txt`, `md`, `csv`, `json`, `xml`, `html`, `log` | +| ⚙️ **Configuration** | `yaml`, `toml` | | 🗜️ **Archives** | `zip`, `targz` (`.tar.gz`) | | 🔊 **Audio** | `wav` | @@ -555,6 +556,7 @@ recipe. `tfg formats ` prints the allowed range or list for each: | `json` | `formatting` | | `svg` | `width`, `height` | | `html` | `structure` | +| `yaml`, `toml` | none in this build - the document is a fixed shape, and a size is the only thing to ask for | ``` tfg generate --format jpg --size 500kb --set width=1920 --set height=1080 --set quality=85 diff --git a/internal/format/all/all.go b/internal/format/all/all.go index 5bb02e34..67ab8fcf 100644 --- a/internal/format/all/all.go +++ b/internal/format/all/all.go @@ -26,10 +26,12 @@ import ( _ "github.com/donislawdev/TestingFilesGenerator/internal/format/svgfile" _ "github.com/donislawdev/TestingFilesGenerator/internal/format/targz" _ "github.com/donislawdev/TestingFilesGenerator/internal/format/tiff" + _ "github.com/donislawdev/TestingFilesGenerator/internal/format/tomlfile" _ "github.com/donislawdev/TestingFilesGenerator/internal/format/txt" _ "github.com/donislawdev/TestingFilesGenerator/internal/format/wav" _ "github.com/donislawdev/TestingFilesGenerator/internal/format/webp" _ "github.com/donislawdev/TestingFilesGenerator/internal/format/xlsx" _ "github.com/donislawdev/TestingFilesGenerator/internal/format/xmlfile" + _ "github.com/donislawdev/TestingFilesGenerator/internal/format/yamlfile" _ "github.com/donislawdev/TestingFilesGenerator/internal/format/zip" ) diff --git a/internal/format/tomlfile/toml.go b/internal/format/tomlfile/toml.go new file mode 100644 index 00000000..25d070fa --- /dev/null +++ b/internal/format/tomlfile/toml.go @@ -0,0 +1,335 @@ +// Package tomlfile generates TOML documents. +// +// The package is not called "toml" so that it cannot be confused with the +// parser the toolkit already pulls into this module, the same reason jsonfile +// is not called json. The format id is "toml". +package tomlfile + +import ( + "context" + "fmt" + "io" + // D11 promises the same bytes from the same seed, so a deliberate, + // reproducible generator is the product rather than a weakness. Nothing + // here ever makes a secret. + // nosemgrep: go.lang.security.audit.crypto.math_random.math-random-used + "math/rand/v2" + "strconv" + + "github.com/donislawdev/TestingFilesGenerator/internal/core" + "github.com/donislawdev/TestingFilesGenerator/internal/format" + "github.com/donislawdev/TestingFilesGenerator/internal/format/textenc" +) + +// Measured on 2026-09-22 across five candidate channels and three readers - +// Python's tomllib, tomlkit 0.15.1 and BurntSushi/toml 1.6.0. Comments, basic +// strings and multi line strings all hold to 10 MB, and odd sizes are +// reachable. Written up in docs/YAML-TOML-2026-09-22.md. +// +// tomllib and tomli are the same code vendored twice, so they count as one +// reader rather than two. The second one is tomlkit. +// +// A comment wins that measurement and loses the design. A parser throws +// comments away, so a document padded with one has the right size and gives +// the system under test nothing to do. The filler goes into the note value of +// the last record, where the reader has to carry it, and the comment takes the +// label instead. +const ( + generatorVersion = "1" + + emailDomain = "@example.com" + + // An array of tables rather than one inline array, because that is the + // shape a person meets in a real TOML file. There is no root key: a TOML + // document is a table already. + openID = "[[records]]\nid = " + openName = "\nname = \"" + openMail = "\"\nemail = \"" + openAmt = "\"\namount = " + openAct = "\nactive = " + openTags = "\ntags = [\"" + tagSep = "\", \"" + openAddr = "\"]\naddress = { city = \"" + openZip = "\", zip = " + openNote = " }\nnote = \"" + tail = "\"\n" + + // maxIDDigits bounds the width of the record number, so the shortest whole + // record holds for every draw rather than for the lucky one. + // + // A constant rather than the width of the next id, for the three reasons + // written out beside the same name in internal/format/yamlfile: Shortest is + // a worst case bound on three axes, core.FillRecords reads it once so a + // growing width goes stale in the loop, and json and xml stand on the same + // constant with their minimums published. + maxIDDigits = 19 + + amountDigits = 9 + zipDigits = 5 + // "false" is the longer of the two, and the minimum has to hold for both. + longestBool = 5 +) + +func init() { + format.Register(format.Descriptor{ + ID: "toml", + Extension: ".toml", + Fidelity: format.FidelityFull, + Determinism: format.DeterminismByte, + + // An empty file is legal TOML - measured, it reads back as an empty + // table. It is still not what a byte count orders, the same answer + // JSON gives about an empty array. The minimum is one whole record. + MinBytes: minimumBytes(), + + Padding: format.PaddingChannel{ + Name: "the note value of the last record", + Where: format.PlacementEnd, + // No ceiling found to 10 MB, which is where the measurement + // stopped. + Capacity: 0, + }, + + Label: format.LabelInternal, + Oracle: "python-toml", + // Record counts and table layouts come later. Declaring only what is + // here makes a recipe asking for them fail loudly. + Properties: nil, + + // Encoding is not a gap here, it is the format. TOML 1.0 says a + // document is UTF-8, and measured on 2026-09-22 both readers refuse a + // byte order mark even in front of UTF-8 - so there is nothing to + // offer rather than something not built yet. Saying so with the reason + // beats the generic "no such property", which reads as a hole in this + // build. + Unsupported: []format.UnsupportedSetting{ + { + Name: textenc.Setting, + Why: "TOML is UTF-8 by its own specification, so there is no other " + + "encoding for a document to be in", + Instead: "Use yaml, xml, txt or md for a file in another encoding.", + }, + { + Name: textenc.SettingBOM, + Why: "both readers on this machine refuse a TOML file that opens with a " + + "byte order mark, even in UTF-8", + Instead: "Use txt or md for a file that opens with a byte order mark.", + }, + }, + GeneratorVersion: generatorVersion, + Generator: generator{}, + }) +} + +type generator struct{} + +type memo struct { + seed uint64 + comment string +} + +func (generator) Plan(r format.Request) (format.Plan, error) { + min := minimumBytes() + if r.Bytes < min { + return format.Plan{}, &format.BelowMinimumError{ + Format: "TOML", + Requested: r.Bytes, + Minimum: min, + Reason: "a document holds whole records, and one table of them with every value type needs that much", + Hint: fmt.Sprintf("Ask for %d B or more.", min), + } + } + + p := format.Plan{ + Bytes: r.Bytes, + Exact: true, + Determinism: format.DeterminismByte, + Properties: map[string]any{ + "encoding": "utf-8", + "line_ending": "lf", + "root": "records", + "style": "array-of-tables", + // A TOML document has no null. The record here carries one value + // of every type the format does have, and the absence is stated + // rather than left for somebody to notice it is shorter than the + // JSON one. + "null_supported": false, + }, + } + + m := memo{seed: r.Seed} + if r.Label { + line := "# " + core.Label("toml", r.Bytes, r.Seed) + "\n" + if int64(len(line))+min <= r.Bytes { + m.comment = line + } else { + p.Notes = append(p.Notes, format.Note{ + Code: "label_omitted", + Detail: fmt.Sprintf( + "The label comment needs %d B and this file has no room for it beside a whole record. Its name and the manifest still identify it.", + len(line)), + }) + } + } + + p.Properties[format.PropertyLabelEmbedded] = m.comment != "" + p.Memo = m + return p, nil +} + +func (generator) Write(ctx context.Context, w io.Writer, p format.Plan) error { + m, ok := p.Memo.(memo) + if !ok { + return fmt.Errorf("toml: the plan was not produced by this generator") + } + + // The comment is the whole of the prologue, so with no label there is + // nothing in front of the first table and the document opens with it. + if err := core.WriteAll(w, []byte(m.comment)); err != nil { + return err + } + + rng := core.NewRand(m.seed) + return core.FillRecords(ctx, w, rng, p.Bytes-int64(len(m.comment)), &records{}) +} + +// minimumBytes is one shortest whole record, with no label. +func minimumBytes() int64 { + r := &records{} + return r.Shortest() +} + +// records builds the tables. It carries the record number, so the id counts up +// the way a real export does. +type records struct { + next int64 +} + +// Shortest is the smallest record this builder can close a document with: the +// widest record number, the longest word in all five places a word appears, +// the longer of the two booleans, and an empty note. +func (r *records) Shortest() int64 { + return int64(maxIDDigits + 5*longestWord + fixed()) +} + +func (r *records) Append(dst []byte, rng *rand.Rand) []byte { + return r.append(dst, rng, -1) +} + +func (r *records) AppendExact(dst []byte, rng *rand.Rand, n int64) []byte { + return r.append(dst, rng, n) +} + +// Discard hands back the number the thrown away record took with it, so the +// ids read 1..N with nothing missing. +func (r *records) Discard() { r.next-- } + +// fixed is every byte of a record that does not depend on a draw, measured +// from the literals beside it rather than written out as a number. +func fixed() int { + return len(openID) + len(openName) + len(openMail) + len(emailDomain) + + len(openAmt) + amountDigits + len(openAct) + longestBool + + len(openTags) + len(tagSep) + len(openAddr) + len(openZip) + zipDigits + + len(openNote) + len(tail) +} + +// append writes one record. A want below zero means whatever length it comes +// out, any other value is the exact length the record must have. +// +// It appends rather than returning a new slice because a document of any size +// is millions of records, and one allocation per record is a multiple of the +// file in garbage. +func (r *records) append(dst []byte, rng *rand.Rand, want int64) []byte { + r.next++ + start := len(dst) + + name := words[rng.IntN(len(words))] + whole := 100000 + rng.IntN(899999) + cents := rng.IntN(100) + active := rng.IntN(2) == 0 + tagA := words[rng.IntN(len(words))] + tagB := words[rng.IntN(len(words))] + city := words[rng.IntN(len(words))] + zip := 10000 + rng.IntN(90000) + + dst = append(dst, openID...) + dst = strconv.AppendInt(dst, r.next, 10) + dst = append(dst, openName...) + dst = append(dst, name...) + dst = append(dst, openMail...) + dst = append(dst, name...) + dst = append(dst, emailDomain...) + dst = append(dst, openAmt...) + dst = strconv.AppendInt(dst, int64(whole), 10) + dst = append(dst, '.') + if cents < 10 { + dst = append(dst, '0') + } + dst = strconv.AppendInt(dst, int64(cents), 10) + dst = append(dst, openAct...) + if active { + dst = append(dst, "true"...) + } else { + dst = append(dst, "false"...) + } + dst = append(dst, openTags...) + dst = append(dst, tagA...) + dst = append(dst, tagSep...) + dst = append(dst, tagB...) + dst = append(dst, openAddr...) + dst = append(dst, city...) + dst = append(dst, openZip...) + dst = strconv.AppendInt(dst, int64(zip), 10) + dst = append(dst, openNote...) + + if want < 0 { + dst = appendPhrase(dst, rng, 3+rng.IntN(5)) + return append(dst, tail...) + } + + used := int64(len(dst)-start) + int64(len(tail)) + dst = core.AppendFiller(dst, words, want-used, nil) + return append(dst, tail...) +} + +// appendPhrase writes a readable note. Words and single spaces only - a basic +// string cannot take a quote or a backslash raw, and neither appears in the +// vocabulary. Measured 2026-09-22: every other byte from 20 to 7E goes in as +// itself, so an escape never costs a byte the arithmetic did not budget for. +func appendPhrase(dst []byte, rng *rand.Rand, n int) []byte { + for i := 0; i < n; i++ { + if i > 0 { + dst = append(dst, ' ') + } + dst = append(dst, words[rng.IntN(len(words))]...) + } + return dst +} + +// longestWord is the widest draw, because the minimum has to hold for every +// draw rather than for the lucky one. +var longestWord = func() int { + longest := 0 + for _, w := range words { + if len(w) > longest { + longest = len(w) + } + } + return longest +}() + +// words is the vocabulary for names, tags, cities and notes. Its own copy +// rather than a shared one, like every other format here: D11 freezes these +// bytes per format, so one shared list would move every file at once the day a +// word changed. +var words = []string{ + "account", "amount", "balance", "branch", "broker", "budget", "buyer", + "carrier", "charge", "client", "column", "contact", "contract", "credit", + "customer", "delivery", "deposit", "discount", "dispatch", "district", + "invoice", "ledger", "manager", "market", "member", "monthly", "order", + "partner", "payment", "pending", "product", "profile", "project", "quarter", + "receipt", "record", "refund", "region", "report", "reseller", "revenue", + "sample", "seller", "service", "shipment", "status", "storage", "summary", + "supplier", "support", "tariff", "ticket", "transfer", "vendor", "voucher", + "warehouse", "weekly", "wholesale", +} diff --git a/internal/format/yamlfile/yaml.go b/internal/format/yamlfile/yaml.go new file mode 100644 index 00000000..bd80edd2 --- /dev/null +++ b/internal/format/yamlfile/yaml.go @@ -0,0 +1,361 @@ +// Package yamlfile generates YAML documents. +// +// The package is not called "yaml" so that it cannot be confused with the +// parser this module already links, the same reason jsonfile is not called +// json and logfile is not called log. The format id is "yaml". +package yamlfile + +import ( + "context" + "fmt" + "io" + // D11 promises the same bytes from the same seed, so a deliberate, + // reproducible generator is the product rather than a weakness. Nothing + // here ever makes a secret. + // nosemgrep: go.lang.security.audit.crypto.math_random.math-random-used + "math/rand/v2" + "strconv" + + "github.com/donislawdev/TestingFilesGenerator/internal/core" + "github.com/donislawdev/TestingFilesGenerator/internal/format" +) + +// Measured on 2026-09-22 across twelve candidate channels and four readers - +// PyYAML 6.0.3, ruamel.yaml 0.19.1, goccy/go-yaml 1.19.2 and gopkg.in/yaml.v3 +// 3.0.1. Comments, quoted scalars, block scalars and blank lines all hold to +// 10 MB, and odd sizes are reachable everywhere. Written up in +// docs/YAML-TOML-2026-09-22.md. +// +// A comment wins that measurement and loses the design, which is the second +// half of the lesson rather than a footnote. A parser throws comments away, so +// a five megabyte document built as two kilobytes of records and five +// megabytes of comment is the right size and a useless fixture - the system +// under test gets two kilobytes of work. It is the same trap JSON turned down +// with whitespace. +// +// So the filler goes where the reader has to carry it: the note value of the +// last record. The comment carries the label instead, and that is worth +// something on its own - a comment is not data, so YAML is the first record +// format here whose label rides inside the file without changing the structure +// under test. CSV and JSON label from the outside for exactly that reason. +const ( + generatorVersion = "1" + + emailDomain = "@example.com" + + // The document is a block mapping holding a block sequence, rather than + // the flow style that would also be legal. YAML is a superset of JSON, so + // a flow document would BE a JSON document with a different extension - + // which is the one thing this format must not produce if it is to be worth + // having beside jsonfile. + rootOpen = "records:\n" + + openID = " - id: " + openName = "\n name: " + openMail = "\n email: " + openAmt = "\n amount: " + openAct = "\n active: " + openTags = "\n tags:\n - " + tagSep = "\n - " + openAddr = "\n address:\n city: " + openZip = "\n zip: " + // The note is quoted where every other scalar is plain, and the reason is + // the smallest file rather than taste. The filler in a closing record can + // come to nought bytes, and a plain scalar with nothing after the colon is + // null - so the type of this value would depend on the size asked for, + // which is a defect no size guard could see. Quoted, it is the empty + // string at every size. + openNote = "\n note: \"" + tail = "\"\n" + + // maxIDDigits bounds the width of the record number, so the shortest whole + // record holds for every draw rather than for the lucky one. A document + // cannot carry more records than this many digits allow. + // + // It is a constant rather than the width of the next id, and that is a + // decision rather than an oversight. Measured 2026-09-22: at the floor the + // closing record carries 27 B of slack, 18 of which is this reserve - so a + // yaml floor of 223 B would be reachable. Three things are wrong with + // taking it. + // + // Shortest is a worst case bound on three axes - the id width, the five + // word draws and the longer boolean - and dropping one of the three while + // keeping two is arbitrary. core.FillRecords reads it ONCE, so a width + // that changes as the ids grow goes stale inside the loop: a closing record + // with a wide id and five long words can then need more than the cached + // bound promised, and AppendExact is asked for a record shorter than the + // one it must write. That window is a handful of bytes wide and needs an + // unlucky draw beside it - 401 sizes were swept without hitting it, which + // says it is rare rather than that it is absent. And json and xml stand on + // this same constant with their minimums published, so moving it is a D11 + // breaking change to two released formats to save 18 B on this one. + maxIDDigits = 19 + + // The amount is always six digits, a point and two more, and the postcode + // is always five. Fixed widths, so the arithmetic does not have to ask. + amountDigits = 9 + zipDigits = 5 + // "false" is the longer of the two, and the minimum has to hold for both. + longestBool = 5 +) + +func init() { + format.Register(format.Descriptor{ + ID: "yaml", + Extension: ".yaml", + Fidelity: format.FidelityFull, + Determinism: format.DeterminismByte, + + // An empty file is legal YAML - measured, all four readers take it and + // return a null document. It is still not what a byte count orders: + // that is a shape request, and it arrives with a record count property + // this build does not have yet. The minimum here is the root and one + // whole record, the same answer JSON gives to the same question about + // an empty array. + MinBytes: minimumBytes(), + + Padding: format.PaddingChannel{ + Name: "the note value of the last record", + Where: format.PlacementEnd, + // No ceiling found to 10 MB, which is where the measurement + // stopped. Above that is not knowledge this project has. + Capacity: 0, + }, + + Label: format.LabelInternal, + Oracle: "python-yaml", + // Record counts, flow style, multiple documents and encodings other + // than UTF-8 come later. Declaring only what is here makes a recipe + // asking for them fail loudly rather than quietly producing something + // else. + // + // UTF-16 is a measured possibility rather than an oversight: with a + // byte order mark in front of it, all four readers took utf-16le and + // utf-16be, and refused both without one. That is a joint rule between + // two settings, so it arrives with them or not at all. + Properties: nil, + GeneratorVersion: generatorVersion, + Generator: generator{}, + }) +} + +type generator struct{} + +type memo struct { + seed uint64 + comment string +} + +func (generator) Plan(r format.Request) (format.Plan, error) { + min := minimumBytes() + if r.Bytes < min { + return format.Plan{}, &format.BelowMinimumError{ + Format: "YAML", + Requested: r.Bytes, + Minimum: min, + Reason: "a document holds a root key and whole records, and one of each needs that much", + Hint: fmt.Sprintf("Ask for %d B or more.", min), + } + } + + p := format.Plan{ + Bytes: r.Bytes, + Exact: true, + Determinism: format.DeterminismByte, + Properties: map[string]any{ + "encoding": "utf-8", + "line_ending": "lf", + "root": "records", + "style": "block", + "depth": 3, + }, + } + + m := memo{seed: r.Seed} + if r.Label { + // A comment carries the label without touching the content, and it + // sits in front of the document where a person opening the file reads + // it first. Nothing in the label needs escaping: it is plain ASCII and + // a comment runs to the end of the line. + line := "# " + core.Label("yaml", r.Bytes, r.Seed) + "\n" + // It has to leave room for a whole document beside it, or the file + // would be a comment and a root key with nothing under it. + if int64(len(line))+min <= r.Bytes { + m.comment = line + } else { + p.Notes = append(p.Notes, format.Note{ + Code: "label_omitted", + Detail: fmt.Sprintf( + "The label comment needs %d B and this file has no room for it beside a whole record. Its name and the manifest still identify it.", + len(line)), + }) + } + } + + p.Properties[format.PropertyLabelEmbedded] = m.comment != "" + p.Memo = m + return p, nil +} + +func (generator) Write(ctx context.Context, w io.Writer, p format.Plan) error { + m, ok := p.Memo.(memo) + if !ok { + return fmt.Errorf("yaml: the plan was not produced by this generator") + } + + head := m.comment + rootOpen + if err := core.WriteAll(w, []byte(head)); err != nil { + return err + } + + rng := core.NewRand(m.seed) + return core.FillRecords(ctx, w, rng, p.Bytes-int64(len(head)), &records{}) +} + +// minimumBytes is the root key and one shortest whole record, with no label. +func minimumBytes() int64 { + r := &records{} + return int64(len(rootOpen)) + r.Shortest() +} + +// records builds the sequence items under the root key. It carries the record +// number, so the id counts up the way a real export does. +type records struct { + next int64 +} + +// Shortest is the smallest record this builder can close a document with: the +// widest record number, the longest word in all five places a word appears, +// the longer of the two booleans, and an empty note. +func (r *records) Shortest() int64 { + return int64(maxIDDigits + 5*longestWord + fixed()) +} + +func (r *records) Append(dst []byte, rng *rand.Rand) []byte { + return r.append(dst, rng, -1) +} + +func (r *records) AppendExact(dst []byte, rng *rand.Rand, n int64) []byte { + return r.append(dst, rng, n) +} + +// Discard hands back the number the thrown away record took with it, so the +// closing record carries it instead and the ids read 1..N with nothing +// missing. The gap that shape leaves was a real defect in three record +// formats, found by looking at a file rather than by any guard. +func (r *records) Discard() { r.next-- } + +// fixed is every byte of a record that does not depend on a draw. +// +// Measured from the literals beside it rather than written out as a number. +// Arithmetic that has to agree with the bytes next to it is a defect waiting +// for the day somebody adds a field and updates one of the two. +func fixed() int { + return len(openID) + len(openName) + len(openMail) + len(emailDomain) + + len(openAmt) + amountDigits + len(openAct) + longestBool + + len(openTags) + len(tagSep) + len(openAddr) + len(openZip) + zipDigits + + len(openNote) + len(tail) +} + +// append writes one record. A want below zero means whatever length it comes +// out, any other value is the exact length the record must have. +// +// It appends rather than returning a new slice because a document of any size +// is millions of records, and one allocation per record is a multiple of the +// file in garbage. The resource guard measures that. +func (r *records) append(dst []byte, rng *rand.Rand, want int64) []byte { + r.next++ + start := len(dst) + + name := words[rng.IntN(len(words))] + whole := 100000 + rng.IntN(899999) + cents := rng.IntN(100) + active := rng.IntN(2) == 0 + tagA := words[rng.IntN(len(words))] + tagB := words[rng.IntN(len(words))] + city := words[rng.IntN(len(words))] + zip := 10000 + rng.IntN(90000) + + dst = append(dst, openID...) + dst = strconv.AppendInt(dst, r.next, 10) + dst = append(dst, openName...) + dst = append(dst, name...) + dst = append(dst, openMail...) + dst = append(dst, name...) + dst = append(dst, emailDomain...) + dst = append(dst, openAmt...) + dst = strconv.AppendInt(dst, int64(whole), 10) + dst = append(dst, '.') + if cents < 10 { + dst = append(dst, '0') + } + dst = strconv.AppendInt(dst, int64(cents), 10) + dst = append(dst, openAct...) + if active { + dst = append(dst, "true"...) + } else { + dst = append(dst, "false"...) + } + dst = append(dst, openTags...) + dst = append(dst, tagA...) + dst = append(dst, tagSep...) + dst = append(dst, tagB...) + dst = append(dst, openAddr...) + dst = append(dst, city...) + dst = append(dst, openZip...) + dst = strconv.AppendInt(dst, int64(zip), 10) + dst = append(dst, openNote...) + + if want < 0 { + dst = appendPhrase(dst, rng, 3+rng.IntN(5)) + return append(dst, tail...) + } + + // Everything written so far, plus the bytes that close the record. + used := int64(len(dst)-start) + int64(len(tail)) + dst = core.AppendFiller(dst, words, want-used, nil) + return append(dst, tail...) +} + +// appendPhrase writes a readable note. Words and single spaces only - the one +// thing a quoted scalar cannot take raw is a quote or a backslash, and neither +// appears in the vocabulary. Measured 2026-09-22: every other byte from 20 to +// 7E goes in as itself, so an escape never costs a byte the arithmetic did not +// budget for. +func appendPhrase(dst []byte, rng *rand.Rand, n int) []byte { + for i := 0; i < n; i++ { + if i > 0 { + dst = append(dst, ' ') + } + dst = append(dst, words[rng.IntN(len(words))]...) + } + return dst +} + +// longestWord is the widest draw, because the minimum has to hold for every +// draw rather than for the lucky one. +var longestWord = func() int { + longest := 0 + for _, w := range words { + if len(w) > longest { + longest = len(w) + } + } + return longest +}() + +// words is the vocabulary for names, tags, cities and notes. Its own copy +// rather than a shared one, like every other format here: D11 freezes these +// bytes per format, so one shared list would move twenty four files at once +// the day a word changed. +var words = []string{ + "account", "amount", "balance", "branch", "broker", "budget", "buyer", + "carrier", "charge", "client", "column", "contact", "contract", "credit", + "customer", "delivery", "deposit", "discount", "dispatch", "district", + "invoice", "ledger", "manager", "market", "member", "monthly", "order", + "partner", "payment", "pending", "product", "profile", "project", "quarter", + "receipt", "record", "refund", "region", "report", "reseller", "revenue", + "sample", "seller", "service", "shipment", "status", "storage", "summary", + "supplier", "support", "tariff", "ticket", "transfer", "vendor", "voucher", + "warehouse", "weekly", "wholesale", +} diff --git a/internal/guard/generatorbytes_test.go b/internal/guard/generatorbytes_test.go index a275d9f8..a35e0b2c 100644 --- a/internal/guard/generatorbytes_test.go +++ b/internal/guard/generatorbytes_test.go @@ -289,6 +289,16 @@ func goldenCases() map[string]engine.Target { // would pin the same file. "xml_8kib_no_label": {ID: "g", Format: "xml", Sizes: engine.Uniform(1, 8192), Label: false}, + // The two configuration formats. Both carry the label inside the file, + // in a comment, so like XML the switch moves their bytes and both + // positions are pinned. Unlike XML the comment sits in front of + // everything, so what the second case pins is the whole document + // shifting rather than a line coming out of the middle. + "yaml_8kib": {ID: "g", Format: "yaml", Sizes: engine.Uniform(1, 8192), Label: true}, + "yaml_8kib_no_label": {ID: "g", Format: "yaml", Sizes: engine.Uniform(1, 8192), Label: false}, + "toml_8kib": {ID: "g", Format: "toml", Sizes: engine.Uniform(1, 8192), Label: true}, + "toml_8kib_no_label": {ID: "g", Format: "toml", Sizes: engine.Uniform(1, 8192), Label: false}, + // The label is a byte affecting switch, not a cosmetic one, so it is // pinned in both positions. "txt_4kib_no_label": {ID: "g", Format: "txt", Sizes: engine.Uniform(1, 4096), Label: false}, diff --git a/internal/guard/layers_test.go b/internal/guard/layers_test.go index 33bee3ff..db6fa2dc 100644 --- a/internal/guard/layers_test.go +++ b/internal/guard/layers_test.go @@ -44,6 +44,8 @@ var layer = map[string]int{ "internal/format/logfile": 1, "internal/format/csvfile": 1, "internal/format/jsonfile": 1, + "internal/format/yamlfile": 1, + "internal/format/tomlfile": 1, "internal/format/xmlfile": 1, "internal/format/htmlfile": 1, "internal/format/svgfile": 1, @@ -109,6 +111,8 @@ var sameLayerAllowed = map[string][]string{ "internal/format/logfile", "internal/format/csvfile", "internal/format/jsonfile", + "internal/format/yamlfile", + "internal/format/tomlfile", "internal/format/xmlfile", "internal/format/htmlfile", "internal/format/svgfile", @@ -138,6 +142,8 @@ var sameLayerAllowed = map[string][]string{ "internal/format/logfile": {"internal/format"}, "internal/format/csvfile": {"internal/format"}, "internal/format/jsonfile": {"internal/format"}, + "internal/format/yamlfile": {"internal/format"}, + "internal/format/tomlfile": {"internal/format", "internal/format/textenc"}, "internal/format/xmlfile": {"internal/format", "internal/format/textenc"}, "internal/format/htmlfile": {"internal/format"}, "internal/format/svgfile": {"internal/format", "internal/format/imagedim"}, diff --git a/internal/guard/oracle_test.go b/internal/guard/oracle_test.go index 6301dfde..5d37a9ad 100644 --- a/internal/guard/oracle_test.go +++ b/internal/guard/oracle_test.go @@ -163,6 +163,10 @@ var structurallyChecked = map[string]bool{ // Since 2026-09-07, when a text file gained something to be checked // against: the encoding it declares. "txt": true, "md": true, + // Since 2026-09-22, and these two carry more here than most: the reader + // beside YAML accepts a duplicate key, and field order is something a + // parser throws away before anybody could ask about it. + "yaml": true, "toml": true, } func TestTheStructuralCheckerCoversEveryFormatItShould(t *testing.T) { diff --git a/internal/guard/parity_test.go b/internal/guard/parity_test.go index 9d7c79df..6471eafa 100644 --- a/internal/guard/parity_test.go +++ b/internal/guard/parity_test.go @@ -132,6 +132,8 @@ var reachableFromTheWindow = []string{ "format:txt", "format:wav", "format:xml", + "format:yaml", + "format:toml", "format:zip", // Drawn from the declaration and nothing else, which is what declaring diff --git a/internal/guard/sitecount_test.go b/internal/guard/sitecount_test.go new file mode 100644 index 00000000..4528261d --- /dev/null +++ b/internal/guard/sitecount_test.go @@ -0,0 +1,126 @@ +package guard + +import ( + "encoding/json" + "os" + "path/filepath" + "regexp" + "strings" + "testing" +) + +// The site's own words never count the formats themselves. +// +// internal/site/view.go runs every piece of language text through the facts, so +// a string may say {{ .Facts.FormatCount }} and mean it. That mechanism exists +// because of a measured defect: a page title read "PDF, DOCX, PNG, ZIP and 16 +// more formats" and the sixteen was typed. +// +// It happened again. On 2026-09-22, while two formats were being added, the +// English pages still described the tool as producing "real files of twenty +// formats" and the formats page still said "All twenty open in the software +// that owns them" - in the description, the Open Graph card and the schema +// metadata. The Polish schema said "w dwudziestu formatach". They had been +// wrong since the twenty first format arrived and nothing had noticed, because +// TestTheSiteSaysWhatTheToolSays compares the published pages against what the +// program renders NOW - and the program rendered the same stale sentence. +// +// So the mechanism was never the missing piece. What was missing is something +// that notices prose going around it, which is this. +func TestTheSiteNeverCountsTheFormatsInItsOwnWords(t *testing.T) { + const escape = "{{ .Facts.FormatCount }}" + + // Two rules, because the two ways this has gone wrong do not look alike. + // + // A digit has to stand next to the word, which is what keeps "tfg generate + // --format png --size 2mb" and "generating ten thousand files" out of it. + // That shape is the 2026 title, "16 more formats". + digit := regexp.MustCompile(`(?i)\b\d+([- ]\w+)? (more )?(formats?|format\w+)\b`) + + // A written out number does not have to stand next to it, because the + // sentence that went stale this time never says the word: "All twenty open + // in the software that owns them". So any of these words in a value that + // talks about formats at all is the error. + // + // The list is only numbers a format COUNT could plausibly be. "one" and + // "ten" are left out deliberately - measured on this content, they appear + // nine times in prose that means neither, and Polish "ten" is not a number + // at all. A guard nobody can leave green is one somebody turns off. + written := regexp.MustCompile(`(?i)\b(` + strings.Join([]string{ + "sixteen", "seventeen", "eighteen", "nineteen", + "twenty", "thirty", "forty", "fifty", "sixty", + "szesnastu", "dwadzieścia", "dwudziestu", + "trzydzieści", "trzydziestu", "czterdziestu", + }, "|") + `)\b`) + mentionsFormats := regexp.MustCompile(`(?i)\bformat`) + + dir := filepath.Join(repoRoot(t), "web", "content") + entries, err := os.ReadDir(dir) + if err != nil { + t.Fatalf("reading the site content: %v", err) + } + if len(entries) == 0 { + t.Fatal("the site has no language directories - this guard would pass against anything") + } + + checked := 0 + for _, e := range entries { + if !e.IsDir() { + continue + } + path := filepath.Join(dir, e.Name(), "site.json") + body, err := os.ReadFile(path) + if err != nil { + t.Errorf("reading %s: %v", path, err) + continue + } + var tree any + if err := json.Unmarshal(body, &tree); err != nil { + t.Errorf("%s is not readable as JSON: %v", path, err) + continue + } + walkStrings(tree, func(where, value string) { + checked++ + if strings.Contains(value, escape) { + return + } + found := digit.FindString(value) + if found == "" && mentionsFormats.MatchString(value) { + found = written.FindString(value) + } + if found != "" { + t.Errorf("%s/%s says %q, and that number is typed rather than counted. "+ + "Write %s instead - the site expands it from the registry, so it cannot "+ + "go stale the way this one did twice", + e.Name(), where, found, escape) + } + }) + } + if checked == 0 { + t.Fatal("no string was read out of the site content - this guard would pass against anything") + } + t.Logf("%d site strings checked for a typed format count", checked) +} + +// walkStrings visits every string in a decoded JSON tree, with the path it sits +// at, so a failure names the key rather than only the file. +func walkStrings(node any, visit func(where, value string)) { + switch v := node.(type) { + case string: + visit("", v) + case []any: + for _, item := range v { + walkStrings(item, visit) + } + case map[string]any: + for key, item := range v { + walkStrings(item, func(where, value string) { + if where == "" { + visit(key, value) + return + } + visit(key+"."+where, value) + }) + } + } +} diff --git a/internal/guard/socialpicture_test.go b/internal/guard/socialpicture_test.go index 607f9e16..b048f059 100644 --- a/internal/guard/socialpicture_test.go +++ b/internal/guard/socialpicture_test.go @@ -95,10 +95,18 @@ func TestTheSocialPictureShowsTheCardAsItIsNow(t *testing.T) { "nothing else notices it is stale - the site guard copies it rather than "+ "rendering it. Measured once already: it sat three formats out of date for "+ "three days.\n"+ - "Take it again, then rewrite the site and the stamp:\n"+ + "The site is written TWICE and that is not a typo. The camera photographs "+ + "web/public/social.html as it is on disk, and the site copies the picture "+ + "back into web/public - so one write puts the new card where the camera can "+ + "see it, and the second carries the new picture into the site. Measured on "+ + "2026-09-22, when three lines in the other order left a picture of the old "+ + "card with a stamp of the new one, and this guard green over both.\n"+ + " TFG_WRITE_SITE=1 go test ./internal/guard/ -run TestTheSiteSaysWhatTheToolSays\n"+ " python tools/probes/social-shot.py web/public web/assets/social-preview.png\n"+ " TFG_WRITE_SITE=1 go test ./internal/guard/ -run TestTheSiteSaysWhatTheToolSays\n"+ - " TFG_WRITE_SOCIAL_STAMP=1 go test ./internal/guard/ -run TestTheSocialPicture", + " TFG_WRITE_SOCIAL_STAMP=1 go test ./internal/guard/ -run TestTheSocialPicture\n"+ + "Then LOOK at web/assets/social-preview.png. If git says it did not change, "+ + "the camera photographed the old card.", was, now) } } diff --git a/internal/guard/testdata/generator-golden.json b/internal/guard/testdata/generator-golden.json index 9a472b47..ef8ecfb8 100644 --- a/internal/guard/testdata/generator-golden.json +++ b/internal/guard/testdata/generator-golden.json @@ -301,6 +301,26 @@ "json_8kib_indented": { "bytes": 8192, "sha256": "9b00c22a11836d66cbf653275bb943b299d6864d391cff8ca13fa8ac7bdcc8c9" + }, + "yaml_8kib": { + "bytes": 8192, + "sha256": "e320add3591cb6708fc7d8adb786cb26de6237b2759eb6f4587dcfe99b54157f", + "measured_on": "2026-09-22" + }, + "yaml_8kib_no_label": { + "bytes": 8192, + "sha256": "83f0fbcd19fa41d5952226e919cc88f99e8cdd3b8729bf3074b745c0e6b38f05", + "measured_on": "2026-09-22" + }, + "toml_8kib": { + "bytes": 8192, + "sha256": "65c7f12bc32bf311dc89f48ca6f99ded025554cabc40d683f1caa4f4301a7a28", + "measured_on": "2026-09-22" + }, + "toml_8kib_no_label": { + "bytes": 8192, + "sha256": "eae81b32d70ff0380a09aa2d5cfbbbe85e8595c34d66187b9ca30cd7cc668540", + "measured_on": "2026-09-22" } }, "remeasured": [ diff --git a/internal/guard/testdata/screens/generate-menu-hovered.png b/internal/guard/testdata/screens/generate-menu-hovered.png index eca08b84..1b0eb6ca 100644 Binary files a/internal/guard/testdata/screens/generate-menu-hovered.png and b/internal/guard/testdata/screens/generate-menu-hovered.png differ diff --git a/internal/guard/testdata/screens/generate-menu-hovered.xml b/internal/guard/testdata/screens/generate-menu-hovered.xml index 84deb1f9..f6fb6f75 100644 --- a/internal/guard/testdata/screens/generate-menu-hovered.xml +++ b/internal/guard/testdata/screens/generate-menu-hovered.xml @@ -440,7 +440,7 @@ - + @@ -572,42 +572,42 @@ - txt + toml - - wav + + txt - - webp + + wav - - xlsx + + webp - - xml + + xlsx - - zip + + xml @@ -687,8 +687,8 @@ - - + + diff --git a/internal/guard/testdata/screens/generate-menu-keyed.png b/internal/guard/testdata/screens/generate-menu-keyed.png index ca291bab..483bae04 100644 Binary files a/internal/guard/testdata/screens/generate-menu-keyed.png and b/internal/guard/testdata/screens/generate-menu-keyed.png differ diff --git a/internal/guard/testdata/screens/generate-menu-keyed.xml b/internal/guard/testdata/screens/generate-menu-keyed.xml index bceb88f1..dfbeff4b 100644 --- a/internal/guard/testdata/screens/generate-menu-keyed.xml +++ b/internal/guard/testdata/screens/generate-menu-keyed.xml @@ -440,7 +440,7 @@ - + @@ -572,42 +572,42 @@ - txt + toml - - wav + + txt - - webp + + wav - - xlsx + + webp - - xml + + xlsx - - zip + + xml @@ -687,8 +687,8 @@ - - + + diff --git a/internal/guard/testdata/screens/generate-menu.png b/internal/guard/testdata/screens/generate-menu.png index 77ce6bf3..1106dffc 100644 Binary files a/internal/guard/testdata/screens/generate-menu.png and b/internal/guard/testdata/screens/generate-menu.png differ diff --git a/internal/guard/testdata/screens/generate-menu.xml b/internal/guard/testdata/screens/generate-menu.xml index 652e1978..e8a1fc9c 100644 --- a/internal/guard/testdata/screens/generate-menu.xml +++ b/internal/guard/testdata/screens/generate-menu.xml @@ -440,7 +440,7 @@ - + @@ -572,42 +572,42 @@ - txt + toml - - wav + + txt - - webp + + wav - - xlsx + + webp - - xml + + xlsx - - zip + + xml @@ -687,8 +687,8 @@ - - + + diff --git a/internal/guard/testdata/screens/preset-menu-setting.png b/internal/guard/testdata/screens/preset-menu-setting.png index 10775fbc..a0a091e5 100644 Binary files a/internal/guard/testdata/screens/preset-menu-setting.png and b/internal/guard/testdata/screens/preset-menu-setting.png differ diff --git a/internal/guard/testdata/screens/preset-menu-setting.xml b/internal/guard/testdata/screens/preset-menu-setting.xml index 584c4706..8e958756 100644 --- a/internal/guard/testdata/screens/preset-menu-setting.xml +++ b/internal/guard/testdata/screens/preset-menu-setting.xml @@ -403,7 +403,7 @@ - + @@ -535,21 +535,21 @@ - txt + toml - - wav + + txt - - webp + + wav @@ -620,8 +620,8 @@ - - + + diff --git a/internal/guard/testdata/screens/preset-menu.png b/internal/guard/testdata/screens/preset-menu.png index 862eaa7e..d7984cb7 100644 Binary files a/internal/guard/testdata/screens/preset-menu.png and b/internal/guard/testdata/screens/preset-menu.png differ diff --git a/internal/guard/testdata/screens/preset-menu.xml b/internal/guard/testdata/screens/preset-menu.xml index 47dcbd1c..2e7b8584 100644 --- a/internal/guard/testdata/screens/preset-menu.xml +++ b/internal/guard/testdata/screens/preset-menu.xml @@ -308,8 +308,8 @@ - 26 files · 31.5 KB (32 214 B) · avif, bmp, csv, docx, gif, html, ico, jpg, json, jxl, log, md, pdf, png, pptx, svg, targz, tiff, - txt, wav, webp, xlsx, xml, zip · will go to /tfg/out + 28 files · 31.9 KB (32 667 B) · avif, bmp, csv, docx, gif, html, ico, jpg, json, jxl, log, md, pdf, png, pptx, svg, targz, tiff, + toml, txt, wav, webp, xlsx, xml, yaml, zip · will go to /tfg/out diff --git a/internal/guard/testdata/screens/preset.png b/internal/guard/testdata/screens/preset.png index 2d237ea3..8eca313a 100644 Binary files a/internal/guard/testdata/screens/preset.png and b/internal/guard/testdata/screens/preset.png differ diff --git a/internal/guard/testdata/screens/preset.xml b/internal/guard/testdata/screens/preset.xml index 8b1b7a5a..61a61c54 100644 --- a/internal/guard/testdata/screens/preset.xml +++ b/internal/guard/testdata/screens/preset.xml @@ -308,8 +308,8 @@ - 26 files · 31.5 KB (32 214 B) · avif, bmp, csv, docx, gif, html, ico, jpg, json, jxl, log, md, pdf, png, pptx, svg, targz, tiff, - txt, wav, webp, xlsx, xml, zip · will go to /tfg/out + 28 files · 31.9 KB (32 667 B) · avif, bmp, csv, docx, gif, html, ico, jpg, json, jxl, log, md, pdf, png, pptx, svg, targz, tiff, + toml, txt, wav, webp, xlsx, xml, yaml, zip · will go to /tfg/out diff --git a/internal/guard/textformats_test.go b/internal/guard/textformats_test.go index e14152c0..c66262a8 100644 --- a/internal/guard/textformats_test.go +++ b/internal/guard/textformats_test.go @@ -444,7 +444,7 @@ func TestAnSVGDrawingCarriesRealShapes(t *testing.T) { // This is the shape docs/OBSERVATIONS.md now calls out: an audit of // completeness has to run FROM THE SOURCE towards the list. Walking the entries // already written down cannot, by construction, find what is missing from them. -var textFormats = []string{"txt", "md", "log", "csv", "json", "xml", "html", "svg"} +var textFormats = []string{"txt", "md", "log", "csv", "json", "xml", "html", "svg", "yaml", "toml"} var binaryFormats = []string{"avif", "bmp", "docx", "gif", "ico", "jpg", "jxl", "pdf", "png", "pptx", "targz", "tiff", "wav", "webp", "xlsx", "zip"} @@ -683,6 +683,8 @@ func TestRecordNumbersRunFromOneWithoutAGap(t *testing.T) { {"csv", regexp.MustCompile(`(?m)^(\d+),`)}, {"json", regexp.MustCompile(`\{"id":(\d+),`)}, {"xml", regexp.MustCompile(`= len(lines): + fail(f"record {count} ends while it is still opening") + m = re.fullmatch(pattern, lines[i]) + if not m: + fail(f"line {i + 1} is {lines[i]!r} and a record opens with {pattern!r}") + if m.groups() and int(m.group(1)) != count: + fail(f"record {count} carries the number {m.group(1)} - the numbering skips") + i += 1 + for name in RECORD_FIELDS[1:]: + for pattern in field_line(name): + if i >= len(lines): + fail(f"record {count} ends before its {name} field") + if not re.fullmatch(pattern, lines[i]): + fail(f"record {count}: line {i + 1} is {lines[i]!r} and the {name} " + f"field should read {pattern!r}") + i += 1 + if count < 1: + fail("the document carries no records") + return count + + +def check_yaml(data): + """Block YAML, hand read rather than parsed. + + Written to the specification instead of calling a parser, and here that is + worth more than usual: measured on 2026-09-22, PyYAML - the reader beside + this one - ACCEPTS a duplicate key. So does no other implementation tried, + and this layer is what stands in for them. + + The two things it looks at that no parser would report are indentation by + tabs, which YAML forbids outright, and the field ORDER, which a parser + throws away before anybody could ask. + """ + try: + text = data.decode("utf-8") + except UnicodeDecodeError as exc: + fail(f"the document is not valid UTF-8: {exc}") + if "\t" in text: + fail("the document indents with a tab, which YAML does not allow") + if not text.endswith("\n"): + fail("the document does not end with a newline") + if "\r" in text: + fail("the document carries a carriage return, and it is written with LF endings") + + lines = text.split("\n")[:-1] + at = 0 + if lines and lines[at].startswith("# "): + if not lines[at].startswith("# tfg - yaml - "): + fail(f"the opening comment is {lines[at]!r} and it should carry the label") + at += 1 + if at >= len(lines) or lines[at] != "records:": + fail("the document does not open with the records key") + at += 1 + + def field(name): + return { + "name": (rf" name: {WORD}",), + "email": (rf" email: {WORD}@example\.com",), + "amount": (rf" amount: {AMOUNT}",), + "active": (rf" active: (?:{BOOL})",), + "tags": (r" tags:", rf" - {WORD}", rf" - {WORD}"), + "address": (r" address:", rf" city: {WORD}", r" zip: [0-9]{5}"), + "note": (r' note: "[^"\\]*"',), + }[name] + + count = scan_records(lines, at, (r" - id: ([0-9]+)",), field) + ok(f"{count} records, fields in order, no tabs, ids without a gap") + + +def check_toml(data): + """An array of tables, hand read rather than parsed. + + The byte order mark is the check worth naming. TOML says a document is + UTF-8 and measured on 2026-09-22 both readers refuse one that opens with a + mark - so a mark here is a file no TOML reader would take, and it is + exactly the kind of thing a size guard and a hash call correct. + """ + if data.startswith(b"\xef\xbb\xbf"): + fail("the document opens with a byte order mark, which no TOML reader accepts") + try: + text = data.decode("utf-8") + except UnicodeDecodeError as exc: + fail(f"the document is not valid UTF-8: {exc}") + if not text.endswith("\n"): + fail("the document does not end with a newline") + if "\r" in text: + fail("the document carries a carriage return, and it is written with LF endings") + + lines = text.split("\n")[:-1] + at = 0 + if lines and lines[at].startswith("# "): + if not lines[at].startswith("# tfg - toml - "): + fail(f"the opening comment is {lines[at]!r} and it should carry the label") + at += 1 + + def field(name): + return { + "name": (rf'name = "{WORD}"',), + "email": (rf'email = "{WORD}@example\.com"',), + "amount": (rf"amount = {AMOUNT}",), + "active": (rf"active = (?:{BOOL})",), + "tags": (rf'tags = \["{WORD}", "{WORD}"\]',), + "address": (rf'address = {{ city = "{WORD}", zip = [0-9]{{5}} }}',), + "note": (r'note = "[^"\\]*"',), + }[name] + + # The table header sits on its own line, so a record opens with two lines + # here where YAML opens with one. + count = scan_records(lines, at, (r"\[\[records\]\]", r"id = ([0-9]+)"), field) + ok(f"{count} records, fields in order, no byte order mark") + + CHECKS = {"png": check_png, "wav": check_wav, "pdf": check_pdf, "zip": check_zip, "log": check_log, "csv": check_csv, "json": check_json, "xml": check_xml, "svg": check_svg, "html": check_html, "targz": check_targz, "bmp": check_bmp, "gif": check_gif, "ico": check_ico, "jpg": check_jpg, "tiff": check_tiff, "webp": check_webp, "avif": check_avif, "jxl": check_jxl, "docx": check_docx, "xlsx": check_xlsx, "pptx": check_pptx, - "txt": check_txt, "md": check_md} + "txt": check_txt, "md": check_md, + "yaml": check_yaml, "toml": check_toml} # Checks that take the shape of the file as well as its bytes. Everything else # is handed the bytes alone, so adding a setting to one check cannot change how diff --git a/web/assets/social-preview.png b/web/assets/social-preview.png index 3f99eba7..edc1df7f 100644 Binary files a/web/assets/social-preview.png and b/web/assets/social-preview.png differ diff --git a/web/content/en/site.json b/web/content/en/site.json index af68da73..313856e3 100644 --- a/web/content/en/site.json +++ b/web/content/en/site.json @@ -15,7 +15,7 @@ "slug": "formats", "nav": "Formats", "title": "{{ .Facts.FormatCount }} Supported File Formats - PDF, DOCX, PNG, ZIP and More", - "description": "Every file format this generator produces, the smallest file each one can be, and the settings each one accepts. All twenty open in the software that owns them." + "description": "Every file format this generator produces, the smallest file each one can be, and the settings each one accepts. All {{ .Facts.FormatCount }} open in the software that owns them." }, { "key": "docs", @@ -51,7 +51,7 @@ "navLabel": "Main", "breadcrumbHome": "Home", "imageAlt": "Testing Files Generator - real test files at any exact size, with a manifest saying how your system should react to each one", - "schemaDescription": "A free and open source generator of test files for QA. It produces real files of twenty formats at any exact size and writes a manifest saying how the system under test should react to each one.", + "schemaDescription": "A free and open source generator of test files for QA. It produces real files of {{ .Facts.FormatCount }} formats at any exact size and writes a manifest saying how the system under test should react to each one.", "ctaDownload": "Download", "ctaSource": "View the source", "ctaNote": "Free and open source, GPL-3.0. Nothing to sign up for. The Windows and macOS downloads are signed and start without a warning.", diff --git a/web/content/pl/site.json b/web/content/pl/site.json index bbd50cd4..2b22a755 100644 --- a/web/content/pl/site.json +++ b/web/content/pl/site.json @@ -51,7 +51,7 @@ "navLabel": "Główna", "breadcrumbHome": "Start", "imageAlt": "Testing Files Generator - prawdziwe pliki testowe o dokładnym rozmiarze, z manifestem mówiącym, jak system ma na nie zareagować", - "schemaDescription": "Darmowy generator plików testowych dla QA o otwartym kodzie. Tworzy prawdziwe pliki w dwudziestu formatach o dokładnie zadanym rozmiarze i zapisuje manifest mówiący, jak testowany system ma na każdy z nich zareagować.", + "schemaDescription": "Darmowy generator plików testowych dla QA o otwartym kodzie. Tworzy prawdziwe pliki w {{ .Facts.FormatCount }} formatach o dokładnie zadanym rozmiarze i zapisuje manifest mówiący, jak testowany system ma na każdy z nich zareagować.", "ctaDownload": "Pobierz", "ctaSource": "Zobacz kod źródłowy", "ctaNote": "Darmowe i otwarte, GPL-3.0. Bez zakładania konta. Pliki dla Windows i macOS są podpisane i uruchamiają się bez ostrzeżenia.", diff --git a/web/public/assets/social-preview.png b/web/public/assets/social-preview.png index 3f99eba7..edc1df7f 100644 Binary files a/web/public/assets/social-preview.png and b/web/public/assets/social-preview.png differ diff --git a/web/public/faq/index.html b/web/public/faq/index.html index d79afecb..28efe592 100644 --- a/web/public/faq/index.html +++ b/web/public/faq/index.html @@ -134,7 +134,7 @@

Frequently asked questions

Which formats are coming next?
-

7z, mp3 and mp4. 24 formats work end to end today.

+

7z, mp3 and mp4. 26 formats work end to end today.

@@ -205,7 +205,7 @@

Frequently asked questions

{ "@type": "Question", "name": "Which formats are coming next?", - "acceptedAnswer": { "@type": "Answer", "text": "7z, mp3 and mp4. 24 formats work end to end today." } + "acceptedAnswer": { "@type": "Answer", "text": "7z, mp3 and mp4. 26 formats work end to end today." } }, { "@type": "Question", diff --git a/web/public/formats/index.html b/web/public/formats/index.html index 7104dcd6..0b876abf 100644 --- a/web/public/formats/index.html +++ b/web/public/formats/index.html @@ -3,8 +3,8 @@ -24 Supported File Formats - PDF, DOCX, PNG, ZIP and More - +26 Supported File Formats - PDF, DOCX, PNG, ZIP and More + @@ -12,8 +12,8 @@ - - + + @@ -21,8 +21,8 @@ - - + + @@ -74,7 +74,7 @@
-

24 file formats, every one generated at an exact size

+

26 file formats, every one generated at an exact size

Each of these is a real file of that format. It opens in the software that owns it, and it is exactly the number of bytes you asked for. None of them is padded zeros with an extension @@ -219,6 +219,13 @@

24 file formats, every one generated at an exact size

full pillow + + toml + .toml + 212 + full + python-toml + txt .txt @@ -254,6 +261,13 @@

24 file formats, every one generated at an exact size

full python-xml + + yaml + .yaml + 241 + full + python-yaml + zip .zip diff --git a/web/public/index.html b/web/public/index.html index eb0f631b..299bf442 100644 --- a/web/public/index.html +++ b/web/public/index.html @@ -3,7 +3,7 @@ -Test File Generator for QA - Exact Size, 24 Real Formats +Test File Generator for QA - Exact Size, 26 Real Formats @@ -12,7 +12,7 @@ - + @@ -21,7 +21,7 @@ - + @@ -35,7 +35,7 @@ "@type": "SoftwareApplication", "name": "Testing Files Generator", "alternateName": "tfg", - "description": "A free and open source generator of test files for QA. It produces real files of twenty formats at any exact size and writes a manifest saying how the system under test should react to each one.", + "description": "A free and open source generator of test files for QA. It produces real files of 26 formats at any exact size and writes a manifest saying how the system under test should react to each one.", "url": "https:\/\/testingfilesgenerator.donislawdev.com/", "applicationCategory": "DeveloperApplication", "applicationSubCategory": "Software Testing", @@ -87,7 +87,7 @@

Generate real test files at any exact size

- PDF, PNG, DOCX, ZIP - 24 formats in all, and every one is a + PDF, PNG, DOCX, ZIP - 26 formats in all, and every one is a real file that opens in the software that owns it, at exactly the size you asked for. Each run also writes down what your application is supposed to do with each file. Command line and desktop window, free and open source, working entirely on your machine. @@ -109,7 +109,7 @@

Generate real test files at any exact size

  • - 24 + 26

    real formats, each one opening in the software that owns it

  • @@ -229,7 +229,7 @@

    Built for a suite that runs unattended

    Ask for 10485761 bytes and get exactly that. A size a format cannot reach is an error with a reason, never a file of the wrong size.

  • -

    24 real formats

    +

    26 real formats

    Not padded zeros with an extension. A generated PNG opens in an image viewer, a DOCX opens in Word, a ZIP extracts. Each one is checked against independent readers before it ships.

  • diff --git a/web/public/pl/faq/index.html b/web/public/pl/faq/index.html index fb208a9f..7a3fba8d 100644 --- a/web/public/pl/faq/index.html +++ b/web/public/pl/faq/index.html @@ -135,7 +135,7 @@

    Najczęstsze pytania

    Jakie formaty są następne w kolejce?
    -

    7z, mp3 i mp4. Formatów działających dziś od początku do końca jest 24.

    +

    7z, mp3 i mp4. Formatów działających dziś od początku do końca jest 26.

    @@ -206,7 +206,7 @@

    Najczęstsze pytania

    { "@type": "Question", "name": "Jakie formaty są następne w kolejce?", - "acceptedAnswer": { "@type": "Answer", "text": "7z, mp3 i mp4. Formatów działających dziś od początku do końca jest 24." } + "acceptedAnswer": { "@type": "Answer", "text": "7z, mp3 i mp4. Formatów działających dziś od początku do końca jest 26." } }, { "@type": "Question", diff --git a/web/public/pl/formaty/index.html b/web/public/pl/formaty/index.html index 375a5a36..4a071af9 100644 --- a/web/public/pl/formaty/index.html +++ b/web/public/pl/formaty/index.html @@ -3,7 +3,7 @@ -24 formatów plików testowych - PDF, DOCX, PNG, ZIP +26 formatów plików testowych - PDF, DOCX, PNG, ZIP @@ -12,7 +12,7 @@ - + @@ -21,7 +21,7 @@ - + @@ -74,7 +74,7 @@
    -

    24 formatów plików, każdy generowany o dokładnym rozmiarze

    +

    26 formatów plików, każdy generowany o dokładnym rozmiarze

    Każdy z nich to prawdziwy plik tego formatu. Otwiera się w programie, do którego należy, i ma dokładnie tyle bajtów, ile zamówiłeś. Żaden nie jest zerami z doklejonym rozszerzeniem. @@ -218,6 +218,13 @@

    24 formatów plików, każdy generowany o dokładnym rozmiarze

    full pillow + + toml + .toml + 212 + full + python-toml + txt .txt @@ -253,6 +260,13 @@

    24 formatów plików, każdy generowany o dokładnym rozmiarze

    full python-xml + + yaml + .yaml + 241 + full + python-yaml + zip .zip diff --git a/web/public/pl/index.html b/web/public/pl/index.html index 8beea433..5c361490 100644 --- a/web/public/pl/index.html +++ b/web/public/pl/index.html @@ -3,7 +3,7 @@ -Generator plików testowych o zadanym rozmiarze - 24 formatów +Generator plików testowych o zadanym rozmiarze - 26 formatów @@ -12,7 +12,7 @@ - + @@ -21,7 +21,7 @@ - + @@ -35,7 +35,7 @@ "@type": "SoftwareApplication", "name": "Testing Files Generator", "alternateName": "tfg", - "description": "Darmowy generator plików testowych dla QA o otwartym kodzie. Tworzy prawdziwe pliki w dwudziestu formatach o dokładnie zadanym rozmiarze i zapisuje manifest mówiący, jak testowany system ma na każdy z nich zareagować.", + "description": "Darmowy generator plików testowych dla QA o otwartym kodzie. Tworzy prawdziwe pliki w 26 formatach o dokładnie zadanym rozmiarze i zapisuje manifest mówiący, jak testowany system ma na każdy z nich zareagować.", "url": "https:\/\/testingfilesgenerator.donislawdev.com/", "applicationCategory": "DeveloperApplication", "applicationSubCategory": "Software Testing", @@ -87,7 +87,7 @@

    Generuj pliki testowe o zadanym rozmiarze

    - PDF, PNG, DOCX, ZIP - razem 24 formatów, a każdy to + PDF, PNG, DOCX, ZIP - razem 26 formatów, a każdy to prawdziwy plik, który otwiera się w programie, do którego należy, i ma dokładnie taki rozmiar, o jaki poprosisz. Każdy przebieg zapisuje też, co Twoja aplikacja ma z każdym plikiem zrobić. Wiersz poleceń i okno, darmowe i otwarte, działające wyłącznie na Twojej maszynie. @@ -109,7 +109,7 @@

    Generuj pliki testowe o zadanym rozmiarze

    • - 24 + 26

      prawdziwych formatów, każdy otwiera się w programie, do którego należy

    • @@ -229,7 +229,7 @@

      Zbudowane pod zestaw, który chodzi bez nadzoru

      Poproś o 10485761 bajtów i tyle dostaniesz. Rozmiar nieosiągalny dla formatu to błąd z powodem, nigdy plik o innym rozmiarze.

    • -

      24 prawdziwych formatów

      +

      26 prawdziwych formatów

      Nie zera z doklejonym rozszerzeniem. Wygenerowany PNG otwiera się w przeglądarce obrazów, DOCX w Wordzie, a ZIP się rozpakowuje. Każdy format jest sprawdzany niezależnym czytnikiem, zanim trafi do wydania.

    • diff --git a/web/public/social.html b/web/public/social.html index 438b9dff..bae00697 100644 --- a/web/public/social.html +++ b/web/public/social.html @@ -210,12 +210,12 @@

      Real test files.
      At any exact size.

      - 24 formats that open in the software that owns them, plus a manifest + 26 formats that open in the software that owns them, plus a manifest saying how your system should react to each file.

      • exact to the byte
        • -
      • 24 real formats
        • +
      • 26 real formats
      • same bytes every run
      • GUI + CLI
      • built for CI
        • diff --git a/web/social-preview.sha256 b/web/social-preview.sha256 index 4b2ac8d4..c775cf2b 100644 --- a/web/social-preview.sha256 +++ b/web/social-preview.sha256 @@ -1 +1 @@ -9ef07b206eecd220eda115f5eb9b2f7beaebb02a0dfe2e9754679beb3f3f536d +a1f399d28e5c2484b80e9b2a27654b2d727899354ead717be02eb279b7787b62