Skip to content

engine: a name up to 255 bytes is written, and a longer one is refused everywhere #137

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
donislawdev merged 1 commit into from
Sep 24, 2026
Merged

engine: a name up to 255 bytes is written, and a longer one is refused everywhere #137

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,13 @@ because it turns other people's test suites red.

### Changed

- **A file name longer than 255 bytes is refused before anything is written,
on every system.** Linux stores at most 255 bytes in a name, while Windows
and macOS count characters, so a name of 200 Chinese or Japanese characters
used to be written on two systems and fail on the third with no reason
given. The refusal says how long the name is and how long it may be. A
letter outside ASCII takes two to four of those bytes.

- **The window's look, after a review of every screen.** An open list and the
explanation beside a field stand on a card with an edge and a shade instead
of a flat grey block, and a list opened under its box shrinks to what the
Expand Down Expand Up @@ -500,6 +507,12 @@ because it turns other people's test suites red.

### Fixed

- **A file name from 238 to 255 bytes long is written.** Every system stores
such a name, and none of them got one: each file is written under a longer
temporary name first, and that one was over the limit. The same held for a
manifest named that long and for `tfg recipe fmt -w` on a recipe file named
that long.

- **The window uses far less memory, and rebuilding a screen or opening a
list no longer adds to it.** Every quiet line on a screen - a subtitle, a
caption, the count of bytes beside a size, the line a folded section keeps,
Expand Down
7 changes: 5 additions & 2 deletions internal/core/limits.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,9 @@ const MaxPlanBytes = 2 << 30
// Every file goes out under a temporary name and is renamed into place, so the
// output directory never holds a half written file. The full name is
// "<final>.tfg-partial-<process id>", with the process id there because two
// runs writing into one directory used to meet on the temporary file.
// runs writing into one directory used to meet on the temporary file. A final
// name too long for that to fit in MaxNameBytes is shortened in front of the
// marker, never after it - SiblingName, since 2026-09-24 (O239).
//
// The marker is declared here rather than built at the point of use, because
// two parts of the tool have to agree on it: the engine writes it, and the
Expand Down Expand Up @@ -124,7 +126,8 @@ func IsPartialName(name string) bool {
// somebody - the manifest being rewritten over an earlier one, or the recipe
// that "recipe fmt -w" is formatting in place. The full name is
// "<final>.tfg-writing", with no process id, because the name is claimed
// exclusively rather than made unique.
// exclusively rather than made unique. Shortened in front of the marker the
// same way when it would not fit, by SiblingPath.
//
// Declared here beside PartialMarker on 2026-09-06, and the argument for it is
// the one already written above: two parts of the tool have to agree on the
Expand Down
4 changes: 3 additions & 1 deletion internal/core/replace.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,9 @@ func ReplaceFile(path string, content []byte) error {
return err
}

tmp := path + writingSuffix
// A sibling rather than a plain join, so a recipe whose own name is up to
// the length every system stores can still be formatted in place (O239).
tmp := SiblingPath(path, writingSuffix)
if err := writeWhole(tmp, content, mode); err != nil {
// Only what this call created is taken away. A refusal from CreateNew
// means the name was already somebody's - a leftover from an
Expand Down
75 changes: 75 additions & 0 deletions internal/core/sibling.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
package core

import (
"crypto/sha256"
"encoding/hex"
"path/filepath"
"unicode/utf8"
)

// MaxNameBytes is the longest file name, in bytes of UTF-8, that every system
// this tool writes on will store.
//
// The three file systems count three different things, each up to 255: ext4
// counts bytes, NTFS counts UTF-16 units and APFS counts characters. A name
// never has fewer bytes than either of the other two, so a name that fits in
// 255 bytes fits on all three. Measured on 2026-09-24 with one recipe on each
// (docs/O239-LONG-NAMES-2026-09-24.md): a name of 200 CJK characters, 604
// bytes, was stored on Windows and on macOS and cannot be on Linux.
const MaxNameBytes = 255

// siblingTagDigits is how much of the digest of the whole name a shortened
// sibling carries, in hex digits. Sixty four bits, which puts the chance that
// two names of one run share a sibling at about three in a million million
// for the largest preset there is, 10 040 files.
const siblingTagDigits = 16

// SiblingName is the name of a file that stands beside another one while it is
// being written: the name with suffix after it, whenever that fits.
//
// It exists because the plain join did not always fit. Every file of a run is
// written as "<name>.tfg-partial-<pid>" and renamed afterwards, and that suffix
// is eighteen bytes, so a name from 238 bytes up was one no file system would
// take in its temporary form - on every system, while the name itself was
// perfectly legal (O239, measured on 2026-09-24 on Windows, Linux and macOS).
//
// When the join is longer than MaxNameBytes, the name is cut to whole
// characters and followed by "~" and the start of the SHA-256 of the whole
// name, and then the suffix. The digest is what keeps two long names that
// begin alike apart - a name template numbering files at the end of a long
// name gives exactly that, inside one run. The suffix stays last in both
// forms, so what an interrupted run leaves behind is recognised by
// IsPartialName and IsWritingName either way.
//
// A name short enough comes back as the plain join, byte for byte, so nothing
// changed for any name this tool could write before.
func SiblingName(name, suffix string) string {
if len(name)+len(suffix) <= MaxNameBytes {
return name + suffix
}
sum := sha256.Sum256([]byte(name))
tag := "~" + hex.EncodeToString(sum[:])[:siblingTagDigits]
return cutToWholeCharacters(name, MaxNameBytes-len(suffix)-len(tag)) + tag + suffix
}

// SiblingPath is SiblingName for a path: the sibling in the same directory,
// with the directory spelt exactly as it was given.
func SiblingPath(path, suffix string) string {
dir, name := filepath.Split(path)
return dir + SiblingName(name, suffix)
}

// cutToWholeCharacters is the longest start of s that is no longer than n bytes
// and does not end part way through a character.
func cutToWholeCharacters(s string, n int) string {
if n <= 0 {
return ""
}
if len(s) <= n {
return s
}
for n > 0 && !utf8.RuneStart(s[n]) {
n--
}
return s[:n]
}
167 changes: 0 additions & 167 deletions internal/engine/engine.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -766,173 +766,6 @@ func renderName(t *Target, d format.Descriptor, index int) (string, error) {
return name, nil
}

// checkFileName keeps a name a name.
//
// A name carrying a path escapes the directory the run was pointed at. A
// recipe travels between teams by design, so "../../something" in a file
// somebody sent over would write outside the directory its reader chose - and
// the free space check, the collision check and cleanup all work on the
// directory, so none of them would be looking in the right place.
//
// Both separators are refused on every system, not just the local one. A name
// holding a backslash is legal on Linux and cannot exist on Windows, and a
// recipe that only works on the machine it was written on is not portable.
func checkFileName(setting, where, name string) error {
switch {
case name == "":
return &RecipeError{Setting: setting, Detail: fmt.Sprintf("%s produces a file with no name", where)}

case strings.ContainsAny(name, `/\`):
return &RecipeError{Setting: setting,
Detail: fmt.Sprintf("%s produces the name %q, which is a path rather than a file name", where, name),
Because: "names stay inside the output directory, and a separator is refused on every system so that a recipe works everywhere",
Remedy: "Choose the directory with the output setting instead"}

// A colon, on every system, for the same reason as a separator.
//
// Windows reads it as the start of an alternate data stream, so
// "AB:c.txt" names a stream called c.txt inside a file called AB.
// Measured on 2026-08-04: the run reported the file as not produced and
// ended with the partial code, and an empty file called AB was left in the
// directory anyway - not in the manifest, reported by verify as something
// nobody asked for, and beyond the reach of cleanup for good. A single
// letter in front of it is read as a drive instead, which is how the same
// recipe came to be accepted on Linux and refused on Windows.
//
// Legal in a name on Linux and macOS, and refused there too. A recipe
// travels between machines by design, and one that quietly leaves debris on
// somebody else's is worse than one refused on all of them.
case strings.Contains(name, ":"):
return &RecipeError{Setting: setting,
Detail: fmt.Sprintf("%s produces the name %q, which holds a colon", where, name),
Because: "Windows reads that as a drive or as an alternate data stream rather than as part of the name, so the file arrives called something else or not at all. It is refused on every system so that a recipe means one thing everywhere",
Remedy: "Take the colon out, or ask for the file inside an archive where the name survives"}

// Characters Windows will not put in a file name, refused on every system
// for the same reason as the separator and the colon above.
//
// Measured on 2026-08-25, on each of <>"|?* and on a name holding a tab.
// All seven planned cleanly, --dry-run answered "1 file in 1 target" and
// exit 0, and the run then failed that one file with the system's own
// words: "open a<b.txt.tfg-partial-53628: The filename, directory name, or
// volume label syntax is incorrect". Three things wrong in one line. The
// dry run answered for a run that could not happen, which is the fault
// preflight exists to stop. The sentence is not ours and carries none of
// the four parts a refusal owes a reader, while carrying the temporary
// name, which is ours and nobody else's business. And the same recipe
// writes the file on Linux, where all of these are legal - the reason the
// separator has been refused everywhere since 2026-08-03.
//
// Producing such a name on purpose is a real test case and it belongs to
// the name laboratory, which writes it into an archive rather than onto
// the host filesystem. See D10.
case firstForbidden(name) != 0:
bad := firstForbidden(name)
return &RecipeError{Setting: setting,
Detail: fmt.Sprintf("%s produces the name %q, which holds %s", where, name, describeForbidden(bad)),
Because: "Windows refuses that character in a file name, so the file is not written there at all. It is refused on every system so that a recipe means one thing everywhere",
Remedy: "Take the character out, or ask for the file inside an archive where the name survives"}

// One reserved device name, and one only.
//
// The folklore list has twenty two - CON, PRN, AUX, COM1 to COM9, LPT1 to
// LPT9 - and every one of them was measured on 2026-08-25 rather than
// remembered, on two editions: Windows 11 Pro 26200 and Windows Server
// 2025 build 26100. con, con.txt, prn, aux, com1, com1.bin, lpt1 and
// conin$ each came back an ordinary file that verify then passed, on both.
// Refusing that list would refuse con.pdf, a name both systems store
// perfectly well, which is a rule written from memory doing damage.
//
// NUL is the exception on both, and it is the one worth catching, because
// it does not fail. The write succeeds and the bytes go nowhere, which is
// the silence rule broken as completely as it can be - a manifest
// describing a file that was never on the disk. The run is refused a step
// later today, by the collision check finding something at the path, and
// that is safe but tells the reader to remove a file nobody can remove.
//
// The bare name only. An extension saves it - nul.txt is an ordinary file
// on both editions - so this is not the "any extension" rule the folklore
// describes either.
case strings.EqualFold(name, "nul"):
return &RecipeError{Setting: setting,
Detail: fmt.Sprintf("%s produces the name %q, which names the null device on Windows rather than a file", where, name),
Because: "writing there succeeds and the bytes go nowhere, so the run would record a file that is not on the disk. It is refused on every system so that a recipe means one thing everywhere",
Remedy: "Give it an extension, nul.txt is an ordinary name, or choose another one"}

case name == "." || name == "..":
return &RecipeError{Setting: setting, Detail: fmt.Sprintf(
"%s produces the name %q, which names a directory rather than a file", where, name)}

// A name Windows stores under a different name than the one it was given.
// Refused on every system for the same reason a separator is: a recipe that
// only works on the machine it was written on is not portable.
//
// Measured on 2026-08-03, and it is the silence rule broken rather than a
// portability nicety. "--name trailing." finished with exit code 0, the
// file landed as "trailing", and the manifest recorded "trailing." - so the
// run described a file that was not there under that name, and "tfg verify"
// on the tool's own output failed with exit code 7 a second later.
//
// Producing such a name deliberately is a real test case and it belongs to
// the name laboratory, which writes it into an archive rather than onto the
// host filesystem for exactly this reason. See D10.
case strings.HasSuffix(name, ".") || strings.HasSuffix(name, " "):
return &RecipeError{Setting: setting,
Detail: fmt.Sprintf("%s produces the name %q, which ends in a dot or a space", where, name),
Because: "Windows stores such a name without it, so the file on disk would not be the file the manifest describes and verify would report both",
Remedy: "Take the last character off, or ask for the file inside an archive where the name survives"}

// Judged the same way on every system, like the separator above. Using
// filepath here asks the machine this build runs on, and the answer
// differs: measured on 2026-08-04, "a:b.txt" was accepted on Linux and
// refused on Windows from one recipe, so a fixture set written on one
// machine failed on the next. That is the failure this rule exists to
// prevent, arriving through the rule itself.
case filepath.IsAbs(name) || core.HasVolumeName(name):
return &RecipeError{Setting: setting,
Detail: fmt.Sprintf("%s produces the absolute path %q", where, name),
Because: "a recipe carries no absolute paths, because then it only works on the machine it was written on",
Remedy: "Choose the directory with the output setting instead"}
}
return nil
}

// forbiddenChars are the printable characters Windows refuses in a file name.
//
// The separator and the colon are not here. They are refused above with their
// own sentences, because what goes wrong with them is not "the file is not
// written" but something worse and worth its own explanation - a name that
// leaves the output directory, and a name Windows reads as a drive or as a
// stream inside another file.
const forbiddenChars = `<>"|?*`

// firstForbidden is the first character of a name that Windows will not store,
// or zero when there is none.
//
// The first rather than all of them, because a refusal naming one character a
// reader can find beats a list they have to compare against their own name.
func firstForbidden(name string) rune {
for _, r := range name {
// Below the space, which is every control character. Windows refuses
// the whole range, and a name holding one is unreadable on any system
// - a tab in a file name is a name nobody can type back.
if r < 0x20 || strings.ContainsRune(forbiddenChars, r) {
return r
}
}
return 0
}

// describeForbidden names a character in a way somebody can act on. A control
// character has nothing to show, so it is given as its number instead of being
// printed into the middle of a sentence where it would do what it says.
func describeForbidden(r rune) string {
if r < 0x20 {
return fmt.Sprintf("a control character, U+%04X", r)
}
return fmt.Sprintf("the character %q", string(r))
}

func runID(seed int64) string {
h := sha256.Sum256([]byte(fmt.Sprintf("run:%d", seed)))
return "run_" + hex.EncodeToString(h[:5])
Expand Down
Loading
Loading