Skip to content

Build an executable from an object per source - #118

Merged
Leitwolf11 merged 3 commits into
mainfrom
feature/incremental-compilation
Oct 31, 2026
Merged

Leitwolf11 merged 3 commits into
mainfrom
feature/incremental-compilation

Conversation

@Leitwolf11

Copy link
Copy Markdown
Contributor

First step of incremental compilation. --incremental=<directory> keeps one object per source, keyed on the Xmm of the source and the signatures of what it calls, and lowers only what changed, in parallel.

Measured on a generated program of 13 sources and about 4,800 methods, with a fastbuild talac on Windows, so absolute times are pessimistic: a changed body lowers 1 of 13 sources, a comment or a moved line 0 of 13, a new method in the third file 9 of 13.

The same measurement shows the larger part of the time is before LLVM: about 6 s in the frontend and about 9 s from Core to Xmm in a fastbuild, against 3 to 5 s in LLVM and the link. So this is not the whole job: the next steps are symbols that keep their number when another function appears, and reuse in the frontend.

Leitwolf11 added 2 commits October 31, 2026 10:20
talac rebuilt everything on every run. --incremental=<directory> builds
one object per source through the source partitions the backend already
had, plus one for the process entry, and keeps them in the directory.

An object is known by the Xmm of the functions of its source, the
signatures of what they call in other sources, and what else decides it:
the compiler itself, the target and the optimization level. The Xmm is
what the frontend and the Xpp and Xmm passes left, so a comment or a moved
line lowers nothing, a changed body lowers its source, and a changed
signature also lowers its callers. Only the sources that changed are
lowered, on several threads, each in an LLVM context of its own.

A key is made from functions that do not form a module, so the wire gets
EncodeFunctions, which verifies nothing and cannot be decoded. An entry
object has no functions of its own, which Options::entry_only says.

The frontend still reads every source, and symbols are numbered across
the program, so a new function lowers the sources after it again. LLVM
sees one source at a time, so calls between sources are not inlined. Both
are in Documents/CLI.md.
Sources without declarations share the same functions, none, so their
keys were equal: two workers wrote one staged file and the linker saw the
object twice. The key now covers the name of the source.

The workers ran on the default thread stack, 512 KiB on macOS, while LLVM
recurses over the code it lowers. They now use llvm::thread with 8 MiB,
which also drops std::jthread, absent from Apple libc++.
@Leitwolf11

Copy link
Copy Markdown
Contributor Author

Self-review of the incremental build.

Checked and sound. xmm::Module holds only its name, functions and source catalog, so an object depends on nothing the key misses: types and literals live in the instructions, and every reference to another function is a Value::Kind::Function or closure_function, both collected. Linking options do not reach Lower, so --sysroot and --static-talart-library rightly stay out of the context. Publish renames a complete file, so an interrupted build leaves no believable object. The backend's only shared state is a call_once, so parallel lowering shares nothing else.

Found and fixed in the last commit.

  • Sources without declarations had equal keys. Two workers wrote one staged file and the linker saw the object twice. The key now covers the source name, with a regression test that failed before the fix.
  • Workers ran on the default thread stack, 512 KiB on macOS, and std::jthread is absent from Apple libc++. They now use llvm::thread with 8 MiB.

Known limits, not defects. Lower verifies and declares the whole module once per recompiled source and still prints IR and bitcode nobody reads here. A new function renumbers later symbols, so it recompiles more than it should. Both are the next steps.

Windows locally: incremental_build_tests passes; CI for this head covers Linux and macOS.

Symbol ids are numbered across the program and are part of the native
name, and generated spellings such as $closure12 carry them too, so one
new method changed the objects of the sources after it.

BuildObjects now derives each id from the source, the spelling without
those digits and which of that spelling the function is. References are
rewritten before the keys are made, so a call that moves to another
function still recompiles its caller.

Measured on 13 sources and about 4800 methods: a new method recompiles
1 source instead of 9.
@Leitwolf11

Copy link
Copy Markdown
Contributor Author

Self-review of the follow-up commit (stable symbols).

BuildObjects rewrites ids before anything is keyed or lowered, so keys and objects see the same names. The risk with a derived id is a stale caller: the id depends on the order of same-spelled functions, so swapping them could bind a caller to the wrong body. It cannot, because a caller's key hashes its rewritten references: a call that moves changes the owned bytes and recompiles it, and a call that moves to another function recompiles the caller covers this. Hash collisions are probed within the build, so ids stay unique. Spellings lose digits only when they contain $, which source identifiers cannot.

Windows locally: incremental_build_tests passes (8 cases), and the generated 13-source program ran after a new method recompiled 1 of 13 sources (9 before). Native symbol names now hold a hash rather than a small number, in incremental builds only.

@Leitwolf11

Copy link
Copy Markdown
Contributor Author

Correction to the self-review and to commit 2ba95cf: "std::jthread is absent from Apple libc++" came from an old comment in OwnershipFuzzer.cpp, not from a check, and the earlier head's macOS checks were green. The move to llvm::thread still stands for the stack size.

@Leitwolf11
Leitwolf11 merged commit 642a297 into main Oct 31, 2026
43 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant