Skip to content

Speed up doc generation for projects with many applications - #2284

Closed
robertoaloi wants to merge 5 commits into
elixir-lang:mainfrom
robertoaloi:faster-large-projects
Closed

robertoaloi wants to merge 5 commits into
elixir-lang:mainfrom
robertoaloi:faster-large-projects

Conversation

@robertoaloi

Copy link
Copy Markdown
Contributor

Speed up doc generation for projects with many applications

On a large Erlang project HTML generation is now ~40x faster, with byte-identical output.

The CLI prepends one code path entry per application, uncached. Every code server lookup (loading, :code.which/1, :code.get_object_code/1, :code.lib_dir/1) then goes through all of them, one at a time, inside a single process. Under an escript, every failed read also resolves symlinks on each path component.

Commits:

  1. Prepend the code paths with cache: true (~20x on its own; no effect on OTP < 26).
  2. Retrieve docs from all directories in one batch instead of one directory at a time.
  3. Load modules without a docs chunk in one call, as Filter modules before loading them and load the kept ones in one call #2283 does for kept modules.
  4. Memoize app/1 and is_app_otp/1 in autolink.
  5. Read beam files from the path :code.which/1 reports instead of using :code.get_object_code/1.

mix test passes on OTP 27 and 28. No public API changes.

Without the cache, every module lookup in the code server tries each
code path entry in turn. The CLI prepends one entry per documented
application, so on projects with hundreds of applications lookups get
slower as the project grows, and they all queue up on the code server.
Each directory used to be processed on its own, waiting for its slowest
module before moving on. Projects made of many small applications now
get the same parallelism as a single large one.
Their references are computed from their exports, callbacks and types,
which requires loading them. Loading them one by one from the filtering
tasks serializes on the code server, like it did for the kept modules.
Finding the application of a module and whether it is an OTP one goes
through the application controller and the code server for every
reference, and both lookups get slower with the number of loaded
applications and code path entries.
:code.get_object_code/1 searches the code path and reads the file through
the code server and the primitive loader. Abstract code and types are
fetched for most modules, concurrently, so these single processes become
a bottleneck on large projects.
@github-actions

Copy link
Copy Markdown

@josevalim

Copy link
Copy Markdown
Member

Thank you, I pushed 1, 2 and 5 manually. For 4, i tried 283a5a7, please measure! I will send another PR for 3.

@josevalim josevalim closed this Nov 12, 2026
@josevalim

Copy link
Copy Markdown
Member

@robertoaloi this is another improvement to avoid object code contention: 09676c7

I played a bit with 4 but I worry that we will know also load modules that are not used at any point. And we shouldn't really be referring to things that have no docs chunk (as they would emit warnings). Thoughts?

@robertoaloi

Copy link
Copy Markdown
Contributor Author

@josevalim That will do. The most important bit of this PR was abc9384, the one that makes the real difference for us. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants