Repository navigation
Speed up doc generation for projects with many applications - #2284
Closed
robertoaloi wants to merge 5 commits into
Closed
robertoaloi wants to merge 5 commits into
robertoaloi wants to merge 5 commits into
Conversation
Without the cache, every module lookup in the code server tries each code path entry in turn. The CLI prepends one entry per documented application, so on projects with hundreds of applications lookups get slower as the project grows, and they all queue up on the code server.
Each directory used to be processed on its own, waiting for its slowest module before moving on. Projects made of many small applications now get the same parallelism as a single large one.
Their references are computed from their exports, callbacks and types, which requires loading them. Loading them one by one from the filtering tasks serializes on the code server, like it did for the kept modules.
Finding the application of a module and whether it is an OTP one goes through the application controller and the code server for every reference, and both lookups get slower with the number of loaded applications and code path entries.
:code.get_object_code/1 searches the code path and reads the file through the code server and the primitive loader. Abstract code and types are fetched for most modules, concurrently, so these single processes become a bottleneck on large projects.
|
📦 Docs artifacts are ready: https://github.com/elixir-lang/ex_doc/actions/runs/43257827722/artifacts/13963600168 |
Member
|
Thank you, I pushed 1, 2 and 5 manually. For 4, i tried 283a5a7, please measure! I will send another PR for 3. |
Member
|
@robertoaloi this is another improvement to avoid object code contention: 09676c7 I played a bit with 4 but I worry that we will know also load modules that are not used at any point. And we shouldn't really be referring to things that have no docs chunk (as they would emit warnings). Thoughts? |
Contributor
Author
|
@josevalim That will do. The most important bit of this PR was abc9384, the one that makes the real difference for us. Thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Speed up doc generation for projects with many applications
On a large Erlang project HTML generation is now ~40x faster, with byte-identical output.
The CLI prepends one code path entry per application, uncached. Every code server lookup (loading,
:code.which/1,:code.get_object_code/1,:code.lib_dir/1) then goes through all of them, one at a time, inside a single process. Under an escript, every failed read also resolves symlinks on each path component.Commits:
cache: true(~20x on its own; no effect on OTP < 26).app/1andis_app_otp/1in autolink.:code.which/1reports instead of using:code.get_object_code/1.mix testpasses on OTP 27 and 28. No public API changes.