Repository navigation
Excessive rebuilding when nothing changed #3346
Description
Describe the bug
In the last 3 weeks (between 36051bd and the current main branch) something changed that causes cargo to do partial rebuild when it is not really necessary. This is especially noticeable in VSCode on slow machines or while working with containers and trying to execute something on command line.
Steps to reproduce (if applicable)
Steps to reproduce the behavior:
- Run the following 3 commands:
cargo clean
cargo build --features=postgres
cargo run --features=postgres -- run --service searcher --service metastore --config ../config/quickwit.yaml
Expected behavior
3 weeks ago the run command started the service immediately. Now it goes through a partial rebuild process that takes minutes on some machine.
Activity
trinity-1686a commentedon May 16, 2023
It might somehow be a compiler regression. I've bisected it to #3244, and the problematic part seems to be going from rust 1.68 to 1.69. On 9c43680 (main right now), I don't have the issue if I force compiling with 1.68
imotov commentedon May 16, 2023
That's unfortunate. I suspected it might be the case, but was hoping it's just some easy to fix bug in a build script.
imotov commentedon May 17, 2023
@guilload suggested running
rustup override set 1.68.0as a work-around. It worked for me.imotov commentedon May 19, 2023
I think we might be also affected by bytecodealliance/rustix#562
I see this all the time while running tests in VSCode on both 1.68 and 1.69
sunfishcode commentedon May 19, 2023
Could you try adding
--verbose? With this flag, cargo adds "Dirty" messages which indicate what prompted it to perform rebuilds.imotov commentedon May 19, 2023
@sunfishcode with
--verboseI get this:That in turn triggers rebuild of the rest of quickwit codebase that depends on quickwit-common.
imotov commentedon May 19, 2023
@sunfishcode one additional observation that might be important here. We also pull rustix v0.37.19 as part of another dependency and it doesn't have the same issue.
sunfishcode commentedon May 19, 2023
Thanks! Looking into it, this is likely to be due to an issue which was fixed in rustix 0.37, so I've now backported the fix to 0.36 and released it in 0.36.14.
imotov commentedon May 20, 2023
@sunfishcode wow, that was fast! Thanks a lot, that really helps. While it doesn't solve the original issue (it seems that we have a somewhat similar problem with proc-macro2) the new version resolved many issues that I had in VS Code interactions while switching between building and running tests.
imotov commentedon May 20, 2023
I think I figured it out. It looks like a regression in rust. Unfortunately, I cannot reproduce it using a small project with a manageable size, but I know the mechanism that is causing the issue and know how to fix it.
I suspect the issue is triggered by this change in 1.69.0 that is producing different command lines for building crates with
edition = "2018"depending on how you build it. We have quite a few of these in our dependency tree, but it all starts with unicode_ident. When it is built using cargo build command it is compiled like thisWhen it is built using
cargo runit is compiled like this:The lines are basically identical with a very small inclusion. The first line has
-C debuginfo=0in it, which also changes fingerprint and causes all that rebuilding. The fix is actually quite silly. Replacingwith
fixes the issue.
Upgrade rustix 0.36.13->0.36.14 and fix debug info
Upgrade rustix 0.36.13->0.36.14 and fix debug info (#3379)
imotov commentedon May 20, 2023
I opened rust-lang/cargo#12163
imotov commentedon May 21, 2023
Based on the description of the issue in the previous comment, we might need to circle back to this. The workaround that I proposed is currently not working in nightly.