Repository navigation
1.55.0 segfault compiling stage1 rustc_codegen_llvm on powerpc64le #89744
Description
Failure log (warning very large): 1.55
There was no failure on 1.54; the same version of LLVM was used in both cases - 12.0.1 (Debian version 1:12.0.1-9)
The only change made to rustc_codegen_llvm between 1.54 and 1.55 was #86416 so CC @Amanieu .
There was a later MR #88350 to fix something on powerpc64, however this just mentions "lack of support" rather than a segfault, so I don't know if this is related. Shall I try backporting it onto 1.55 in the meantime?
I will continue trying to debug the segfault.
The output contains the following stack-trace-like dump, not sure if it's useful:
/<<PKGBUILDDIR>>/build/powerpc64le-unknown-linux-gnu/stage1/lib/librustc_driver-8a51db685c3a3652.so(+0x786b6c)[0x7fffa65f6b6c]
linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x7fffaa190514]
/<<PKGBUILDDIR>>/build/powerpc64le-unknown-linux-gnu/stage1/lib/librustc_driver-8a51db685c3a3652.so(+0x2b16f64)[0x7fffa8986f64]
/<<PKGBUILDDIR>>/build/powerpc64le-unknown-linux-gnu/stage1-rustc/release/deps/libcstr-786feb209dfce7ae.so(+0xc433c)[0x7fff87f7433c]
/<<PKGBUILDDIR>>/build/powerpc64le-unknown-linux-gnu/stage1-rustc/release/deps/libcstr-786feb209dfce7ae.so(+0xb1e50)[0x7fff87f61e50]
/<<PKGBUILDDIR>>/build/powerpc64le-unknown-linux-gnu/stage1-rustc/release/deps/libcstr-786feb209dfce7ae.so(_ZN10proc_macro12token_stream95_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$proc_macro..TokenStream$GT$9
into_iter17h32ab5102ba0e87e0E+0x28)[0x7fff87f47978]
/<<PKGBUILDDIR>>/build/powerpc64le-unknown-linux-gnu/stage1-rustc/release/deps/libcstr-786feb209dfce7ae.so(_ZN91_$LT$proc_macro2..imp..TokenStream$u20$as$u20$core..iter..traits..collect..IntoIterator$GT$9into_iter17h2efdc81b2e8ae367E+
0x1e0)[0x7fff87f35280]
/<<PKGBUILDDIR>>/build/powerpc64le-unknown-linux-gnu/stage1-rustc/release/deps/libcstr-786feb209dfce7ae.so(_ZN11proc_macro212token_stream96_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$proc_macro2..TokenStream$GT
$9into_iter17h315ff4dea038bfc3E+0x54)[0x7fff87f3ab84]
/<<PKGBUILDDIR>>/build/powerpc64le-unknown-linux-gnu/stage1-rustc/release/deps/libcstr-786feb209dfce7ae.so(+0x6a200)[0x7fff87f1a200]
/<<PKGBUILDDIR>>/build/powerpc64le-unknown-linux-gnu/stage1-rustc/release/deps/libcstr-786feb209dfce7ae.so(+0x6bfa0)[0x7fff87f1bfa0]
/<<PKGBUILDDIR>>/build/powerpc64le-unknown-linux-gnu/stage1-rustc/release/deps/libcstr-786feb209dfce7ae.so(+0x6dbf8)[0x7fff87f1dbf8]
/<<PKGBUILDDIR>>/build/powerpc64le-unknown-linux-gnu/stage1-rustc/release/deps/libcstr-786feb209dfce7ae.so(+0x6e654)[0x7fff87f1e654]
/<<PKGBUILDDIR>>/build/powerpc64le-unknown-linux-gnu/stage1-rustc/release/deps/libcstr-786feb209dfce7ae.so(+0x6d478)[0x7fff87f1d478]
/<<PKGBUILDDIR>>/build/powerpc64le-unknown-linux-gnu/stage1/lib/librustc_driver-8a51db685c3a3652.so(+0x2a474f0)[0x7fffa88b74f0]
/<<PKGBUILDDIR>>/build/powerpc64le-unknown-linux-gnu/stage1/lib/librustc_driver-8a51db685c3a3652.so(_ZN89_$LT$rustc_expand..proc_macro..BangProcMacro$u20$as$u20$rustc_expand..base..ProcMacro$GT$6expand17h4da03810eb58a713E+0x12c)[0x7ff
fa889171c]
/<<PKGBUILDDIR>>/build/powerpc64le-unknown-linux-gnu/stage1/lib/librustc_driver-8a51db685c3a3652.so(_ZN12rustc_expand6expand13MacroExpander21fully_expand_fragment17hf42bf6c7c5cee112E+0x17f0)[0x7fffa8875590]
/<<PKGBUILDDIR>>/build/powerpc64le-unknown-linux-gnu/stage1/lib/librustc_driver-8a51db685c3a3652.so(_ZN12rustc_expand6expand13MacroExpander12expand_crate17h900deade1746e5d6E+0x6e0)[0x7fffa8873540]
/<<PKGBUILDDIR>>/build/powerpc64le-unknown-linux-gnu/stage1/lib/librustc_driver-8a51db685c3a3652.so(+0x855080)[0x7fffa66c5080]
/<<PKGBUILDDIR>>/build/powerpc64le-unknown-linux-gnu/stage1/lib/librustc_driver-8a51db685c3a3652.so(+0x88ed34)[0x7fffa66fed34]
/<<PKGBUILDDIR>>/build/powerpc64le-unknown-linux-gnu/stage1/lib/librustc_driver-8a51db685c3a3652.so(_ZN15rustc_interface7queries7Queries9expansion17h4bcb4cf9c0b271cfE+0x38c)[0x7fffa672a07c]
/<<PKGBUILDDIR>>/build/powerpc64le-unknown-linux-gnu/stage1/lib/librustc_driver-8a51db685c3a3652.so(+0x6fd664)[0x7fffa656d664]
/<<PKGBUILDDIR>>/build/powerpc64le-unknown-linux-gnu/stage1/lib/librustc_driver-8a51db685c3a3652.so(+0x6e53fc)[0x7fffa65553fc]
/<<PKGBUILDDIR>>/build/powerpc64le-unknown-linux-gnu/stage1/lib/librustc_driver-8a51db685c3a3652.so(+0x6fed88)[0x7fffa656ed88]
/<<PKGBUILDDIR>>/build/powerpc64le-unknown-linux-gnu/stage1/lib/librustc_driver-8a51db685c3a3652.so(+0x6f59a8)[0x7fffa65659a8]
/<<PKGBUILDDIR>>/build/powerpc64le-unknown-linux-gnu/stage1/lib/librustc_driver-8a51db685c3a3652.so(+0x75d010)[0x7fffa65cd010]
/<<PKGBUILDDIR>>/build/powerpc64le-unknown-linux-gnu/stage1/lib/librustc_driver-8a51db685c3a3652.so(+0x6db274)[0x7fffa654b274]
/<<PKGBUILDDIR>>/build/powerpc64le-unknown-linux-gnu/stage1/lib/libstd-5259f91ddc4e5b37.so(rust_metadata_std_f9b0c9e9385aeadc6c2155a37e531624+0x8df54)[0x7fffa5ccdf54]
/lib/powerpc64le-linux-gnu/libpthread.so.0(+0x90a4)[0x7fff9f5e90a4]
/lib/powerpc64le-linux-gnu/libc.so.6(clone+0x74)[0x7fffa5b4d054]
Activity
Mark-Simulacrum commentedon Oct 10, 2021
That stack trace looks like it's coming from procmacro expansion (cstr), though it doesn't look like that dependency changed. It doesn't seem like anything in particular changed with the proc macro code, either...
It would be good to get a better stack trace - at least symbols...
infinity0 commentedon Oct 10, 2021
Here is a stack trace of the failing command. It's a bit spammy; I have the session still open if you want to explore e.g. the local variables or something, let me know what I should run.
Mark-Simulacrum commentedon Oct 10, 2021
Hm, yeah, that definitely seems like something is going wrong with the procedural macro interface, but I'm not sure what would be responsible.
It looks like
rust/library/proc_macro/src/lib.rs
Line 259 in 68dfa07
It seems like the segfault suggests that
envis not a valid pointer here (it does look a bit too small / wrong, but not sure if that's typical on powerpc)...Mark-Simulacrum commentedon Oct 10, 2021
It seems unlikely (AFAICT, the closure code here doesn't use any unstable details) but perhaps #81360 is somehow at fault? (cc @Aaron1011)
Maybe you can check a temporary revert of that...
infinity0 commentedon Oct 10, 2021
I've minimised the segfaulting code to:
Deleting either the function or the
usestatement makes the compilation go successfully; the import can also be replaced by one of the otherrustc_*modules e.g. _codegen_ssa or _middle, but switching it to e.g. std makes the segfault also go away...infinity0 commentedon Oct 10, 2021
Further found out that
rustc_data_structuresas the import also works to preserve the segfault, but using any of its dependencies makes it go away. If this raises any bells let me know - I don't know what to do after that, so I'll wipe this build directory and retry with #81360 reverted.infinity0 commentedon Oct 10, 2021
@Mark-Simulacrum unfortunately the build still fails with the same error, after reverting that PR specifically on top of 1.55.0.
Wondering if @cuviper has any ideas; apparently 1.55 built fine on Fedora https://kojipkgs.fedoraproject.org//packages/rust/1.55.0/1.el7/data/logs/ppc64le/build.log (though there are more test failures than we usually have on ppc64el in Debian).
infinity0 commentedon Oct 10, 2021
I am trying to figure out if there are any significant differences between rust and Debian's LLVM; is it just the [rust] commits in https://github.com/rust-lang/llvm-project/commits/bdb386270f55cb8e95793daa296f27a95a6d4834 ?
infinity0 commentedon Oct 10, 2021
I can work around the problem by patching away the calls to
cstr!inrustc_codegen_llvmto instead directly say what the proc-macro would have generated (or actually, doing it with a regular macro also works). The rest of the rustc build then works. Do you think this is acceptable?Separately, is there a proc-macro heavy crate or system of crates I can test this against, to ensure it doesn't break them?
cuviper commentedon Oct 11, 2021
I noticed that you have a very old
cargo ppc64el 0.47.0-3+b1, wherecargo -Vreports1.46.0. It is generally expected that Rust and Cargo versions should be in sync, and while it might work with some drift, it is not going to be well-supported.To that end, I can reproduce the crash in a
debian:sidcontainer using the system rustc, cargo, and llvm-12. However, if I configure it to use the upstream binary for cargo 1.54.0, still with system rustc and llvm-12, then it builds fine. I don't know what specifically changed in cargo that would affect this, nor why that would only affect powerpc64le.infinity0 commentedon Oct 11, 2021
Thanks for that, then it seems I could probably use the work-around mentioned above, then un-apply it after we do find the time to update cargo in Debian.
FWIW I did check over the release notes of cargo and didn't notice anything that stood out. As I understand, the interface between rustc and cargo is a high-level interface involving basic interprocess UNIX things like envvars and command-line flags. (As opposed to, low-level ABI things like linking and CPU/hardware-specific details.) Even if cargo is out-of-sync with rustc, it would just be supplying unexpected command-line options or envvars or something along these lines, which a user could have done manually. Indeed when reproducing this segfault manually for debugging and minimisation, I am only calling rustc and not dealing with cargo at all. Even the dependencies that it's using, were ultimately directly produced by rustc, and only indirectly via cargo. So along these lines, I would imagine that rustc is not supposed to be segfaulting even if cargo goes wildly out-of-date with it, and there is still a bug somewhere?
cuviper commentedon Oct 11, 2021
It also fails with upstream 0.47.0 / 1.46.0, but it works with cargo 0.48.0 / 1.47.0, and this does have a relevant change:
So if there's a powerpc64le optimization bug arising in proc-macros, as your investigation has pinpointed in
cstr, then newer cargo will avoid it by not optimizing. That's not very satisfying...Yes, I agree, failures due to version mismatch should be much more obvious / less harmful than a segfault.
cuviper commentedon Oct 12, 2021
Another test: using the system cargo and rustc, and the bundled LLVM, it still crashes in the same place.
infinity0 commentedon Oct 12, 2021
Thanks for that, I've also reproduced the same crash with Debian rustc + upstream cargo 1.55.0 by setting
export CARGO_PROFILE_RELEASE_BUILD_OVERRIDE_OPT_LEVEL=3as that PR says. Calling it a night now but the next thing would be to try it with the upstream rustc.infinity0 commentedon Oct 12, 2021
The crash does not reproduce with plain upstream rustc 1.55, using upstream rustc+cargo 1.54 as bootstrap. I'll try to figure out what the difference is with the Debian rustc that might be triggering this.
infinity0 commentedon Oct 13, 2021
OK, I can reproduce the crash with plain upstream (non-Debian) rustc 1.55 using the following minimal config.toml, and setting
CARGO_PROFILE_RELEASE_BUILD_OVERRIDE_OPT_LEVEL=3:The segfault reproduces with either rls, clippy, rustfmt in place of cargo in tools, and goes away (compiles OK) when replaced with analysis or src.
To confirm, the segfault is during building stage1 rustc_codegen_llvm, before the build even gets to building the tools so this is a bit confusing. Would you have any theories @Mark-Simulacrum ?
Mark-Simulacrum commentedon Oct 13, 2021
This definitely seems.. surprising, I wouldn't expect that stage of the build to even be affected by which tools are enabled/disabled. This is deterministic? Is there any difference in logs (e.g., maybe the features set by Cargo are different for some crate?)
I did just notice one interesting thing, that might be worth checking. The compile log you showed above has -Ztls-model=initial-exec being passed, but on our current master, we've disabled that for powerpc targets after #81334 was filed -- #85807 landed in 1.56, so it would not have been included in 1.55. However we added that to the build system in 1.49 (#78201) so I would have expected this problem to arise earlier...
Given that the problem seems to be proc-macro related, it seems at least plausible that the cause is due to lacking support in LLVM that has been undetected until now. Procedural macros definitely make extensive use of TLS to hold state while passing it back and forth, so it wouldn't surprise me too much if the bugs are related.
So I think next steps:
cuviper commentedon Oct 13, 2021
That change for "powerpc-" would only affect 32-bit, not powerpc64le.
Mark-Simulacrum commentedon Oct 13, 2021
Ah, good catch. Still seems worth finding out if we should actually disable on 64-bit as well...
infinity0 commentedon Oct 13, 2021
We've actually been including #85807 in Debian since 1.49 but yeah that's only for 32-bit ppc. I amended it to include 64-bit ppc but the segfault remains. (And in my original testing, the segfault reproduced without any special flags, although that was just repeatedly attempting to compile rustc_codgen_llvm and leaving the already-built dependencies alone.)
In the meantime I've found that the segfault reproduces with CARGO_PROFILE_RELEASE_BUILD_OVERRIDE_OPT_LEVEL=2 but not with =1.
Also the extended / tools thing was a just red herring, it was just because I was running
./x.py buildwithout any explicit targets, it wasn't even building stage 1 rustc in the cases where it "succeeded". If I run./x.py build compiler/rustcthe segfault reproduces even without any config.toml, i.e. from a clean extraction of the release tarball (with OPT=2 or 3).Next, I will try to figure out which particular crate's build.rs switching between OPT 1 vs 2 makes the difference.
cuviper commentedon Oct 13, 2021
Can you see if beta and nightly also fail? Beta has LLVM 13, and nightly has further enabled the new pass manager.
infinity0 commentedon Oct 13, 2021
Beta segfaults, nightly works (with beta as bootstrap). Tested with
CARGO_PROFILE_RELEASE_BUILD_OVERRIDE_OPT_LEVEL=3.It seems cargo doesn't support per-crate overrides, only whole-workspace overrides, so I can't easily track this down. https://doc.rust-lang.org/cargo/reference/profiles.html#overrides Since it's fixed on nightly, perhaps I'll leave the investigation here.