Repository navigation
x build fails copying files into stage0-sysroot using local rust #143735
Description
Summary
When building rust with a local toolchain, boostrap fails with errors like:
thread 'main' panicked at src/bootstrap/src/lib.rs:1729:17:
failed to copy `/usr/lib/cups/backend/gutenprint53+usb` to `/home/paumurph/git/rust-dev/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/cups/backend/gutenprint53+usb`: Permission denied (os error 13)
Command used
./configure --local-rust-root=/usr --llvm-root=/usr
RUST_BACKTRACE=1 ./x --stage=1 build sysrootExpected behaviour
The x command succeeds
Actual behaviour
bootstrap attempts to copy the contents /usr/lib and fails when it encounters files with unexpected permissions.
Bootstrap configuration (bootstrap.toml)
See configure command above.
Operating system
Fedora 41
HEAD
On the beta branch, based off b5e10d8c000a153cae6a14d70372dc4b39439a3f
Additional context
Build Log
0: __rustc::rust_begin_unwind
at /builddir/build/BUILD/rust-1.88.0-build/rustc-1.88.0-src/library/std/src/panicking.rs:697:5
1: core::panicking::panic_fmt
at /builddir/build/BUILD/rust-1.88.0-build/rustc-1.88.0-src/library/core/src/panicking.rs:75:14
2: bootstrap::Build::copy_link_internal
at ./src/bootstrap/src/lib.rs:1729:17
3: bootstrap::Build::copy_link
at ./src/bootstrap/src/lib.rs:1682:9
4: bootstrap::Build::cp_link_r
at ./src/bootstrap/src/lib.rs:1757:17
5: bootstrap::Build::cp_link_r
at ./src/bootstrap/src/lib.rs:1755:17
6: bootstrap::Build::cp_link_r
at ./src/bootstrap/src/lib.rs:1755:17
7: <bootstrap::core::build_steps::compile::StdLink as bootstrap::core::builder::Step>::run
at ./src/bootstrap/src/core/build_steps/compile.rs:814:13
8: bootstrap::core::builder::Builder::ensure
at ./src/bootstrap/src/core/builder/mod.rs:1596:23
9: <bootstrap::core::build_steps::compile::Std as bootstrap::core::builder::Step>::run
at ./src/bootstrap/src/core/build_steps/compile.rs:156:13
10: bootstrap::core::builder::Builder::ensure
at ./src/bootstrap/src/core/builder/mod.rs:1596:23
11: <bootstrap::core::build_steps::compile::Rustc as bootstrap::core::builder::Step>::run
at ./src/bootstrap/src/core/build_steps/compile.rs:1074:9
12: bootstrap::core::builder::Builder::ensure
at ./src/bootstrap/src/core/builder/mod.rs:1596:23
13: <bootstrap::core::build_steps::compile::Assemble as bootstrap::core::builder::Step>::run
at ./src/bootstrap/src/core/build_steps/compile.rs:2155:28
14: bootstrap::core::builder::Builder::ensure
at ./src/bootstrap/src/core/builder/mod.rs:1596:23
15: bootstrap::core::builder::Builder::compiler
at ./src/bootstrap/src/core/builder/mod.rs:1298:9
16: <bootstrap::core::build_steps::compile::Std as bootstrap::core::builder::Step>::make_run
at ./src/bootstrap/src/core/build_steps/compile.rs:123:23
17: bootstrap::core::builder::StepDescription::maybe_run
at ./src/bootstrap/src/core/builder/mod.rs:475:13
18: bootstrap::core::builder::StepDescription::run
at ./src/bootstrap/src/core/builder/mod.rs:607:17
19: bootstrap::core::builder::Builder::run_step_descriptions
at ./src/bootstrap/src/core/builder/mod.rs:1271:9
20: bootstrap::core::builder::Builder::execute_cli
at ./src/bootstrap/src/core/builder/mod.rs:1251:9
21: bootstrap::Build::build
at ./src/bootstrap/src/lib.rs:705:17
22: bootstrap::main
at ./src/bootstrap/src/bin/main.rs:99:5
23: core::ops::function::FnOnce::call_once
at /builddir/build/BUILD/rust-1.88.0-build/rustc-1.88.0-src/library/core/src/ops/function.rs:250:5
My first attempt at working around was the oneliner, though I am not sure this is correct:
--- a/src/bootstrap/src/core/build_steps/compile.rs
+++ b/src/bootstrap/src/core/build_steps/compile.rs
@@ -811,7 +811,7 @@ fn run(self, builder: &Builder<'_>) {
let _ = fs::remove_dir_all(sysroot.join("lib/rustlib/src/rust"));
}
- builder.cp_link_r(&builder.initial_sysroot.join("lib"), &sysroot.join("lib"));
+ builder.cp_link_r(&builder.initial_sysroot.join("lib/rustlib"), &sysroot.join("lib/rustlib"));
} else {
if builder.download_rustc() {
// Ensure there are no CI-rustc std artifacts.
Activity
Kobzol commentedon Jul 10, 2025
Hi, what does
/usr/bin/rustc --print sysrootoutput?pmur commentedon Jul 10, 2025
cuviper commentedon Jul 10, 2025
ISTR bootstrap used to forgo all copying for externally-provided stage0 like this.
Kobzol commentedon Jul 10, 2025
We copy the stage0 sysroot to the
stage0-sysrootdirectory, because during the build, we actually need to copy stuff (rlibs/rmetas) into that directory. Normally, the sysroot is a self-contained directory with Rust-specific files, but in your case it's essentially just/usr. We don't really know how to identify all the relevant files in the sysroot, so we just copy everything, which is a problem for your use-case. If we didn't do this copying, then we would either start putting things into/usr(which is even more problematic), or it won't be possible to do a check/build of the stage1 compiler.We could maybe hotfix this by ignoring files with bad permissions, but I'm not sure if I like that hack. I'm inclined to say that this is a wontfix, and that if you want to use a custom external rustc, it should have a self-contained sysroot, same as what you get with rustup.
cuviper commentedon Jul 10, 2025
I assume that's copying newly built stuff, but what exactly? Why is this different than ever before?
That's more difficult to arrange in a distro context, so I'd like to figure out how we can continue supporting what used to work fine. (And of course, we can contribute patches to that end, once I understand what's going on...)
Kobzol commentedon Jul 10, 2025
When you check/build a stage 1 rustc using stage 0 rustc, the rmetas/rlibs of the stage 1 rustc get copied into the stage 0 sysroot. This is required in order to check/build stage 1 rustc tools (miri, rust-analyzer, rustfmt, clippy, etc.). Currently it happens unconditionally, so even if you don't build these tools, these artifacts will be copied into the stage 0 sysroot.
I can't say that I love this behavior, but apparently rustc can't/doesn't support multiple sysroots, so it currently kind of has to work like this, at least from the rustc side.
There are probably some ways to improve this situation though:
stage0-sysrootdirectory. Maybe we can just hardcode something like "copy lib/rustlib, bin/<looks like librustc_driver/looks like LLVM>". Sounds quite hacky...It should have behaved like this since forever, I think? It's possible that bootstrap had some bugs that hid this. Also the stage 0 redesign might have changed behavior here, the build/check staging was quite wild, I'm now slowly trying to consolidate it.
cuviper commentedon Jul 10, 2025
I believe it's new from the redesign, added here:
5dc0d89#diff-5a1e05f2688d271039171a547d407d0c8a96715ee64d35562fc76b4c9a874303R797
Before, it only copied stuff if "not set to a custom path".
We don't need miri since that's a nightly-only thing, but we do ship most other tools including clippy. Even then, I think those should be stage2 builds for dist, so they probably still don't need compiler libs in
stage0-sysroot, right?I suspect
lib/rustlibmight suffice -- it's only copyinglibright now, not anybinat all. For Linux, that would includelib/{librustc_driver,libLLVM}too, but we have those elsewhere in/usr/lib64anyway. I imagine most distros are similar, or else have rpaths to find them relative to the uncopiedbinanyway.Kobzol commentedon Jul 10, 2025
If you do a stage 2 rustfmt build, only the stage1 rlibs/rmetas have to be available, so we don't need the compiler libs in stage0-sysroot, indeed. I guess that this could be the simplest fix, that if you do a stage N tool build, then only copy the rlibs to the N-1 rustc sysroot, and don't do this for N-2. (I am excited to see what other use-cases this will break =D)
cuviper commentedon Jul 10, 2025
On second thought, why do the compiler libs need to be in the sysroot? Couldn't we provide them with
-L?(That is a way to get "multiple sysroots" in some sense.)
Kobzol commentedon Jul 10, 2025
Let's ask someone who knows more about this :D (https://rust-lang.zulipchat.com/#narrow/stream/326414-t-infra.2Fbootstrap/topic/Proposal.20to.20cleanup.20stages.20and.20steps.20after.20the.20redesign/near/527359940)
heftig commentedon Aug 8, 2025
I had to solve another problem while working around this:
The compiler prefers
lib64overlib, so if a symlink/usr/lib64 -> libexists,lib64also shows up inrustc --print target-libdir.Bootstrap will create an empty
lib64/rustlib/<target>/lib/in the sysroot but copy the crates tolib/rustlib/<target>/lib/.The compiler then finds the former dir and doesn't look for the latter, so it can't find the crates.
#t-infra/bootstrap > ✔ 1.89.0 bootstrap attempts to copy entire /usr/lib
ARCHLINUX: bootstrap: Workaround for system stage0
ARCHLINUX: bootstrap: Workaround for system stage0
SPECS: rust: Upgrade to 1.94.1.
SPECS: rust: Upgrade to 1.94.1.
SPECS: rust: Upgrade to 1.94.1.