Repository navigation
Error cargo building std as a dylib #36501
Description
Or should cargo building std only produce a .rlib?
Now that #35021 landed and compiler-rt intrinsics are built as part of the std crate dependencies,
one can (cross) compile binaries if the alloc_system (or alloc_jemalloc), panic_unwind and
std crates are listed under the crate Cargo.toml:
$ cargo new --bin hello
$ cd hello
$ edit src/main.rs && cat src/main.rs
#![feature(alloc_system)]
extern crate alloc_system;
fn main() {
println!("Hello, world!");
}
$ edit Cargo.toml && cat Cargo.toml
[package]
name = "hello"
[dependencies]
alloc_syste = { path = "$(rustc --print sysroot)/lib/rustlib/src/rust/src/liballoc_system" }
panic_unwind = { path = "$(rustc --print sysroot)/lib/rustlib/src/rust/src/libpanic_unwind" }
std = { path = "$(rustc --print sysroot)/lib/rustlib/src/rust/src/libstd" }
# if alloc_jemalloc is used
# [features]
# default = ["std/jemalloc"]
$ cargo build --target $CROSS
if and only if the Cargo.toml of the std crate is modified as follows
[lib]
name = "std"
path = "lib.rs"
-crate-type = ["dylib", "rlib"]
[dependencies]
alloc = { path = "../liballoc" }to not build a dylib. Without this change cargo building hello fails with:
$ cargo build --target $CROSS
Running `rustc $sysroot/lib/rustlib/src/rust/src/libstd/lib.rs --crate-name std --crate-type dylib --crate-type rlib -C prefer-dynamic -g -C metadata=7a8ef155232f363e --out-dir $PWD/target/$CROSS/debug/deps --emit=dep-info,link --target $CROSS -L dependency=$PWD/target/$CROSS/debug/deps --extern core=$PWD/target/$CROSS/debug/deps/libcore.rlib --extern compiler_builtins=$PWD/target/$CROSS/debug/deps/libcompiler_builtins.rlib --extern rand=$PWD/target/$CROSS/debug/deps/librand.rlib --extern panic_abort=$PWD/target/$CROSS/debug/deps/libpanic_abort.rlib --extern libc=$PWD/target/$CROSS/debug/deps/liblibc.rlib --extern rustc_unicode=$PWD/target/$CROSS/debug/deps/librustc_unicode.rlib --extern alloc=$PWD/target/$CROSS/debug/deps/liballoc.rlib --extern alloc_system=$PWD/target/$CROSS/debug/deps/liballoc_system.rlib --extern collections=$PWD/target/$CROSS/debug/deps/libcollections.rlib --extern panic_unwind=$PWD/target/$CROSS/debug/deps/libpanic_unwind.rlib --extern unwind=$PWD/target/$CROSS/debug/deps/libunwind.rlib --cfg cargobuild -l dl -l rt -l pthread -L native=$PWD/target/$CROSS/debug/build/compiler_builtins-31a1189c65ac3d55/out`
warning: ar: `u' modifier ignored since `D' is the default (see `U')
Compiling std v0.0.0 (file://$sysroot/lib/rustlib/src/rust/src/libstd)
error[E0463]: can't find crate for `alloc_jemalloc`
error: aborting due to previous error
error: Could not compile `std`.
during the compilation of the std crate. (On that note, I'm sure why building std as a dylib
fails? Is it because rustc injects a extern crate alloc_jemalloc into std source when compiling
a dylib but alloc_jemalloc is an optional dependency of std that's not build (the feature is
disabled) in this example?)
I discussed this with @alexcrichton and we were wondering if we should modify the std crate to only
compile the rlib to make this case work out of the box. And then hack bootstrap to force it to build
std as dylib because it is needed to link rustc.
cc @Ericson2314 I don't how/if this change would affect your Cargo+std RFC. (Probably not?)
Activity
Ericson2314 commentedon Sep 15, 2016
Looks good to me! (I hope once make is gone we can put things in their normal locations and the entire
[lib]section can be elided.)IIUC, we eventually want cargo to be smart enough to build deps (and later still, only public deps) as dylibs. This would be the decision of the bottom crate / builder not any library, so hacking rustbuild for now is a step in the right direction.
All these changes anticipate what I want to do once my RFC is merged, so keep 'em coming!
alexcrichton commentedon Sep 26, 2016
For this specific error, I wonder if perhaps a more targeted fix would suffice? It looks like it's because jemalloc isn't compiled which the the target the standard library is being compiled for assumes will exist.
Basically, on this line, we'd link
alloc_systemfor bothstage0andcfg(not(feature = "jemalloc")).This does have interesting implications, though, for jemalloc. Does mean that building std in a custom fashion is a bit harder.
Ericson2314 commentedon Sep 29, 2016
mmm, how's that better? Custom
stds may indeed want jemalloc, and I don't see whatcrate-typegets us long term.alexcrichton commentedon Sep 29, 2016
The problem is that when the compiler builds a dylib it attempts to link the default dylib allocator. For many targets that's jemalloc, but jemalloc isn't compiled unless you enable a feature.
japaric commentedon Sep 29, 2016
(Alternatively, we could default to system alloc everywhere. Less C is always good in my book)
Ericson2314 commentedon Sep 29, 2016
The "default dylib allocator" stuff is just a stopgap until we get general needs-provides, so I wouldn't sweat it.
alexcrichton commentedon Sep 29, 2016
@japaric eventually hopefully!
std: link to liballoc_system if compiled without the jemalloc feature
aidanhs commentedon Jan 16, 2017
I read this and was confused, so to elaborate for other readers - dylibs are partitioned into 'dylibs used as rust dependencies' and 'other dylibs'. The former (the definition being encountered here) use the same allocator as the current rust compiler will give to executables, typically jemalloc. The latter is basically always the system allocator. See
rust/src/librustc_metadata/creader.rs
Line 822 in ff591b6
-C prefer-dynamicis the way the former is indicated to rustc, and is (I think, more hazy here) detected in cargo by checking whether the crate is both a dylib and is not a member of the current workspace (i.e. is a dep somewhere).aidanhs commentedon Jan 16, 2017
Oh, and (again, just for the record) I think the issue here is relatively easily solved if your
--targetis a .json - I believe you just need to set "exe_allocation_crate" as "alloc_system". Unfortunately I'm not aware of a way to override parts of targets on the fly (I'd be delighted if I'm wrong!), so it's harder otherwise.alexcrichton commentedon Nov 3, 2018
I think this is likely fixed with jemalloc now removed in #55238 so closing