internal compiler error: cannot lex source_file without source: /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/ppv-lite86-0.2.6/src/x86_64/mod.rs

The setup under which this occurs is somewhat complicated, but I'll try to reduce it as much as I can and update this issue with details. But basically it's using the latest nightly rustc and running cargo check on the latest glean code (https://github.com/mozilla/glean, rev a30e58cebe6888a6c15fcb58d32dc00a9c38bc17).

Code

RUSTFLAGS='-Z save-analysis' cargo +nightly check --all --target-dir=/mnt/index-scratch/glean/objdir

Error output

error: internal compiler error: cannot lex `source_file` without source: /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/ppv-lite86-0.2.6/src/x86_64/mod.rs

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports

note: rustc 1.43.0-nightly (158127853 2020-03-10) running on x86_64-unknown-linux-gnu

note: compiler flags: -Z save-analysis -C debuginfo=2 --crate-type lib

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
end of query stack
error: aborting due to previous error

error: could not compile `c2-chacha`.

(See full output in the backtrace details below)

Backtrace

 Downloading crates ...
  Downloaded bincode v1.2.1
  Downloaded regex v1.3.4
  Downloaded chrono v0.4.10
  Downloaded uuid v0.8.1
  Downloaded serde v1.0.104
  Downloaded once_cell v1.3.1
  Downloaded env_logger v0.7.1
  Downloaded serde_json v1.0.48
  Downloaded rkv v0.10.3
  Downloaded ffi-support v0.4.0
  Downloaded num-integer v0.1.42
  Downloaded regex-syntax v0.6.16
  Downloaded libc v0.2.67
  Downloaded serde_derive v1.0.104
  Downloaded byteorder v1.3.4
  Downloaded arrayref v0.3.6
  Downloaded ryu v1.0.2
  Downloaded termcolor v1.1.0
  Downloaded quick-error v1.2.3
  Downloaded atty v0.2.14
  Downloaded itoa v0.4.5
  Downloaded rand v0.7.3
  Downloaded failure v0.1.6
  Downloaded ordered-float v1.0.2
  Downloaded rand_core v0.5.1
  Downloaded failure_derive v0.1.6
  Downloaded proc-macro2 v1.0.9
  Downloaded bitflags v1.2.1
  Downloaded url v2.1.1
  Downloaded syn v1.0.16
  Downloaded lmdb-rkv v0.12.3
  Downloaded rand_chacha v0.2.1
  Downloaded getrandom v0.1.14
  Downloaded idna v0.2.0
  Downloaded c2-chacha v0.2.3
  Downloaded percent-encoding v2.1.0
  Downloaded lmdb-rkv-sys v0.9.6
  Downloaded unicode-normalization v0.1.12
  Downloaded ppv-lite86 v0.2.6
  Downloaded cc v1.0.50
  Downloaded pkg-config v0.3.17
  Downloaded smallvec v1.2.0
   Compiling libc v0.2.67
   Compiling proc-macro2 v1.0.9
   Compiling unicode-xid v0.2.0
   Compiling syn v1.0.16
    Checking cfg-if v0.1.10
   Compiling autocfg v1.0.0
   Compiling getrandom v0.1.14
   Compiling serde v1.0.104
   Compiling byteorder v1.3.4
   Compiling pkg-config v0.3.17
    Checking ppv-lite86 v0.2.6
   Compiling cc v1.0.50
   Compiling bitflags v1.2.1
   Compiling log v0.4.8
    Checking smallvec v1.2.0
   Compiling failure_derive v0.1.6
    Checking matches v0.1.8
   Compiling ryu v1.0.2
    Checking percent-encoding v2.1.0
    Checking lazy_static v1.4.0
    Checking itoa v0.4.5
    Checking regex-syntax v0.6.16
    Checking arrayref v0.3.6
    Checking once_cell v1.3.1
    Checking quick-error v1.2.3
    Checking termcolor v1.1.0
    Checking unicode-normalization v0.1.12
    Checking unicode-bidi v0.3.4
   Compiling num-traits v0.2.11
   Compiling num-integer v0.1.42
    Checking humantime v1.3.0
    Checking c2-chacha v0.2.3
error: internal compiler error: cannot lex `source_file` without source: /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/ppv-lite86-0.2.6/src/x86_64/mod.rs

thread 'rustc' panicked at 'Box<Any>', src/librustc_errors/lib.rs:880:9
stack backtrace:
    Checking ffi-support v0.4.0
   Compiling quote v1.0.2
    Checking idna v0.2.0
   0:     0x7fd8b7d5bea4 - backtrace::backtrace::libunwind::trace::h79b6ad55bf71ea4d
                               at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.44/src/backtrace/libunwind.rs:86
   1:     0x7fd8b7d5bea4 - backtrace::backtrace::trace_unsynchronized::hcfa816a17c383c01
                               at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.44/src/backtrace/mod.rs:66
   2:     0x7fd8b7d5bea4 - std::sys_common::backtrace::_print_fmt::h1fbf4e8dad31c04d
                               at src/libstd/sys_common/backtrace.rs:78
   3:     0x7fd8b7d5bea4 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h691b7317de5d1611
                               at src/libstd/sys_common/backtrace.rs:59
   4:     0x7fd8b7d9489c - core::fmt::write::h0edcddf75fa815b5
                               at src/libcore/fmt/mod.rs:1063
   5:     0x7fd8b7d4dbc3 - std::io::Write::write_fmt::hbcb4a18a26d43db2
                               at src/libstd/io/mod.rs:1426
   6:     0x7fd8b7d60cd5 - std::sys_common::backtrace::_print::h6ea84893fd9c4f5c
                               at src/libstd/sys_common/backtrace.rs:62
   7:     0x7fd8b7d60cd5 - std::sys_common::backtrace::print::hb27542ff92a9665f
                               at src/libstd/sys_common/backtrace.rs:49
   8:     0x7fd8b7d60cd5 - std::panicking::default_hook::{{closure}}::hdda292fa66c32722
                               at src/libstd/panicking.rs:204
   9:     0x7fd8b7d60a22 - std::panicking::default_hook::hb282c50c9862e02d
                               at src/libstd/panicking.rs:224
  10:     0x7fd8b83b90c9 - rustc_driver::report_ice::h556269f1013a04f1
  11:     0x7fd8b7d61405 - std::panicking::rust_panic_with_hook::hbe9819d9d894172d
                               at src/libstd/panicking.rs:474
  12:     0x7fd8bab017b3 - std::panicking::begin_panic::h39b92cee9cd52b71
  13:     0x7fd8baaca750 - rustc_errors::HandlerInner::bug::ha8fcca766396bc52
  14:     0x7fd8baac94c0 - rustc_errors::Handler::bug::h524f2cc66bc71cfd
  15:     0x7fd8b9dc7989 - rustc_parse::lexer::StringReader::new::h83554f3f7ce8325c
  16:     0x7fd8b9dc7b97 - rustc_parse::lexer::StringReader::retokenize::haf8932c92ef7f0cb
  17:     0x7fd8b84f789c - rustc_save_analysis::span_utils::SpanUtils::sub_span_of_token::h3680c12557eb839e
  18:     0x7fd8b84b5544 - rustc_save_analysis::dump_visitor::DumpVisitor::process_use_tree::h17df36bb5159a5b2
  19:     0x7fd8b84b731d - <rustc_save_analysis::dump_visitor::DumpVisitor as rustc_ast::visit::Visitor>::visit_item::h415b0b015651a8cb
  20:     0x7fd8b84bbf0f - <rustc_save_analysis::dump_visitor::DumpVisitor as rustc_ast::visit::Visitor>::visit_item::h415b0b015651a8cb
  21:     0x7fd8b84b900a - <rustc_save_analysis::dump_visitor::DumpVisitor as rustc_ast::visit::Visitor>::visit_item::h415b0b015651a8cb
  22:     0x7fd8b84b6a2a - <rustc_save_analysis::dump_visitor::DumpVisitor as rustc_ast::visit::Visitor>::visit_mod::hb3951d0b06d3a669
  23:     0x7fd8b8440ff3 - rustc_ast::visit::walk_crate::h7559d33fc73f1f47
  24:     0x7fd8b842d05e - rustc::dep_graph::graph::DepGraph::with_ignore::h32d5b32739cb0104
  25:     0x7fd8b840102e - rustc_session::utils::<impl rustc_session::session::Session>::time::hdf63c34b18b22e5e
  26:     0x7fd8b843df0b - rustc::ty::context::tls::enter_global::h992246c787b278e1
  27:     0x7fd8b840e263 - rustc_interface::interface::run_compiler_in_existing_thread_pool::h84d51f9f8b68c8ca
  28:     0x7fd8b83c25bd - scoped_tls::ScopedKey<T>::set::h4cb060ee70710273
  29:     0x7fd8b83bf514 - rustc_ast::attr::with_globals::h10cf6ded6d7b02ee
  30:     0x7fd8b83c3c00 - std::sys_common::backtrace::__rust_begin_short_backtrace::h72f84d91a30b8442
  31:     0x7fd8b7d726d7 - __rust_maybe_catch_panic
                               at src/libpanic_unwind/lib.rs:86
  32:     0x7fd8b84110d6 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h17127d11297e0161
  33:     0x7fd8b7d3d66f - <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::hce6b09a53f302609
                               at /rustc/15812785344d913d779d9738fe3cca8de56f71d5/src/liballoc/boxed.rs:1017
  34:     0x7fd8b7d70fdd - <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::h7d11f09ec1b28d83
                               at /rustc/15812785344d913d779d9738fe3cca8de56f71d5/src/liballoc/boxed.rs:1017
  35:     0x7fd8b7d70fdd - std::sys_common::thread::start_thread::h5a725994801096bc
                               at src/libstd/sys_common/thread.rs:13
  36:     0x7fd8b7d70fdd - std::sys::unix::thread::Thread::new::thread_start::h84c763d83a5c132e
                               at src/libstd/sys/unix/thread.rs:80
  37:     0x7fd8b7ad76db - start_thread
  38:     0x7fd8b73f488f - __clone
  39:                0x0 - <unknown>

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports

note: rustc 1.43.0-nightly (158127853 2020-03-10) running on x86_64-unknown-linux-gnu

note: compiler flags: -Z save-analysis -C debuginfo=2 --crate-type lib

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
end of query stack
error: aborting due to previous error

error: could not compile `c2-chacha`.

To learn more, run the command again with --verbose.
warning: build failed, waiting for other jobs to finish...
error: build failed

Also to note: so far this has happened 2/2 times today. It didn't happen yesterday (which presumably was running using a slightly-older rustc and checking slightly-older glean code).

Re-running the command:

RUSTFLAGS="-Z save-analysis" cargo +nightly check --all --target-dir=/mnt/index-scratch/glean/objdir

in the proper directory on the machine in question reproduces the error. The file it's complaining about (/home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/ppv-lite86-0.2.6/src/x86_64/mod.rs) exists and doesn't appear corrupted.

Running the command over and over changes which crate it errors out on. In the initial run it was erroring out on c2-chacha but on subsequent runs it changes to synstructure, serde_derive, etc.

@Xanewok

If I remove the RUSTFLAGS="-Z save-analysis" bit then it works fine. So seems related to the save-analysis code.

/cc @Xanewok

I'm able to reproduce this in a local environment as well. Minimal steps so far (on a Ubuntu 18.04 VM)

rustup update nightly-2020-03-11 # actually installs a rustc where --version reports 2020-03-10
rustup default nightly-2020-03-11
git clone https://github.com/mozilla/glean
cd glean
git reset --hard a30e58cebe6888a6c15fcb58d32dc00a9c38bc17
RUSTFLAGS="-Z save-analysis" cargo check

I ran cargo with --verbose and the specific rustc invocation that's reliably reproducing the failure is:

rustc --crate-name c2_chacha --edition=2018 /home/vagrant/.cargo/registry/src/github.com-1ecc6299db9ec823/c2-chacha-0.2.3/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata -C debuginfo=2 --cfg 'feature="simd"' --cfg 'feature="std"' -C metadata=8cbde2cd3443e18e -C extra-filename=-8cbde2cd3443e18e --out-dir /home/vagrant/mozilla-index/glean/glean/target/debug/deps -L dependency=/home/vagrant/mozilla-index/glean/glean/target/debug/deps --extern ppv_lite86=/home/vagrant/mozilla-index/glean/glean/target/debug/deps/libppv_lite86-ff6fe2823b6b9168.rmeta --cap-lints allow -Z save-analysis

(Obviously your paths may vary)

Also note that this does NOT happen with rustup default nightly-2020-03-10 (which installs a rustc built on 2020-03-09) so it's a regression between rustc version 3dbade652 and rustc version 158127853.

Rust regression range: 3dbade6...1581278

If you're unable to repro or need more detail feel free to let me know. Otherwise for now I'm leaving this with you (i.e. rust developers) to dig into.

I have the very same crush with rustc 1.43.0-nightly (c20d7ee 2020-03-11) on both of Mac and WSL2/Ubuntu, every time Rust language Server starts up in VSCode.

$rustc --version --verbose
rustc 1.43.0-nightly (c20d7eecb 2020-03-11)
binary: rustc
commit-hash: c20d7eecbc0928b57da8fe30b2ef8528e2bdd5be
commit-date: 2020-03-11
host: x86_64-unknown-linux-gnu
release: 1.43.0-nightly
LLVM version: 9.0

No problem in rustc 1.43.0-nightly (beac68a 2020-03-01).

$rls --version
rls 1.41.0 (5fde462 2020-02-21)

Error message:

{"message":"cannot lex `source_file` without source: /home/sisshiki/.cargo/registry/src/github.com-1ecc6299db9ec823/ppv-lite86-0.2.6/src/x86_64/mod.rs","code":null,"level":"error: internal compiler error","spans":[],"children":[],"rendered":"\u001b[0m\u001b[1m\u001b[38;5;9merror: internal compiler error\u001b[0m\u001b[0m\u001b[1m: cannot lex `source_file` without source: /home/sisshiki/.cargo/registry/src/github.com-1ecc6299db9ec823/ppv-lite86-0.2.6/src/x86_64/mod.rs\u001b[0m\n\n"}
thread 'rustc' panicked at 'Box<Any>', src/librustc_errors/lib.rs:880:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports

note: rustc 1.43.0-nightly (c20d7eecb 2020-03-11) running on x86_64-unknown-linux-gnu

note: compiler flags: -C debuginfo=2 --crate-type lib

note: some of the compiler flags provided by cargo are hidden

{"message":"aborting due to previous error","code":null,"level":"error","spans":[],"children":[],"rendered":"\u001b[0m\u001b[1m\u001b[38;5;9merror\u001b[0m\u001b[0m\u001b[1m: aborting due to previous error\u001b[0m\n\n"}

I used cargo bisect-rustc to narrow the range to dd155df...1581278

@Centril

Therefore regression from #66364, cc @Centril

Actually, this doesn't have to fail at all either, it's just hardcoded to non-external files:

rust/src/librustc_parse/lexer/mod.rs

Lines 49 to 54 in e0f5df0

    
           if source_file.src.is_none() { 
        
               sess.span_diagnostic 
        
                   .bug(&format!("cannot lex `source_file` without source: {}", source_file.name)); 
        
           } 
        
           let src = (*source_file.src.as_ref().unwrap()).clone();

Before checking for the source, it would need to do this:

sess.source_map.ensure_source_file_source_present(source_file.clone());

And then something like this to handle both local and external sources:

rust/src/librustc_span/source_map.rs

Lines 586 to 590 in e0f5df0

    
           if let Some(ref src) = local_begin.sf.src { 
        
               return extract_source(src, start_index, end_index); 
        
           } else if let Some(src) = local_begin.sf.external_src.borrow().get_source() { 
        
               return extract_source(src, start_index, end_index); 
        
           } else {

Speaking of which, just noticed the first snippet clones a String to parse a file, could we speed up parsing by removing that clone (e.g. by replacing String with Rc<str>)?
Or would it not make a big difference due to how slow parsing is by comparison?
EDIT: nevermind about that last part, it's Lrc<String> already.

So what are next steps here? I don't know what rustc's policy is with respect to regressions, is the regressing patch author responsible for fixing the regression?

Downgrading to nightly-2020-03-09 fixes the issues in the vscode plugin for now.

Sorry, I'll go implement #69933 (comment) now.
EDIT: up at #70172.

I'm not sure why the original reproduction seemed so daunting, it triggered the bug as expected.

I was also able to reproduce it on the problem crate itself:

git clone https://github.com/cryptocorrosion/cryptocorrosion
cd cryptocorrosion
RUSTFLAGS="-Z save-analysis" cargo +nightly check

and it succeeds with this (which is my local build including #70172):

RUSTFLAGS="-Z save-analysis" cargo +rust-2-stage1 check

glean also gets past the ICEs, but I can't easily do a full build because build scripts failed.

Rollup merge of rust-lang#70172 - eddyb:retokenize-external-src, r=pe…

Thank you for the fix! At the moment using rustup to pull a nightly still gives me an older nightly (due to newer ones missing various components) but once I can get a nightly with the fix I'll report back whether or not it solved my original issue.

Confirmed the fix by forcing a nightly update. Thanks!

I am running into this / similar issue when async-attributes is used as dependency:

RUSTFLAGS="-Z save-analysis" cargo check
   Compiling proc-macro2 v1.0.10
   Compiling unicode-xid v0.2.0
   Compiling syn v1.0.17
   Compiling quote v1.0.3
   Compiling async-attributes v1.1.1
error: internal compiler error: cannot lex `source_file` without source: /Users/gozala/.cargo/registry/src/github.com-1ecc6299db9ec823/quote-1.0.3/src/lib.rs

thread 'rustc' panicked at 'Box<Any>', src/librustc_errors/lib.rs:880:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports

note: rustc 1.44.0-nightly (f509b26a7 2020-03-18) running on x86_64-apple-darwin

note: compiler flags: -Z save-analysis -C prefer-dynamic -C debuginfo=2 --crate-type proc-macro

note: some of the compiler flags provided by cargo are hidden

error: aborting due to previous error

error: could not compile `async-attributes`.

To learn more, run the command again with --verbose.

versions

Default host: x86_64-apple-darwin
rustup home:  /Users/gozala/.rustup

installed toolchains
--------------------

stable-x86_64-apple-darwin
nightly-2020-03-09-x86_64-apple-darwin
nightly-x86_64-apple-darwin (default)

active toolchain
----------------

nightly-x86_64-apple-darwin (default)
rustc 1.44.0-nightly (f509b26a7 2020-03-18)

Here is the simple reproducible test case https://github.com/Gozala/rls-vscode-bug-report

Never mind I see now that version I have is 2020-03-18. If I I do force update it works.

Update rust toolchain to 2020-04-20

Navigation Menu

Repository navigation

internal compiler error: cannot lex `source_file` without source: /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/ppv-lite86-0.2.6/src/x86_64/mod.rs #69933

Code

Meta

Error output

Error message:

3 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	if source_file.src.is_none() {
	sess.span_diagnostic
	.bug(&format!("cannot lex `source_file` without source: {}", source_file.name));
	}

	let src = (*source_file.src.as_ref().unwrap()).clone();

	if let Some(ref src) = local_begin.sf.src {
	return extract_source(src, start_index, end_index);
	} else if let Some(src) = local_begin.sf.external_src.borrow().get_source() {
	return extract_source(src, start_index, end_index);
	} else {

internal compiler error: cannot lex source_file without source: /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/ppv-lite86-0.2.6/src/x86_64/mod.rs #69933

Description

Code

Meta

Error output

Activity

staktrace commented on Mar 11, 2020

staktrace commented on Mar 11, 2020

staktrace commented on Mar 11, 2020

staktrace commented on Mar 11, 2020

staktrace commented on Mar 12, 2020

staktrace commented on Mar 12, 2020

staktrace commented on Mar 12, 2020

sisshiki1969 commented on Mar 12, 2020

Error message:

staktrace commented on Mar 15, 2020

staktrace commented on Mar 15, 2020

3 remaining items

eddyb commented on Mar 15, 2020

staktrace commented on Mar 20, 2020

thecodewarrior commented on Mar 20, 2020

eddyb commented on Mar 20, 2020

eddyb commented on Mar 21, 2020

staktrace commented on Mar 23, 2020

staktrace commented on Mar 24, 2020

Gozala commented on Apr 2, 2020

Gozala commented on Apr 2, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

internal compiler error: cannot lex `source_file` without source: /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/ppv-lite86-0.2.6/src/x86_64/mod.rs #69933