TokenStream::parse does not resolve with Span::call_site()

Updated description

The following procedural macro:

#![feature(proc_macro)]

extern crate proc_macro;

use proc_macro::*;

#[proc_macro]
pub fn foo(input: TokenStream) -> TokenStream {
    let mut body = Vec::<TokenTree>::new();

    // input = `let a = 2;`
    body.extend(input.into_iter());

    // manually add `let b = a;`
    body.push(Term::new("let", Span::call_site()).into());
    body.push(Term::new("b", Span::call_site()).into());
    body.push(Op::new('=', Spacing::Alone).into());
    body.push(Term::new("a", Span::call_site()).into());
    body.push(Op::new(';', Spacing::Alone).into());

    // parse and add `let c = b;`
    body.extend("let c = b;".parse::<TokenStream>().unwrap().into_iter());

    let mut ret = Vec::<TokenTree>::new();
    ret.push(Term::new("fn", Span::call_site()).into());
    ret.push(Term::new("main", Span::call_site()).into());
    ret.push(Group::new(Delimiter::Parenthesis, TokenStream::empty()).into());
    ret.push(Group::new(Delimiter::Brace, body.into_iter().collect()).into());
    ret.into_iter().collect()
}

fails when invoked as:

#![feature(proc_macro)]

extern crate foo;

foo::foo! {
    let a = 3;
}

with the error message:

error[E0425]: cannot find value `b` in this scope
 --> src/main.rs:6:1
  |
6 | / foo::foo! {
7 | |     let a = 3;
8 | | }
  | |_^ not found in this scope

error: aborting due to previous error

For more information about this error, try `rustc --explain E0425`.

Original description

Linking the futures-await crate and a crate that useserror-chain-derive together breaks the build. Here's a simple example: https://github.com/golddranks/test_futures_await_error_chain_derive Either of the two dependencies (futures-await and dotenv, which uses error-chain-derive) builds on its own (using the latest nightly), but together they refuse to build.

This seems to have something to do with the defsite hygiene: error-chain-derive is using derive macros 1.1, and the syntax for using it is as follows: (from https://github.com/purpliminal/rust-dotenv/blob/master/src/errors.rs )

#[derive(Debug, ErrorChain)]
#[cfg_attr(not(feature = "backtrace"), error_chain(backtrace = "false"))]
pub enum ErrorKind {
    // generic error string, required by derive_error_chain
    Msg(String),
    #[error_chain(custom)]
    #[error_chain(description = r#"|_| "Parsing Error""#)]
    #[error_chain(display = r#"|l| write!(f, "Error parsing line: '{}'", l)"#)]
    LineParse(String),
    #[error_chain(foreign)]
    Io(::std::io::Error),
    #[error_chain(foreign)]
    EnvVar(::std::env::VarError),
}

|l| write!(f, "Error parsing line: '{}'", l) breaks, because it can't find f: cannot find value f in this scope

It seems like just the presence of futures-await somehow affects how the Derive macros 1.1 hygiene works; however, the derive macros are stable, so this is an undesirable thing.

Looking at the symptoms, likely to be the same bug as #50061

This still happens on the latest nightly (79252ff4e 2018-04-29), even after the patches #50152 and alexcrichton@e934873 landed.

@petrochenkov

Ping @petrochenkov @alexcrichton

@petrochenkov

I've updated the OP and issue title to reflect the underlying issue, minimized at https://gist.github.com/alexcrichton/6b4c56549c6732e697eb0aa9986bf398.

The tl;dr; is that a TokenStream created through TokenStream::from_str (aka parsing code) cannot refer to tokens with the span Span::call_site() (nor tokens input to the macro itself). This means that for whatever reason the hygienic context applied to tokens created through parsing strings is different than that of Span::call_site, exposing "oddness" of hygiene.

@petrochenkov or @nrc, do y'all know where this might be occurring?

@alexcrichton

@alexcrichton
A guess: this may be a regression from changes in src/libsyntax/ext/quote.rs in #49154 (DUMMY_SP -> ident.span for ast::Ident and ast::Lifetime).
Could you test whether reverting changes in those two lines fixes the regression?
If it doesn't, then I'll try to investigate further sometime during this week.

Sure yeah, I will test out that hypothesis tomorrow morning!

Ah unfortunately this diff on 5a662bf produced the same error:

$ ./build/x86_64-unknown-linux-gnu/stage2/bin/rustc foo.rs --crate-type proc-macro
$ ./build/x86_64-unknown-linux-gnu/stage2/bin/rustc bar.rs -L .
error[E0425]: cannot find value `b` in this scope
 --> bar.rs:5:1
  |
5 | / foo::foo! {
6 | |     let a = 3;
7 | | }
  | |_^ not found in this scope

error: aborting due to previous error

For more information about this error, try `rustc --explain E0425`.

@alexcrichton

@alexcrichton,

The tl;dr; is that a TokenStream created through TokenStream::from_str (aka parsing code) cannot refer to tokens with the span Span::call_site() (nor tokens input to the macro itself). This means that for whatever reason the hygienic context applied to tokens created through parsing strings is different than that of Span::call_site, exposing "oddness" of hygiene.

I am probably missing something, but to me it sounds about right. I would expect manually parsed code to be its own context. That's the hygiene. Similar to literal code in template macros, except proc macros don't have definition site, so the context is simply fresh. To refer to the symbol from the call site, you can replace it with the Token from input or manually created with Span::call_site(), no?

@jan-hudec

@jan-hudec I think you're right yeah, but the problem is that we've been advocating Macros 1.1 and 1.2 as "hygiene comes later, we're just copy/pasting code". That seems to not be the case and there's actual hygiene involved, which is unintended currently

Except if parse::<TokenStream> is now made unhygienic, making it hygienic later won't be backward compatible. So it can't be as easy as just copy/pasting code.

@jan-hudec

@jan-hudec oh it was always the intention that stringifcation loses all hygiene information. The upcoming proc_macro API is how we'll allow adding hygiene in the future (aka via Span)

If I won't be able to implement MarkKind::Transparent (or something equivalent) quickly enough, I'll open a PR with that patch.

@petrochenkov

Ok awesome, thanks @petrochenkov!

Status update: I started working on MarkKind::Transparent and have more detailed understanding of how it should work now, but tomorrow I'm leaving on vacation so finishing this work is going to be postponed for 1-2 weeks.

Some status update: I have an implementation for MarkKind::Transparent and it works by itself and interacts with "opaque"/"modern" macros (declarative macro or procedural Span::def_site()) correctly, but interactions with macro_rules are buggy.
In particular stuff like this:

macro_rules! m {
    () => {
        let_x_transparent!();
        let y = x;
    }
}

m!();

generates an "unresolved x" error while it shouldn't

So far it looks like I need to really dig into the core hygiene/resolution algorithms to fix this properly, perhaps by refactoring hygiene data in some way in which these opaque/transparent/semi-transparent interactions can be expressed more naturally than the current "syntax context transplantation" technique (see the large comment in fn apply_mark).

In the meantime I'll probably just sumbit a PR with the buggy version this weekend.

#51762 is submitted.
(It has the "transparent vs macro_rules" issue mentioned in the previous comment fixed.)

Rollup merge of rust-lang#51762 - petrochenkov:oh-hi-mark, r=oli-obk

Is this fully fixed? We're still seeing issues on rwf2/Rocket#635, and I can also externally confirm that I'm seeing similar issues with other proc-macro/macro interactions.

@SergioBenitez

@SergioBenitez
The issue as described in #50050 (comment) is fully fixed and the the snippet from the issue description now works.
There can be other issues, of course, please file a separate ticket with minimized reproduction if something looks like a bug.

Navigation Menu

Repository navigation

TokenStream::parse does not resolve with Span::call_site() #50050

Updated description

Original description

9 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

TokenStream::parse does not resolve with Span::call_site() #50050

Description

Updated description

Original description

Activity

golddranks commented on Apr 19, 2018

golddranks commented on May 1, 2018

golddranks commented on May 1, 2018

alexcrichton commented on May 1, 2018

petrochenkov commented on May 1, 2018

alexcrichton commented on May 2, 2018

alexcrichton commented on May 2, 2018

jan-hudec commented on May 9, 2018

alexcrichton commented on May 9, 2018

jan-hudec commented on May 9, 2018

alexcrichton commented on May 9, 2018

9 remaining items

petrochenkov commented on May 21, 2018

alexcrichton commented on May 21, 2018

petrochenkov commented on May 30, 2018

petrochenkov commented on Jun 19, 2018

petrochenkov commented on Jun 24, 2018

SergioBenitez commented on Jul 8, 2018

petrochenkov commented on Jul 8, 2018

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions