Skip to content

fix: Fix matching of doc comments in MBEs - #23314

Draft
ChayimFriedman2 wants to merge 2 commits into
rust-lang:masterfrom
ChayimFriedman2:skip-macro-doc-comments-v2
Draft

fix: Fix matching of doc comments in MBEs#23314
ChayimFriedman2 wants to merge 2 commits into
rust-lang:masterfrom
ChayimFriedman2:skip-macro-doc-comments-v2

Conversation

@ChayimFriedman2

@ChayimFriedman2 ChayimFriedman2 commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

They should be ignored, but only if they stay in the original representation as tt::DocComment. Which first means we need to author tt::DocComment, then handle the cases where it's not preserved: in MBE input, and when proc macros see it (but lazily; not when passed to proc macros, only when they inspect it - i.e. only when it's converted to the proc-macro-bridge's lossy representation.

Invisible delimiters of MBE, one of the most frequent origins for bugs we have, also work in the same way (both originate from how rustc does not separate between its AST and macro input), so it's a precedent more important than just this edge case.

I don't think we have a way to test the interactions with proc macros (this requires a real proc macro server with the bridge), but I tested all gory details manually with a built proc macro server.

This requires a tiny adaptation from the RustRover folks (we should probably notify them): the serialization of FlatTree has changed slightly (even in the JSON format) for version >= DOC_COMMENT_LEAF. They can send (and receive) an empty doc_comments list if they do not send non-desugared doc comment token trees (which they do not, currently).

Specifically, in those versions the schema went from:

interface FlatTree {
    subtree: u32[],
    literal: u32[],
    punct: u32[],
    ident: u32[],
    token_tree: u32[],
    text: string[],
}

Into:

interface FlatTree {
    pre_doc_comment_leaf: {
        subtree: u32[],
        literal: u32[],
        punct: u32[],
        ident: u32[],
        token_tree: u32[],
        text: string[],
    },
    doc_comments: u32[], // Can be empty for RustRover.
}

Fixes #23088.

This sits on the top of #23152 which is required for this.

@rustbot rustbot added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Sep 7, 2026
@ChayimFriedman2
ChayimFriedman2 marked this pull request as draft September 7, 2026 21:57
@rustbot rustbot removed the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Sep 7, 2026
@ChayimFriedman2
ChayimFriedman2 force-pushed the skip-macro-doc-comments-v2 branch from aba3f22 to 1fac479 Compare September 7, 2026 21:59
…he tt crate

And not in the representation of the proc macro bridge. We still do not use the contiguous representation the rest of r-a uses, and prefer a tree with `Rc`s (this changed from `Arc` because we don't need thread safety for the bridge) for cheap cloning, but the types are the same.

The reason for that is that conversion to the bridge's types is a lossy conversion, in rustc as well: the internal token tree representation contains details that are lost in this conversion. This is a prerequisite to fix (don't close GH) rust-lang#23088 - the only way to fix it properly is to have an additional kind of token tree for doc comments, like rustc does, as evidenced by the fact that if a proc macro would create this macro (that has a doc comment in its matcher), the doc comment would *not* be ignored for matching, but if the macro's matcher would have been passed to the proc macro and it wouldn't have touch it (meaning, not even listing the `TokenTree`s then giving them back), the comment would still be ignored.

It is also required for supporting invisible groups properly for the same reason (there're actually several kinds of invisible delimiters, and the proc macro bridge lossily converts them all into one).

While we're at, I've also tried to untangle the mess of Cargo features of the proc macro server, and reduce the code it contains:

There are now *two* somewhat-orthogonal axes: the `in-rust-tree` feature, and the `in-ra` vs. `in-proc-macro-srv` feature. The former only decides whether we should import rustc crates from crates.io or from the sysroot. The latter decides what code to enable - the proc macro server doesn't need all the code in tt and proc-macro-api. I also moved the proc macro server's `TokenStream` from proc-macro-srv into proc-macro-api and inverted their dependency, as was required for this work.

You cannot enable `in-proc-macro-srv` and disable `in-rust-tree` (in practice this just disables all proc macro server code since such combination can arise when compiling r-a, also proc-macro-api supports this mode for tests), and you are not supposed to enable both `in-ra` and `in-proc-macro-srv`, except for tests. Enabling `in-rust-tree` and disabling `in-proc-macro-srv` is possible, though, and done when compiling r-a in-tree. In the future, this mode may use sysroot crates not just for the proc macro server, like was done in the past.
They should be ignored, but only if they stay in the original representation as `tt::DocComment`. Which first means we need to *author* `tt::DocComment`, then handle the cases where it's *not* preserved: in MBE input, and when proc macros see it (but lazily; not when passed to proc macros, only when they inspect it - i.e. only when it's converted to the proc-macro-bridge's lossy representation.

Invisible delimiters of MBE, one of the most frequent origins for bugs we have, also work in the same way (both originate from how rustc does not separate between its AST and macro input), so it's a precedent more important than just this edge case.

I don't think we have a way to test the interactions with proc macros (this requires a real proc macro server with the bridge), but I tested all gory details manually with a built proc macro server.

This requires a tiny adaptation from the RustRover folks (we should probably notify them): the serialization of `FlatTree` has changed slightly (even in the JSON format) for `version >= DOC_COMMENT_LEAF`. They can send (and receive) an empty `doc_comments` list if they do not send non-desugared doc comment token trees (which they do not, currently).
@ChayimFriedman2
ChayimFriedman2 force-pushed the skip-macro-doc-comments-v2 branch from 1fac479 to f1ce6f2 Compare September 8, 2026 03:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

FP unexpected token in input macro with doc comment

2 participants