Why is "-x" trying to parse as a literal and failing in declarative macros?

Question

I have a little macro I wrote to help me explore what things are parsed as. With a few arms lopped off, it looks like:

macro_rules! show_me{
    ($l : literal $($tail : tt)*) => {
        println!("{} is a literal and then I see {}", stringify!($l), stringify!($($tail)*));
        show_me!($($tail)*)
    };
    (- $($tail : tt)*) => {
        println!("{} is a - and then I see {}", stringify!(-), stringify!($($tail)*));
        show_me!($($tail)*)
    };
    ($i : ident $($tail : tt)*) => {
        println!("{} is an ident and then I see {}", stringify!($i), stringify!($($tail)*));
        show_me!($($tail)*)
    };
    ($token : tt $($tail : tt)*) => {
        println!("{} is a mystery and then I see {}", stringify!($token), stringify!($($tail)*));
        show_me!($($tail)*)
    };
    () => {}
}

This behaves as expected on some inputs using rustc 1.66.0 (69f9c33d7 2022-12-12):

show_me!(x + 1);
# x is an ident and then I see + 1
# + is a mystery and then I see 1
# 1 is a literal and then I see

But if I try a negative number, I get (a little) surprised:

show_me!(-4);
# -4 is a literal and then I see

This would make perfect sense, except that https://doc.rust-lang.org/reference/procedural-macros.html says (under the description of declarative macros):

Literals ("string", 1, etc)

Note that negation (e.g. -1) is never a part of such literal tokens, but a separate operator token.

Where it gets weird to me is if I try to run show_me!(-x):

error: unexpected token: `x`
  --> src/main.rs:54:15
   |
54 |     show_me!(-x);
   |               ^
   |
  ::: src/utils/macros.rs:27:6
   |
27 |     ($l : literal $($tail : tt)*) => {
   |      ------------ while parsing argument for this `literal` macro fragment

Note that if I comment out the literal arm of show_me, or just move it to the bottom of the matcher, the other arms are perfectly happy with this:

show_me!(-x);
# - is a - and then I see x
# x is an ident and then I see

My expectation would be that the matching arm would either match successfully, or fall through to the next pattern. Instead, it's throwing compile errors. What is happening in this case?

Also, am I misreading the documentation or is it incorrect? To me it seems to clearly say that - should never be part of literal expressions in declarative macros (though it may be part of procedural ones.)

@E_net4thecommentflagger I was on `1.59`, but on reading your comment I updated - now both `rustc --version` and `cargo --version` show `1.66`, but the behavior is the same. `rustc 1.66.0 (69f9c33d7 2022-12-12)`, `rustc 1.66.0 (69f9c33d7 2022-12-12)` — Edward Peters, Jan 05 '23 at 14:59
The link you point to in the docs talks about matching _token trees_ (i.e. `tt`), which when given a negative number will only take the `-` instead of taking the whole number. However matching a literal will happily take a whole negative number: [playground](https://play.rust-lang.org/?version=stable&mode=release&edition=2021&gist=1613dbafc76c20d436ecc1ad0e2f494d) — Jmb, Jan 05 '23 at 15:44
@Jmb That explains my issue with the documentation, but I'd still like to understand why it throws a compile error rather than simply not matching and moving on to the next arm? — Edward Peters, Jan 05 '23 at 15:48
@Jmb It should still not match, [as `:literal` captures a _LiteralExpression_](https://doc.rust-lang.org/reference/macros-by-example.html#metavariables) and it does not include the `-`. — Chayim Friedman, Jan 05 '23 at 17:24
There's lots of discussion regarding `-` for `literal`s in [the issue introducing it](https://github.com/rust-lang/rust/issues/35625). However, I don't think this interaction in particular is brought up. I would probably create an issue for it, since it is not intuitive to me (`macro_rules` arms that don't match try the next one), but it may be intended depending on how [transcribing](https://doc.rust-lang.org/reference/macros-by-example.html#transcribing) should be understood. — kmdreko, Jan 05 '23 at 17:58
@ChayimFriedman the document you linked shows `literal: matches -?LiteralExpression` - there's an optional `-` specified there. — Edward Peters, Jan 05 '23 at 21:44

score 2 · Accepted Answer · answered Jan 06 '23 at 00:10

This was asked in issue #82968, and the response was:

This is working as intended, non-terminal metavariables always fully commit to parsing a part of the syntax, if this fails, the whole macro call is rejected.

You can see the same behavior with e.g. expr.

score 1 · Answer 2 · answered Jan 06 '23 at 17:57

Adding to the accepted answer, I've learned the reason for this is that future versions of Rust may support literals (or exprs, or whatevers) that the current version does not. This could lead to breaking code. For instance, suppose that in some version of rust, -3/2 were not a literal, and matching just passed over it. You could then have:

macro_rules! some_macro{
  ($l : literal) => {std::secret_methods::fire_every_nuclear_warhead()};
  ($($tail : tt)*) => {std::secret_methods::pet_a_puppy()`}
}

Then calls to some_macro(-3/2) would resolve safely.. but then when another version of Rust came out that added native support for literal expressions, suddenly someone's production code would cause serious problems.

By instead blowing up at compilation when the first arm fails to match, it's much less likely that presently legal code will become broken in the future.

Why is "-x" trying to parse as a literal and failing in declarative macros?

2 Answers2