Question on invoking another macro_rules in macro_rules definition

Question

I'm implementing writing TLV packet to somewhat impl std::io::Write.

First I implement WriteBE<T> trait, whose write_be(&mut self, data: T) method can write data with type T to Self. (implementation details omitted)

And I'm trying to use macro_rules! to implement calculation of total packet length in compile time (because most packets have fixed length in my case). macros are as follows:

macro_rules! len_in_expr {
    (
        self.write_be( $data: expr $(,)? ) $(?)? $(;)*
    ) => {
        std::mem::size_of_val(&$data)
    };
    (
        write_be(self, $data: expr $(,)? ) $(?)? $(;)*
    ) => {
        std::mem::size_of_val(&$data)
    };
    (
        $other: expr
    ) => {
         0
    };
}

/// calculate total write size in block
macro_rules! tlv_len_in_block {
    ({
        $( $e: expr );* $(;)?
    }) => {
        0 $(
            + ( len_in_expr!($e) )
        )*
    };
}

But when I calculating total length like this:

fn main() {
    let y = tlv_len_in_block!({
        write_be(self, 0u32,)?;
    });
    println!("y={}", y);
}

I get a result 0.

If I comment the $other: expr match arm, I get a compile error:

6  |   macro_rules! len_in_expr {
   |   ------------------------ when calling this macro
...
30 |               + ( len_in_expr!($e) )
   |                                ^^ no rules expected this token in macro call
...
39 |       let y = tlv_len_in_block!({
   |  _____________-
40 | |         write_be(self, 0u32,)?;
41 | |     });
   | |______- in this macro invocation

What's the problem with my code? And how can I fix it?

Answer 1

Once metavariables inside macro_rules! are captured into some fragment specifier (e.g. expr), they cannot be decomposed anymore. Quoting the reference:

When forwarding a matched fragment to another macro-by-example, matchers in the second macro will see an opaque AST of the fragment type. The second macro can't use literal tokens to match the fragments in the matcher, only a fragment specifier of the same type. The ident, lifetime, and tt fragment types are an exception, and can be matched by literal tokens. The following illustrates this restriction:

macro_rules! foo {
    ($l:expr) => { bar!($l); }
// ERROR:               ^^ no rules expected this token in macro call
}

macro_rules! bar {
    (3) => {}
}

foo!(3);

The following illustrates how tokens can be directly matched after matching a tt fragment:

// compiles OK
macro_rules! foo {
    ($l:tt) => { bar!($l); }
}

macro_rules! bar {
    (3) => {}
}

foo!(3);

Once tlv_len_in_block!() captured write_be(self, 0u32,)? inside $e, it cannot be decomposed into write_be(self, $data:expr $(,)? ) and thus, cannot be matched by the second case of the len_in_expr!()` macro, as it should have been.

There are generally two solutions to this problem:

The first is, if possible, decomposing them from the beginning. The problem is that this is not always possible. In this case, for example, I don't see a way for that to work.

The second way is much more complicated and it is using the Push-down Accumulation technique together with tt Munching.

The idea is as follows: instead of parsing the input as whole, we parse each piece one at a time. Then, recursively, we forward the parsed bits and the yet-to-parse bit to ourselves. We also should have a stop condition on an empty input.

Here is how it will look like in your example:

macro_rules! tlv_len_in_block_impl {
    // Stop condition - no input left to parse.
    (
        parsed = [ $($parsed:tt)* ]
        rest = [ ]
    ) => {
        $($parsed)*
    };
    (
        parsed = [ $($parsed:tt)* ]
        rest = [
            self.write_be( $data:expr $(,)? ) $(?)? ;
            $($rest:tt)*
        ]
    ) => {
        tlv_len_in_block_impl!(
            parsed = [
                $($parsed)*
                + std::mem::size_of_val(&$data)
            ]
            rest = [ $($rest)* ]
        )
    };
    (
        parsed = [ $($parsed:tt)* ]
        rest = [
            write_be(self, $data:expr $(,)? ) $(?)? ;
            $($rest:tt)*
        ]
    ) => {
        tlv_len_in_block_impl!(
            parsed = [
                $($parsed)*
                + std::mem::size_of_val(&$data)
            ]
            rest = [ $($rest)* ]
        )
    };
}

/// calculate total write size in block
macro_rules! tlv_len_in_block {
    ({
        $($input:tt)*
    }) => {
        tlv_len_in_block_impl!(
            parsed = [ 0 ]
            rest = [ $($input)* ]
        )
    };
}

(Note that this is not exactly the same as your macro - mine requires a trailing semicolon, while in yours it's optional. It's possible to make it optional here, too, but it will be much more complicated.

Playground.