how to debug a custom proc macro?

Question

I am studying Rust macro, and i want to see the expand output of the macro. I had tried by using command: rustc +nightly -Zunpretty=expanded xxx.rs follow the tutorial The Book of Rust Macros - Debugging. but i got the error when i use the custom macro in the same crate: can't use a procedural macro from the same crate

But when i moved the test code into other file, it still failed with another message: unresolved import

And then when i changed the code followed the help message, it change the error message: can't find crate

here are my demo:

use syn;
use quote::{quote, ToTokens, TokenStreamExt};
use proc_macro;

#[proc_macro_derive(HelloMacro)]
pub fn hello_macro_derive(input: proc_macro::TokenStream) -> proc_macro::TokenStream {
    // Construct a representation of Rust code as a syntax tree
    // that we can manipulate
    let ast = syn::parse(input).unwrap();

    // Build the trait implementation
    impl_hello_macro(&ast)
}

fn impl_hello_macro(ast: &syn::DeriveInput) -> proc_macro::TokenStream {
    let name = &ast.ident;
    let gen = quote! {
        impl HelloMacro for #name {
            fn hello_macro() -> String {
                String::from(stringify!(#name))
            }
        }
    };
    gen.into()
}

#[cfg(test)]
mod tests {
    use super::*;
    
    #[test]
    fn test_macro () {
        trait HelloMacro {
            fn hello_macro() -> String {
                String::from("666")
            }
        }

        #[derive(HelloMacro)]
        struct A;

        
        assert_eq!(A::hello_macro(), "A".to_string());

    }
}

I had also tried using cargo-expand, but the output still incorrect.

error[E0514]: found crate `my_macro` compiled by an incompatible version of rustc
 --> my_macro\tests\test_macro.rs:1:5
  |
1 | use my_macro::{ HelloMacro };
  |     ^^^^^^^^
  |
  = note: the following crate versions were found:
          crate `my_macro` compiled by rustc 1.66.1 (90743e729 2023-01-10): \\?\D:\vscode\rust-study\game\my_mine_sweeper\target\debug\deps\my_macro-b82452821ca4c556.dll
  = help: please recompile that crate using this compiler (rustc 1.69.0-nightly (2773383a3 2023-02-10)) (consider running `cargo clean` first)
error: cannot determine resolution for the derive macro `HelloMacro`
  --> my_macro\tests\test_macro.rs:11:14
   |
11 |     #[derive(HelloMacro)]
   |              ^^^^^^^^^^
   |
   = note: import resolution is stuck, try simplifying macro imports
For more information about this error, try `rustc --explain E0514`.
error: could not compile `my_macro` due to 2 previous errors

#![feature(prelude_import)]
#[prelude_import]
use std::prelude::rust_2021::*;
#[macro_use]
extern crate std;
use my_macro::HelloMacro;
extern crate test;
#[cfg(test)]
#[rustc_test_marker = "test_macro"]
pub const test_macro: test::TestDescAndFn = test::TestDescAndFn {
    desc: test::TestDesc {
        name: test::StaticTestName("test_macro"),
        ignore: false,
        ignore_message: ::core::option::Option::None,
        compile_fail: false,
        no_run: false,
        should_panic: test::ShouldPanic::No,
        test_type: test::TestType::IntegrationTest,
    },
    testfn: test::StaticTestFn(|| test::assert_test_result(test_macro())),
};
fn test_macro() {
    trait HelloMacro {
        fn hello_macro() -> String {
            String::from("666")
        }
    }
    struct A;
    match (&A::hello_macro(), &"A".to_string()) {
        (left_val, right_val) => {
            if !(*left_val == *right_val) {
                let kind = ::core::panicking::AssertKind::Eq;
                ::core::panicking::assert_failed(
                    kind,
                    &*left_val,
                    &*right_val,
                    ::core::option::Option::None,
                );
            }
        }
    };
}
#[rustc_main]
pub fn main() -> () {
    extern crate test;
    test::test_main_static(&[&test_macro])
}

I just want to know how to debug the procedural macro and see the expand output since it is hard to know whether the result is correct.

Answer 1

The invocation of a macro and the macro itself cannot be in the same crate.

That's why many projects are split into two crates:

• <cratename>
• <cratename>-macros

For example tokio and tokio-macros.

Usually, this is then organized into a workspace layout, with <cratename> and <cratename>-macros as subcrates. As in tokio. (Note that tokio has many more subcrates, but only tokio and tokio-macros are important for your problem)

---

You are probably aware of all of this already and only want to know how to test the macros in your <cratename>-macros crate.

You cannot test macros in the same file. You can, however, use the tests directory. It's more appropriate anyway, because your tests are integration tests. In-file tests are more meant for unit tests, as you have full access to struct internals. In the tests directory the visibility of your crate is more representative of how users would see your crate.

Like this:

• src/lib.rs:

use proc_macro;
use quote::quote;
use syn;

#[proc_macro_derive(HelloMacro)]
pub fn hello_macro_derive(input: proc_macro::TokenStream) -> proc_macro::TokenStream {
    // Construct a representation of Rust code as a syntax tree
    // that we can manipulate
    let ast = syn::parse(input).unwrap();

    // Build the trait implementation
    impl_hello_macro(&ast)
}

fn impl_hello_macro(ast: &syn::DeriveInput) -> proc_macro::TokenStream {
    let name = &ast.ident;
    let gen = quote! {
        impl HelloMacro for #name {
            fn hello_macro() -> String {
                String::from(stringify!(#name))
            }
        }
    };
    gen.into()
}

• tests/test_macros.rs:

use rust_playground::HelloMacro;

#[test]
fn test_macro() {
    trait HelloMacro {
        fn hello_macro() -> String {
            String::from("666")
        }
    }

    #[derive(HelloMacro)]
    struct A;

    assert_eq!(A::hello_macro(), "A".to_string());
}

$ cargo test
   Compiling rust_playground v0.1.0 (/home/martin/work/rust_playground)
    Finished test [unoptimized + debuginfo] target(s) in 0.45s
     Running unittests src/lib.rs (target/debug/deps/rust_playground-4e67636c4d9901bb)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running tests/test_macros.rs (target/debug/deps/test_macros-1278dc310f04b8db)

running 1 test
test test_macro ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

   Doc-tests rust_playground

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

---

More (potentially) helpful tips:

For quick and dirty feedback you can also use cargo-expand to show what your macro expands to.

To install it, run:

cargo install cargo-expand
rustup component add rustfmt

Then, let's for example expand the tests/test_macros.rs file:

$ cargo expand --test test_macros
#![feature(prelude_import)]
#[prelude_import]
use std::prelude::rust_2021::*;
#[macro_use]
extern crate std;
use rust_playground::HelloMacro;
extern crate test;
#[cfg(test)]
#[rustc_test_marker = "test_macro"]
pub const test_macro: test::TestDescAndFn = test::TestDescAndFn {
    desc: test::TestDesc {
        name: test::StaticTestName("test_macro"),
        ignore: false,
        ignore_message: ::core::option::Option::None,
        compile_fail: false,
        no_run: false,
        should_panic: test::ShouldPanic::No,
        test_type: test::TestType::IntegrationTest,
    },
    testfn: test::StaticTestFn(|| test::assert_test_result(test_macro())),
};
fn test_macro() {
    trait HelloMacro {
        fn hello_macro() -> String {
            String::from("666")
        }
    }
    struct A;
    impl HelloMacro for A {
        fn hello_macro() -> String {
            String::from("A")
        }
    }
    match (&A::hello_macro(), &"A".to_string()) {
        (left_val, right_val) => {
            if !(*left_val == *right_val) {
                let kind = ::core::panicking::AssertKind::Eq;
                ::core::panicking::assert_failed(
                    kind,
                    &*left_val,
                    &*right_val,
                    ::core::option::Option::None,
                );
            }
        }
    };
}
#[rustc_main]
pub fn main() -> () {
    extern crate test;
    test::test_main_static(&[&test_macro])
}

It is a little bit noisy because it expands all macros, including #[test]. But you can find the expansion of your macro in there:

    struct A;
    impl HelloMacro for A {
        fn hello_macro() -> String {
            String::from("A")
        }
    }

Answer 2

You can fully debug proc macros by creating yet another project that your macro project refers to. It will be a normal (non-macro) project that your macro will refer to. It must use the proc_macro2 crate in place of the proc_macro crate. You can use Rust workspaces to organize all these projects.

I think it's worth adding this extra layer because it makes creating and testing macro code as easy as normal code.

For details and examples see rule #1 in Nine Rules for Creating Procedural Macros in Rust.

Answer 3

As @Carl pointed out,

You can fully debug proc macros by creating yet another project that your macro project refers to. It will be a normal (non-macro) project that your macro will refer to. It must use the proc_macro2 crate in place of the proc_macro crate. You can use Rust workspaces to organize all these projects.

And I agree that it's worth adding this extra layer because it makes creating and testing macro code as easy as normal code.

For convenience, I've created proc_macro_template to simplify the task. It is based on "Nine Rules for Creating Procedural Macros in Rust".