How to pass parameters loaded from configuration file to a procedural macro function?

Question

Here is a problem I am trying to solve. I have multiple procedural macro functions that generate tables of pre-computed values. Currently my procedural macro functions take parameters in the form of literal integers. I would like to be able to pass these parameters from a configuration file. I could re-write my functions to load parameters from macro themselves. However, I want to keep configuration from a top level crate, like in this example:

top-level-crate/
    config/
        params.yaml
    macro1-crate/
    macro2-crate/
     

Since the input into a macro function is syntax tokens not run-time values, I am not able to load a file from top-level-crate and pass params.

    use macro1_crate::gen_table1;
    use macro2_crate::gen_table2;

    const TABLE1: [f32;100] = gen_table1!(500, 123, 499);
    const TABLE2: [f32;100] = gen_table2!(1, 3);

    fn main() {
       // use TABLE1 and TABLE2 to do further computation.
        
    }

I would like to be able to pass params to gen_table1 and gen_table2 from a configuration file like this:

    use macro1_crate::gen_table1;
    use macro2_crate::gen_table2;
   
    // Load values PARAM1, PARAM2, PARAM3, PARAM4, PARAM5

    const TABLE1: [f32;100] = gen_table1!(PARAM1, PARAM2, PARAM3);
    const TABLE2: [f32;100] = gen_table2!(PARAM4, PARAM5);

    fn main() {
       // use TABLE1 and TABLE2 to do further computation.
        
    }

The obvious problem is that PARAM1, PARAM2, PARAM3, PARAM4, PARAM5 are runtime values, and proc macros rely on build time information to generate tables.

One option I am considering is to create yet another proc macro specifically to load configuration into some sort of data-structure built using quote! tokens. Then feed this into macros. However, this feels hackish and the configuration file needs to be loaded several times. Also the params data structure need to be tightly coupled across macros. The code might look like this:

    use macro1_crate::gen_table1;
    use macro2_crate::gen_table2;
   
    const TABLE1: [f32;100] = gen_table1!(myparams!());
    const TABLE2: [f32;100] = gen_table2!(myparams!());

    fn main() {
       // use TABLE1 and TABLE2 to do further computation.
        
    }

Any improvements or further suggestions?

Answer 1

gen_table1!(myparams!()); won't work: macros are not expanded from the inside out, like function calls. Your gen_table1 macro will receive the literal token stream myparams ! () and won't be able to evaluate this macro, thus not having access to the "return value" of myparams.

Right now, I only see one real way to do what you want: load the parameters from the file in gen_table1 and gen_table2, and just pass the filename of the file containing the parameters. For example:

const TABLE1: [f32; 100] = gen_table1!("../config/params.yaml");
const TABLE2: [f32; 100] = gen_table2!("../config/params.yaml");

Of course, this could lead to duplicate code in these two macros. But that should be solvable with the usual tools: extract that parameter loading into a function (in case both macros live in the same crate) or into an additional utility crate (in case the two macros live in different crates).

---

You also keep mentioning the term "runtime values". I think you mean "a const value, not a literal" and that you are referring to something like this:

const PARAM1: u32 = load_param!();
const TABLE1: [f32; 100] = gen_table1!(PARAM1); // <-- this does not work as expected!

Because here, again, your macro receives the literal token stream PARAM1 and not the value of said parameter.

So yes, I think that's what you mean by "runtime value". Granted, I don't have a better term for this right now, but "runtime value" is misleading/wrong because the value is available at compile time. If you were talking about an actual runtime value, i.e. a value that is ONLY knowable at runtime AFTER compilation is already done, then it would be impossible to do what you want. That's because proc macros run once at compile time, and never at runtime.