0

I have a tokens.ml file which has a type token statement in it. I also have a tokens.mli with the same type token statement. Now, I have a parser.mly which uses the tokens from tokens.mly. I want to keep my tokens in tokens.ml/mli and my parser in parser.mly.

So, I tried compiling my parser using the command

menhir parser.mly --table --explain --external-tokens Tokens

This gives me an error saying one of my tokens does not exist. Specifically,

File "parser.mly", line 173, characters 4-12:
Error: OPERATOR is undefined.

So, menhir is not finding the Tokens module. I don't know how to make it visible to menhir. I tried making a tokens.cma library, but even then I still get the same error.

Menhir doesn't seem to care if the module doesn't exist, because if I run the command

menhir parser.mly --table --explain --external-tokens SomeNonExistentModule

It still gives the same error about OPERATOR being undefined.

How do I get Menhir to find my tokens module. I would prefer to not use ocamlbuild. If you suggest an ocamlbuild solution, please at least explain the intermediate manual steps I could do instead. I want to understand what Menhir expects.

Enrico Borba
  • 1,877
  • 2
  • 14
  • 26
  • In your parser.mly file, do you still have the %token OPERATOR directive at the top? – zehnpaard Jul 03 '19 at 01:02
  • Do I need to have that? I definitely do not have that – Enrico Borba Jul 03 '19 at 01:03
  • I believe so... My understanding is that the --external-tokens directive exists only so that the generated parser.ml and .mli files will not have the token type defined, and instead will reference the external token module. You still need %token ... inside the .mly file – zehnpaard Jul 03 '19 at 01:04
  • You need it anyway for specifying priority/associativity, and also the [docs](http://gallium.inria.fr/~fpottier/menhir/manual.html#sec3) allude to autogenerating the tokens.ml file from the grammar specification, so it's assumed that the %token directives are there – zehnpaard Jul 03 '19 at 01:09
  • 1
    Yes but the docs also say the tokens file can be written by hand and not autogenerated. I'll try that though, thank you. – Enrico Borba Jul 03 '19 at 01:13
  • Yep that did it. Feel free to write an answer and I'll accept. Thanks a ton! – Enrico Borba Jul 03 '19 at 07:16

1 Answers1

3

As discussed in the comments, the error arises because you do not have the

%token OPERATOR

declaration in your .mly file.

Menhir's --external-tokens T option exists to have the generated parser module use T.token instead of generating the token type from the declarations, however the declarations are still necessary inside the .mly file.

As a side note, you can have your tokens in a separate .mly file (e.g. tokens.mly), which will look like below:

tokens.mly:

%token <int> INT
%token EOF
%%

parser.mly:

%start <int> f
%%

f : n = INT; EOF { n }

and then you can run the following commands:

menhir tokens.mly --only-tokens
menhir parser.mly tokens.mly --external-tokens Tokens --base parser

which can be useful if you want to reuse the tokens across parsers etc.

(you can also skip the --only-tokens bit and write tokens.ml by hand, to be consistent with tokens.mly)

zehnpaard
  • 6,003
  • 2
  • 25
  • 40