Bison and Flex do not stop when reached EOF, only say "Reading a token:" and wait for input

Question

Well, I've searched my problem for quite a long time and I got nothing. Questions I seen below could not help my situation:

bison end of file

Lex/Flex - Scanning for the EOF character

Flex/Bison EOF propagation from stdin vs a file

END OF FILE token with flex and bison (only works without it)

Situation description:

1. I use Bison and Flex to parse a custome language, say this language's name is MYLAN;

2. in MYLAN syntax, I wrote %start system_decl and system_decl:module_decls{...}; in bison file .ypp, no token denoting "system begin" or "system end" as you can see, the system is simply a comibnation of modules. A file written in MYLAN may look like this:

   module
   ...
   end
   module
   ...
   end
   module
   ...
   end
   

3. the parser worked fine for all the syntax described in .ypp, but when it reached the end of file (it should be EOF but I'm not sure since I tried <<EOF>> but the problem still exists), it could not stop, it keeped wating for a token:

   [MYLANParser] $ ./MYLANParser example.mylan
   ...
   Stack now 0 3
   Entering state 6
   Reducing stack by rule 50 (line 458):
   $1 = nterm module_decls ()
   $2 = nterm module_decl ()
   $$ = nterm module_decls ()
   Stack now 0
   Entering state 3
   Reading a token:
   

It just stuck here and the parser will not stop, anyone can help? Many thanks in advance!

Answer 1

The Reading a token: string is printed immediately before the parser calls yylex. After yylex returns, it will print the token (or EOF) that was returned, so what appears to be happening is that your lexer is hanging when it gets to the EOF.

The most likely reason for something like that is having an <<EOF>> rule or yywrap function that tells the lexer to continue reading from the input but doesn't actually reset the input to point to something else (so it just reads another EOF and calls the <<EOF>> rule or yywrap function again, resulting in an infinite loop.)

Show your lexer (.l file and yywrap function) and someone will probably be able to tell you what is wrong with it.

Answer 2

I realize that this is old but with 2k views maybe it's still helpful.

Look into yyterminate(). Think about whether you want to just terminate or pass a final token up indicating end of file which the grammar includes.

My parsers usually have c-include-equivalent functionality, so I use

yypush_buffer_state(yy_create_buffer(yyin, YY_BUF_SIZE));

when opening files (you'll need to fopen the file at yyin before doing that) and

yypop_buffer_state()
if (!YY_CURRENT_BUFFER) {yyterminate();}

when hitting the

<<EOF>>

pattern.