Bison get next possible tokens or determine which rule is being attempted

Question

I'm using bison to parse a lang from a spec I don't have control over it. In its definition there is a recursive rule and since the language uses indentation this resulted in reduce/reduce errors. So to get rid of the reduce/reduce I added a semi colon. I have a layout tracker which auto inserts semi colons and braces already. I was thinking of extending the rules for inserting semi colons to support the one I added that's not in the spec but I can't think of a way of knowing when we're at the end of the recursive rule.

Is there a reliable way of knowing when I'm at the end of the recursive rule or any suggestions on a different approach? Or as messy as it'd be is there some way to get two way comms between the parser and lexer?

Currently using a pull parser. I thought using a push parser would enable me to keep better track of where I am in the lexer but when I try the define directive to generate a push parser the option isn't recognised. I'm using bison 3.0.4 with a custom lexer, generating a pure parser with the C++ api.

EDIT:

exp                     : infixexp  TYPE_SEP  type                                                      {}                     
                        | infixexp  TYPE_SEP  context RIGHT_ARROW type                                  {}
                        | infixexp                                                                      {}

infixexp                : lexp qop infixexp                                                             {}    
                        | MINUS infixexp                                                                {}      
                        | lexp                                                                          {}

lexp                    : BACKSLASH apat_list FUNC_TYPE_CONS exp                                        {}   
                        | KW_LET decls KW_IN exp                                                        {}    
                        | KW_IF exp SEMI_COLON KW_THEN exp SEMI_COLON KW_ELSE exp                       {}    //conditional
                        | KW_CASE exp KW_OF L_BRACE alts R_BRACE                                        {}   
                        | KW_DO L_BRACE stmts R_BRACE                                                   {}   
                        | fexp SEMI_COLON                                                               {}

fexp                    : aexp                                                                          {}   
                        | fexp  aexp                                                                    {}  

literal                 : INTEGER                                                                       {}
                        | FLOAT                                                                         {}
                        | CHAR                                                                          {}
                        | STRING                                                                        {}

aexp                    : qvar                                                                          {}  
                        | gcon                                                                          {}   
                        | literal
                        | L_PAREN exp R_PAREN                                                           {}   
                        | L_PAREN exp_list R_PAREN                                                      {}   
                        | L_BRACKET exp R_BRACKET                                                       {}  
                        | L_BRACKET exp_list R_BRACKET                                                  {}  
                        | L_BRACKET exp DOTDOT R_BRACKET                                                {}  
                        | L_BRACKET exp DOTDOT exp R_BRACKET                                            {}  
                        | L_BRACKET exp COMMA exp DOTDOT exp R_BRACKET                                  {}  
                        | L_BRACKET exp PIPE qual_list R_BRACKET                                        {}  
                        | L_PAREN infixexp qop  R_PAREN                                                 {}  
                        | L_PAREN qop infixexp R_PAREN                                                  {}   
                        | qcon  L_BRACE fbind_list R_BRACE                                              {}   
                        | aexp  L_BRACE fbind_list R_BRACE                                              {}   
apat                    : qvar                                                                          {}  
                        | qvar AT_SYM  apat                                                             {}  
                        | gcon                                                                          {}  
                        | qcon  L_BRACE fpat_list R_BRACE                                               {}  
                        | literal                                                                       {}
                        | WILDCARD                                                                      {}
                        | L_PAREN pat R_PAREN                                                           {}  
                        | L_PAREN pat COMMA pat_list R_PAREN                                            {}  
                        | L_BRACKET pat_list  R_BRACKET                                                 {} 
                        | TILDE apat                                                                    {}  

Added a section from the grammar. It's basically a modified version of the Haskell 2010 spec. The reduce/reduce conflicts were resolved by adding the semi colon after fexp in the definition of lexp.

I am simulating indent/dedents and inserting open and curly braces. And I was basically thinking of the lexer hack but couldn't figure out how to do it with Bison. And there are multiple recursive rules but only the one was causing reduce/reduce errors.

EDIT 2:

jsfiddle with the original reduce/reduce errors

Answer 1

The usual way to handle indentation-aware languages is to fabricate INDENT and DEDENT tokens in the lexical scanner. That's easier with a push interface, so it's unfortunate that you are using the bison C++ API which does not implement that feature.

But it can also be done without too much trouble using a shim in between the lexical scanner and the parser. You can see an example of a Python shim in this answer; ply doesn't offer a push parser interface either, so the shim keeps a small persistent queue of tokens which will be sent to the parser and checks that queue before asking the real lexical scanner for the next token.

As that answer indicates, in most layout-aware languages not all newlines are actually semantically significant. For example, in Python itself a newline inside parentheses, braces or brackets is just ordinary whitespace. That rule can easily be implemented by the shim as well (although I didn't complicate the code by doing so in the linked answer), simply by tracking the level of parentheses.

Not all languages make life so easy; you may have a language in which indentation could reassert itself inside a bracketed list because of the presence of a function literal, for example. Or you may have a language like ecmascript where the semi-colon insertion rule allows run-on lines even outside of parentheses, if the alternative parse would not be possible. Haskell has a similar rule, where a brace can be inserted if an alternative parse would not be possible.

The ecmascript rule was drafted with a view to making it possible to write a parser. (Or, more accurately I think, the rule was drafted by reverse-engineering an existing parser, but I can't prove that.) As it turns out, it is possible to implement ecmascript automatic semi-colon insertion by constructing a dictionary of pairs of tokens which can be separated by a newline without a semicolon being inserted. (Or, alternatively, pairs of tokens which must have a semicolon inserted between them if possible, which is the inverse of the other set.) These sets can be constructed automatically by grammar analysis, using the FIRST and FOLLOW sets of each production. (The details of the ecmascript rules require a bit of adjustment because there are some token pairs which could appear in a valid program but which are not allowed to be separated by a newline. For example, return 3 is a valid statement, but if the return is at the end of a line and the 3 is on the following line, a semicolon must be automatically inserted'). Bison does not do this analysis automatically, so it depends on a custom tool, but it is not particularly difficult.

Haskell does not seem to be so accommodating. I see in the Haskell report, section 9.3, at the end of that section:

The parse-error rule is hard to implement in its full generality, because doing so involves fixities. For example, the expression

do a == b == c

has a single unambiguous (albeit probably type-incorrect) parse, namely

(do { a == b }) == c

because (==) is non-associative. Programmers are therefore advised to avoid writing code that requires the parser to insert a closing brace in such situations.

That's not very promising, but it also suggests that implementations are not expected to be perfect, and that programmers are kindly requested to not expect a perfect parser implementation :)

I think translating the IndentWrapper shim in the linked answer into C++ would not be difficult even for someone not too familiar with Python, so I haven't bothered doing it here. If that assumption is incorrect, let me know.