0

I'm working on a grammar for a language that supports the right shift operator and generic types. For example:

function rectangle(): Pair<Tuple<Float, Float>> {
    let x = 0 >> 2; 
}

My problem is, during scanning, the right shift operator is correctly tokenized, but the >> in Pair<Tuple<Float, Float>> becomes a single >> token instead of two separate > tokens (unless I add a space). This is because I have the >> before the > in my .jison file:

">>"    { return '>>' }
">"     { return '>' }

Is there a good way to resolve this in Jison? I feel like this is a common problem as my syntax is similar to every other C-style language, but I haven't found a solution to it yet (besides writing a pre-scan script that manually space-delimits the >s).

WaltersGE1
  • 813
  • 7
  • 26

1 Answers1

2

The easiest solution is to just not recognize >> as a single token in the lexer. Instead, in your parser, recognize two consecutive > tokens as a right shift, and then check to make sure there's nothing (no whitespace or comments) between them (and give a syntax error if there is).

Chris Dodd
  • 119,907
  • 13
  • 134
  • 226
  • That's actually what I'm doing now. But the problem is that, since I'm ignoring all whitespace between tokens, the statement ```let x = 0 > > 2;``` is actually valid, which I don't want it to be. I guess I could try making whitespace a token (instead of ignoring it) and include it in my grammar everywhere else. – WaltersGE1 Jun 03 '18 at 04:28
  • 2
    @WaltersGE1 You can check whether two tokens have white space between them while still ignoring white space by checking that their locations touch. Something like `expr '>' '>' expr { if (@2.last_column == @3.first_column) { $$ = makeRightShift($1, $4); } else { error("Unexpected '>'", @3); } }` – sepp2k Jun 03 '18 at 13:24