Is this the job of the lexer?

Question

Let's say I was lexing a ruby method definition:

def print_greeting(greeting = "hi")  
end

Is it the lexer's job to maintain state and emit relevant tokens, or should it be relatively dumb? Notice in the above example the greeting param has a default value of "hi". In a different context, greeting = "hi" is variable assignment which sets greeting to "hi". Should the lexer emit generic tokens such as IDENTIFIER EQUALS STRING, or should it be context-aware and emit something like PARAM_NAME EQUALS STRING?

score 5 · Accepted Answer · answered Jun 15 '11 at 14:40

5

I tend to make the lexer as stupid as I possibly can and would thus have it emit the IDENTIFIER EQUALS STRING tokens. At lexical analysis time there is (most of the time..) no information available about what the tokens should represent. Having grammar rules like this in the lexer only polutes it with (very) complex syntax rules. And that's the part of the parser.

answered Jun 15 '11 at 14:40

Jasper Bekkers

6,711
32
46

1

Without context sensitivity, how do you pull off syntax such as `new Map>()`? – Flosculus Oct 23 '15 at 15:52

score 3 · Answer 2 · answered Jun 15 '11 at 14:43

3

I think that lexer should be "dumb" and in your case should return something like this: DEF IDENTIFIER OPEN_PARENTHESIS IDENTIFIER EQUALS STRING CLOSE_PARENTHESIS END. Parser should do validation - why split responsibilities.

answered Jun 15 '11 at 14:43

Zuljin

2,612
17
14

score 2 · Answer 3 · answered Jun 16 '11 at 21:00

Distinction between lexical analysis and parsing is an arbitrary one. In many cases you wouldn't want a separate step at all. That said, since the performance is usually the most important issue (otherwise parsing would be mostly trivial task) then you need to decide, and probably measure, whether additional processing during lexical analysis is justified or not. There is no general answer.

score 1 · Answer 4 · answered Jun 15 '11 at 15:27

Don't work with ruby, but do work with compiler & programming language design.

Both approches work, but in real life, using generic identifiers for variables, parameters and reserved words, is more easier ("dumb lexer" or "dumb scanner").

Later, you can "cast" those generic identifiers into other tokens. Sometimes in your parser.

Sometimes, lexer / scanners have a code section, not the parser , that allow to do several "semantic" operations, incduing casting a generic identifier into a keyword, variable, type identifier, whatever. Your lexer rules detects an generic identifier token, but, returns another token to the parser.

Another similar, common case, is when you have an expression or language that uses "+" and "-" for binary operator and for unary sign operator.

Is this the job of the lexer?

4 Answers4