Is the word "lexer" a synonym for the word "parser"?

Question

The title is the question: Are the words "lexer" and "parser" synonyms, or are they different? It seems that Wikipedia uses the words interchangeably, but English is not my native language so I can't be sure.

Note that a "lexer" is sometimes also called a "tokenizer" or "scanner". — Bart Kiers, May 09 '11 at 18:52
@Bart thanks, I had accidentally written the question down wrong. I meant "Is lexer a synonym for tokenizer" so your comment is actually the answer. Thanks. — Seth Carnegie, May 09 '11 at 19:00

hammar · Answer 1 · 2011-05-09T19:01:41.917

A lexer is used to split the input up into tokens, whereas a parser is used to construct an abstract syntax tree from that sequence of tokens.

Now, you could just say that the tokens are simply characters and use a parser directly, but it is often convenient to have a parser which only needs to look ahead one token to determine what it's going to do next. Therefore, a lexer is usually used to divide up the input into tokens before the parser sees it.

A lexer is usually described using simple regular expression rules which are tested in order. There exist tools such as lex which can generate lexers automatically from such a description.

[0-9]+  Number
[A-Z]+  Identifier
+       Plus

A parser, on the other hand, is typically described by specifying a grammar. Again, there exist tools such as yacc which can generate parsers from such a description.

expr ::= expr Plus expr
       | Number
       | Identifier

score 7 · Accepted Answer · answered May 09 '11 at 18:50

No. Lexer breaks up input stream into "words"; parser discovers syntactic structure between such "words". For instance, given input:

velocity = path / time;

lexer output is:

velocity (identifier)
= (assignment operator)
path (identifier)
/ (binary operator)
time (identifier)
; (statement separator)

and then the parser can establish the following structure:

= (assign)
  lvalue: velocity
  rvalue: result of
    / (division)
      dividend: contents of variable "path"
      divisor: contents of variable "time"

score 6 · Answer 3 · answered May 09 '11 at 18:49

6

No. A lexer breaks down the source text into tokens, whereas a parser interprets the sequence of tokens appropriately.

answered May 09 '11 at 18:49

Ignacio Vazquez-Abrams

776,304
153
1,341
1,358

score 3 · Answer 4 · answered May 09 '11 at 18:50

They're different.

A lexer takes a stream of input characters as input, and produces tokens (aka "lexemes") as output.

A parser takes tokens (lexemes) as input, and produces (for example) an abstract syntax tree representing statements.

The two are enough alike, however, that quite a few people (especially those who've never written anything like a compiler or interpreter) treat them as the same, or (more often) use "parser" when what they really mean is "lexer".

score 1 · Answer 5 · answered May 09 '11 at 18:52

As far as I know, lexer and parser are allied in meaning but are not exact synonyms. Though many sources do use them as similar a lexer (abbreviation of lexical analyser) identifies tokens relevant to the language from the input; while parsers determine whether a stream of tokens meets the grammar of the language under consideration.

Is the word "lexer" a synonym for the word "parser"?

5 Answers5