Stack Overflow
Stack Overflow

Questions tagged [lexical-analysis]

Process of converting a sequence of characters into a sequence of tokens.

In computer science, lexical analysis is the process of converting a sequence of characters into a sequence of tokens. A program or function that performs lexical analysis is called a lexical analyzer, lexer, tokenizer, or scanner.

The lexical syntax is usually a regular language, whose atoms are individual characters, while the phrase syntax is usually a context-free language, whose atoms are words (tokens produced by the lexer). While this is a common separation, alternatively, a lexer can be combined with the parser in scannerless parsing.

843 questions
-4
votes
1 answer

Python - How does the following expression get evaluated?

a=5;b=10;a----------------++++++++++++++++++b # Output: 15 The above is the output of a python script. Could you please explain how it works? I was quite surprised that it works and gives an answer!! However a similar such expression doesn't work…
Varun Rao
  • 393
  • 1
  • 5
  • 19
-4
votes
2 answers

Lexical analysis: Comments at end of line

I am tokenizing a string such as: BAS=W34 N29 E24 S29$FOP=E6 S6 W6 N6$. A Comment The period is the "end of command" character, not a "beginning of comment" character. How can I add a regular expression rule to the lexer such that the period is a…
Chet
  • 3,461
  • 1
  • 19
  • 24
-5
votes
1 answer

Why this code returns undefined when AST already knows the value

var name = 'ali' function sayHi() { console.log(name); console.log(age); var name = 'Lydia'; let age = 21; } sayHi() When i put this code AST explorer and run it. I actually see that name has already been assigned to variable 'Lydia'…
ikarus
  • 11
  • 3
1 2 3
…
56
57