Stack Overflow
Stack Overflow

Questions tagged [lexical-analysis]

Process of converting a sequence of characters into a sequence of tokens.

In computer science, lexical analysis is the process of converting a sequence of characters into a sequence of tokens. A program or function that performs lexical analysis is called a lexical analyzer, lexer, tokenizer, or scanner.

The lexical syntax is usually a regular language, whose atoms are individual characters, while the phrase syntax is usually a context-free language, whose atoms are words (tokens produced by the lexer). While this is a common separation, alternatively, a lexer can be combined with the parser in scannerless parsing.

843 questions
-1
votes
1 answer

Can't quite get the Lexical Analyzer to work in my Java program

I am trying to get this piece of Java code to lexically analyze the phrase "(sum + 47) / total" and spit it out as: Next token is: 25 Next lexeme is ( Next token is: 11 Next lexeme is sum Next token is: 21 Next lexeme is + Next…
JirenBlanco
  • 3
  • 1
-1
votes
1 answer

Flex regular expression for strings with either single or double quotes

I am writing a regular expression for a lexical analyzer for class. I presently have a regular expression written for regular strings with double quotes, however my professor wants us to account for single quotes, too. Here is my current regular…
poppy14s
  • 37
  • 11
-1
votes
1 answer

How does the handling of syntax errors involves the use of parser and the lexical analyzer?

I am a little confused about this question. Someone please clarify, "how does the handling of syntax errors involves the use of parser and the lexical analyzer?" A syntax error is basically handled at the parser phase of the compiler, so how does…
MohsinPak
  • 1
-1
votes
1 answer

How to convert for loop into while loop in LEX?

I am stuck at a program where I have to convert “for-loop”/“do-while loop” to “while loop” without changing the meaning of the program. There may be loops inside loops and so on. The input would be a C language program and output would be a valid C…
PeakyBlinder
  • 1,059
  • 1
  • 14
  • 35
-1
votes
2 answers

How to tokenize printf("result is %d\n",a) ; statement by lexical analyser

If the lexical anlyzer takes "result is %d\n" as one token in: printf("result is %d\n",a); then how and at what stage is %d and \n are recognized as format specifier and next line respectively.
smiley
  • 3
  • 2
-1
votes
2 answers

stack translator in formal languages

Can someone explain how a stack translator works ? I think it is mainly used for lexical analysis (i could be very much wrong) . Any additional material or links are welcome ! Thanks !
Alex
  • 10,869
  • 28
  • 93
  • 165
-1
votes
1 answer

Practical uses of inclusive start conditions in scanner generator

What are some real-world (not-contrived) lexical-scanning problems where "inclusive scan conditions" (as opposed to "exclusive" ones) are a better solution? That is, when is %s FOO any better than %x FOO in a (f)lex definition? I understand the…
Ian
  • 4,421
  • 1
  • 20
  • 17
-1
votes
1 answer

How to implement an efficient tokenizer with a regex, using group names

I am trying to write a tokenizer for parsing a text body (input string) using a Regex. What I want is to split the input in individual tokens and store these in a List, where token is a (C#) class like class Token { string value; string…
John Pool
  • 87
  • 9
-1
votes
1 answer

Symbol Table and line number population in compiler phases

We know compiler saves line numbers in symbol table during lexical analysis. I had been wondering if it is possible to save line numbers in any other phases of analysis of source code? If possible then when and how?
Murad Ahmed
  • 1
-1
votes
1 answer

how extract a part of "strings" android

I have been trying to extract "Near Gangotri hospital" and "no 125/6 8th Miles Bus Stop" but i was not able to extract it can any one please help me to extract the string I need to extract 3 rd string which is after 2 nd Pipe(|)…
Nageash Reddy
  • 19
  • 3
-1
votes
2 answers

Lexical analyser : how to identify the end of a token

I need a function that identifies the end of token so that i can save in it an array and send it to my automata for identification(Operator,Keyword,Identifiers) the automata is working great when i enter only 1 token , but when there is `lots of…
Sierra
  • 11
  • 3
-1
votes
1 answer

Ruby LittleLexer

I'm wondering if Ruby's LittleLexer project might have been renamed or moved? Following the link below doesn't seem to work any longer. http://rubyforge.org/projects/littlelexer/
user3908316
  • 73
  • 5
-1
votes
1 answer

how to split multiple line input using split function java

i want to split a multiple line input using split function when i tried it it was not working public static void main(String [] args) { String TER = ","; int i=0; java.util.Scanner a = new java.util.Scanner(System.in); …
dan diago
  • 17
  • 6
-1
votes
4 answers

If you are forced to simplify C# looping keywords, choose only one you want to preserve

If you are forced to simplify C# keywords that can be used for looping, choose only one you want to preserve. for do-while while goto-label-if foreach Is there any performance consideration regarding your decision? Actually I don't know the…
Second Person Shooter
  • 14,188
  • 21
  • 90
  • 165
-1
votes
2 answers

Good ways to visualize a word-like document

What are some good ways to visualize a word-like document? Edit: It doesn't have to be a .doc, it could be a text file or blog post...
Comma
  • 1,567
  • 3
  • 15
  • 22
1 2 3
…
56 57