Stack Overflow
Stack Overflow

Questions tagged [lexical-analysis]

Process of converting a sequence of characters into a sequence of tokens.

In computer science, lexical analysis is the process of converting a sequence of characters into a sequence of tokens. A program or function that performs lexical analysis is called a lexical analyzer, lexer, tokenizer, or scanner.

The lexical syntax is usually a regular language, whose atoms are individual characters, while the phrase syntax is usually a context-free language, whose atoms are words (tokens produced by the lexer). While this is a common separation, alternatively, a lexer can be combined with the parser in scannerless parsing.

843 questions
-1
votes
1 answer

Compilers-What is the difference between: 'String literal' and 'Any String' in this exercise?

The dragon book, exercise 3.3.10 The UNIX shell command .sh uses the operators in fig 3.9 in filenames expressions to define sets of file names. For example, the file name expression *.o matches all file names ending in .o; sort1.? matches all file…
Bruno Braga
  • 570
  • 2
  • 4
  • 13
-1
votes
2 answers

Display Output in a text file

I have taken the input code through file and i have to generate data according to it and output it's result in a text file as well.. My Output Code is below.. public void Generator() { /// ....... Code public void DisplayTOKENS() …
Shahtaj Khalid
  • 31
  • 1
  • 11
-1
votes
2 answers

Can Machine Learning be used for Natural Language Understanding

This is based on my earlier question. Can I use machine learning algorithms to help me with understanding sentences? (I will use a closely related example as I used in my previous question). For example, I want my algorithm/code to start a program…
user5139637
  • 775
  • 3
  • 10
  • 29
-1
votes
1 answer

Scanner+Parser created for a Compiler

I have successfully created a scanner for a parser which in turn will be used with a code generator to create a complete compiler. My parser right now can successfully parse assignments, if, if-else, while-do, comparisons, read and write commands.…
RCR1994
  • 29
  • 5
-1
votes
1 answer

Design lexical analyzer and parser in C#

I am going to design a compiler with the help of Gold Parse Builder (GPB) to design lexical analyzer and parser. I am having input x1 = x3/(x2/x5); and need its output like following: [ 1] Variable x1 at 1,1 [ 2] Equals =…
Jaswinder
  • 95
  • 13
-1
votes
3 answers

Why is buffering used in lexical analysis?

Why is buffering used in lexical analysis?and what is best value for EOF?
Pouria golshanrad
  • 102
  • 1
  • 12
-1
votes
1 answer

How can I write a regular expression to recognize the plus operator and the plus sign?

I am trying to write a lexical analyzer for C# language, but I can't figure out how can I differentiate the plus sign from the plus operator, except the context. I need the next token from the source file. So, when I encounter a + how do I now it…
Radu Mardari
  • 131
  • 3
  • 6
-1
votes
2 answers

.class expected in Java?

I am trying to get the next char for my scanner. My constructor is : public S(InputStream inStream) { in = new BufferedReader(new InputStreamReader(inStream)); e = false; // if input stream is done getNextChar(); …
Surz
  • 984
  • 3
  • 11
  • 36
-1
votes
1 answer

How IDE "learns" my code with logic and structure?

Once I use an IDE to compile my C++ code, this IDE seems know everything of my code: I could find the most original definition of any element in my code, no matter how deep it is from usage to definition. The IDE seems know: my code logic my code…
user3789144
  • 1
  • 1
-1
votes
1 answer

Debug parser by printing useful information

I would like to parse a set of expressions, for instance:X[3], X[-3], XY[-2], X[4]Y[2], etc. In my parser.mly, index (which is inside []) is defined as follows: index: | INTEGER { $1 } | MINUS INTEGER { 0 - $2 } The token INTEGER, MINUS etc. are…
SoftTimur
  • 5,630
  • 38
  • 140
  • 292
-1
votes
1 answer

Token recognition error in Flex

I am implementing a parser of which the scanner (implemented using Flex) is not recognizing all the tokens at a time. It is just taking the first token from the input and terminating. Can someone please help me sort out this. Here is my ".lex"…
Ujwal Potluri
  • 39
  • 2
  • 7
-2
votes
1 answer

Lexical Analyzer in Java. Operators Shouldn't be Tokenized as individuals like '++' or '>=' and any unlisted tokens shouldnt print out anything

I am using a Lexical analyzer to tokenize some operators, conditions, and syntaxes. My approach is checking each and every character and when it finds a space between characters, it tokenizes the combined characters. eg. when it finds 'String' it…
hamsoace
  • 77
  • 1
  • 1
  • 8
-2
votes
2 answers

(python - cpp) - How to split the c++ codes while writing a lexical analyzer in python?

I wrote a lexical analyzer for cpp codes in python, but the problem is when I use input.split(" ") it won't recognize codes like x=2 or function() as three different tokens unless I add an space between them manually, like: x = 2 . also it fails to…
B Khorram
  • 1
  • 1
-2
votes
2 answers

What is a "path_expression" in BigQuery

BigQuery describes a path_expression in the Syntax page as follows: A path expression describes how to navigate to an object in a graph of objects and generally follows this…
David542
  • 104,438
  • 178
  • 489
  • 842
-2
votes
1 answer

How do I find the regular expression for- All binary numbers greater than 110011

I am trying to solve a problem set to practice for an exam. How can I approach questions like these ? Is there a way to verify solutions or is it just trial and error ?
Ash Rivers
  • 121
  • 4
1 2 3
…
56 57