Stack Overflow
Stack Overflow

Questions tagged [lexical-analysis]

Process of converting a sequence of characters into a sequence of tokens.

In computer science, lexical analysis is the process of converting a sequence of characters into a sequence of tokens. A program or function that performs lexical analysis is called a lexical analyzer, lexer, tokenizer, or scanner.

The lexical syntax is usually a regular language, whose atoms are individual characters, while the phrase syntax is usually a context-free language, whose atoms are words (tokens produced by the lexer). While this is a common separation, alternatively, a lexer can be combined with the parser in scannerless parsing.

843 questions
-2
votes
2 answers

How to do a mathematical operation with a single string?

A single string contains two numbers and operator, the two numbers have to be detected and then do the mathematical operation. I already did the part that to detect the two numbers and the operator, but I could not make a subtring for each number…
ives rodriguez
  • 7
  • 6
-2
votes
2 answers

What is Subset Construction in Compiler design?

here is the algorithm : I am going through the aho,Ulman reference book on Compiler Construction, which explains the subset construction implementation algorithm for NFA to DFA conversion. The explanation there is very brief. I would like to have a…
RJSD3V
  • 101
  • 1
  • 6
-2
votes
1 answer

Why do we need a conditional equals operator?

Why do we need to differentiate between an assignment and a boolean condition, as in the '=' vs '==' operators? Wouldn't context be enough to prevent ambiguity? Is there an example of when it would be ambiguous? It's not really a programming…
Cameron Cassidy
  • 73
  • 6
-2
votes
1 answer

How to recognize ID, Literals and Comments in Lex file

I have to write a lex program that has these rules: Identifiers: String of alphanumeric (and _), starting with an alphabetic character Literals: Integers and strings Comments: Start with ! character, go to until the end of the line Here is what I…
reand
  • 11
  • 7
-2
votes
1 answer

C# (My own Programming Language) - How to find PRINT STRING more than once when parsing

So I am currently making my own programming language based off of howCode's programming language in Python, but I simply took an hour or so to attempt to convert it into C#, and it went great, although, when I tell the parse to parse the tokens we…
One Ace
  • 31
  • 5
-2
votes
1 answer

how to find multiline comments from a java file?

I've read my java source file and stored it's contents in String s However i'm facing difficulty in finding the multiline comments from file My task is to find the multiline comments like these:- /* i am helpful i am great */ and display them
Harkamal Singh
  • 45
  • 13
-2
votes
1 answer

How to write a Python lexical analyser?

I'm trying to write a C module to lexically analyse Python code. How can I do it?
dish
  • 19
  • 1
  • 1
-2
votes
1 answer

Solve An Equation In C++

my question is how would you go about doing a program that solves an equation in C++. For example, User Input: -5*2+(5+1)+(5-2) Program Output: -1 Any code examples, links, help in general is greatly apreciated! :D Thanks!
Brian Moreno
  • 109
  • 10
-2
votes
1 answer

Cause for Segmentation Fault?

I've written some code for a scanner in C++, but I keep receiving a segmentation fault. The strange thing is that the segmentation fault happens at the completion of the code. I think it has to do with my scan function and the use of file.get() in…
Evytyn
  • 81
  • 1
  • 9
-2
votes
1 answer

The lexical analysis part: analyse ?: as an operator instead of separator

My target is to implement a very simple lexical analysis of the C language, such as when you read an expression ab=3, then It will be analysed as: ID OP<=> LIT<8> (op will be "operator",LIT will be "literal"). There is also an situation…
ryu
  • 651
  • 9
  • 23
-2
votes
2 answers

A weird situation about matching in flex

I am writing a scanner in flex, and I have the following two definitions: %% "int" printf("JUST_INT"); "int"[ \t\n]+"matrix" printf("MATRIX_INT"); [A-Za-z][A-Za-z0-9]* printf("IDENTIFIER"); %% When the input to the scanner is int matrixM = 3; the…
yrazlik
  • 10,411
  • 33
  • 99
  • 165
-3
votes
1 answer

Iterate over a list of values within a Hashmap

Lexical Analysis program: with a text file, segregate each token under categories such as keyword, characters, digits, integers. So I created a hashMap where Each key in the hash map have to be segregated in a way that i.e Keyword = [if, then,…
karina
  • 41
  • 7
-3
votes
1 answer

Counting and averaging words in sentences

I must use Python to print the number of words and mean length of words in each sentence of a text file. I cannot use NLTK or Regex for this assignment. The sentence in the file ends with a period, exclamation point, or question mark. A hyphen,…
Bala
  • 11
  • 2
-3
votes
1 answer

Regular Expression excluding of expression like int keyword

I've written a mini scanner for a compiler and it reads from a file and I want to write instead of variables, id and for keywords do nothing (a group of words that want exclude from my variable form expressions), below line get my variable, how can…
user3356317
  • 15
  • 3
-4
votes
1 answer

How can I find all language specific tokens/lexicons for javascript?

I need basically all language-specific tokens, symbols for javascript. Basically, all the keywords, all the identifiers, all the punctuators, all the logical operation symbols and etc. Where can I actually find it?
Berkay Berabi
  • 1,933
  • 1
  • 10
  • 26
1 2 3
…
56
57