Stack Overflow
Stack Overflow

Questions tagged [lexer]

A program converting a sequence of characters into a sequence of tokens

A lexer is a program whose purpose is the conversion of a sequence of characters into a sequence of tokens. It is also often referred to as a scanner. A lexer often exists as a single function, which is called by a parser or another function.

1050 questions
-1
votes
1 answer

Weird string new line thing

I am currently trying to right a Lexer in C# (probably stupid but it's just programming practice and it's the language I know best) and I'm getting a strange issue. I have split a separate file into tokens and am displaying it into the console.…
Shakier
  • 1
  • 1
-1
votes
1 answer

How to throw meaningful error on syntax or parsing error

I have a grammar for parsing jcl The jcl looks like below /*comment line //PR1290@ JOB (10),'ISPW COB SN900E' lexer and parser is working perfectly fine. Suppose instead of // if jcl starts from / currently lexer is throwing 1:0 token recognition…
kiranNswamy
  • 126
  • 9
-1
votes
1 answer

C preprocessor assigning value to multiple types

So im writing a Lexer and have come across a problem with my macro to generate a token (token is a struct with a value type (enum) and a value (union). #define MAKE_TOKEN(token, tokenType, valueType, tokValue) \ token.type=tokenType; \ …
kingstoncompsci
  • 163
  • 11
-1
votes
1 answer

Lexer for highlighting syntax language specified by the BNF grammar?

Good day. I wanted to implement the syntax highlighting for the language using a lexer for this. The essence is simple, we find a token, frame it with a set of symbols for a particular color. But the fact is that language tokens are described in…
jeudesprits
  • 45
  • 1
  • 4
-1
votes
1 answer

Error while compile a Parser rule

The code for parser that I had: funcall: ID LB exp? RB ; exp: funcall | INTLIT ; Then my assignment requires that I need to do like this for the "funcall": An invocation expression is a function call which starts with an identifier followed by…
Deen Thatcher
  • 3
  • 5
-1
votes
1 answer

Receiving "expected unqualified-id" when creating a function

Bear with me here, as I'm coming from a pure Java/Python background and have about a 10% knowledge in the basics of C++. I'm defining a tokenising class for a Lexer, and have already run into a problem, with the Eclipse compiler throwing: expected…
user5549921
-1
votes
1 answer

How print string in compiler bison/lexer?

I created a small compiler and need help to fix it. Code of my compiler: t.l: %{ #include #include #include #include "y.tab.h" %} %x DOUBLE_QUOTES %% [s|S][h|H][o|O][w|W] {return…
user5899862
  • 31
  • 1
  • 8
-1
votes
1 answer

Arrow functions as a block

With es6 we could (() => {})() Is there any way to write self-executing block in a way like we can do it in jsx? I mean, just { ["a"][0] }, witch should return "a". What's I found so far…
fckt
  • 571
  • 4
  • 12
-1
votes
1 answer

Which langages let you use fully customised lexems, including keywords and all symbol defined in their grammar?

I wish to code fully with Esperanto lexemes. That is, not ending up with a English/Esperanto mix up. Perligata is a good example of the kind of result I would like, but it use Latin where I want to use Esperanto. So Perl seems to be a valid answer…
psychoslave
  • 2,783
  • 3
  • 27
  • 44
-1
votes
3 answers

Parsing C# code to evaluate expressions (basically, implementing Intellisense)

I'm trying to evaluate C# code as it gets typed, think of it as if I'm trying to write an IDE. So a person types code, I want to find out what code did he just write: String x = ""; I want to now register that x is a type of String. And now…
halivingston
  • 3,727
  • 4
  • 29
  • 42
-1
votes
1 answer

string escape processing in flex

For a class on compilers I am building a lexer. I have completed the assignment, but am left with one point that I am not fully satisfied with. The language supports string literals with escape sequences, where a string literal is defined as a…
Bart van Ingen Schenau
  • 15,488
  • 4
  • 32
  • 41
-1
votes
2 answers

Antlr3 - Non Greedy Double Quoted String with Escaped Double Quote

The following Antlr3 Grammar file doesn't cater for escaped double quotes as part of the STRING lexer rule. Any ideas why? Expressions working: \"hello\" ref(\"hello\",\"hello\") Expressions NOT working: \"h\"e\"l\"l\"o\" ref(\"hello\",…
user550241
-1
votes
1 answer

Parser and abstract syntax tree

Recently I've decided to try to implement a very tiny language just to see what I can do. Over the past few hours I managed to write a lexical analyzer for my language that works quite nicely. So after reading theory and stuff I understand the…
APott
  • 318
  • 4
  • 11
-1
votes
1 answer

Is more performant to search a small array or use an if statement in c#?

Suppose I'm writing a parser and I need to check if the current token returned by Scanner::NextToken (for example) is one of a small set of values (say 5-10 items; few less or few more). In this small open source project…
gsscoder
  • 3,088
  • 4
  • 33
  • 49
-1
votes
1 answer

How to use ANTLR3 lexer only, without parser?

I need to use lexer only in ANTLR3, i don't need parser. How can i do it ? I use the following code (in main.c), which i found somehwere in the internet. #include #include int ANTLR3_CDECL main (int argc, char…
user1665530
  • 35
  • 5
1 2 3
…
69
70