Stack Overflow
Stack Overflow

Questions tagged [lexer]

A program converting a sequence of characters into a sequence of tokens

A lexer is a program whose purpose is the conversion of a sequence of characters into a sequence of tokens. It is also often referred to as a scanner. A lexer often exists as a single function, which is called by a parser or another function.

1050 questions
-2
votes
1 answer

I'm using boost::spirit to write a parser lexer

I'm using boost::spirit to write a parser, lexer here is what i want to do. i want to put functions and classes into data structures with the variables they use. so i want to know what would be the best way to do this. and what parts of…
zeitue
  • 1,674
  • 2
  • 20
  • 45
-2
votes
0 answers

Creation of a programming language

I have several classes for a Programming Language. Class Lexer: from rply import LexerGenerator class Lexer(): def __init__(self): self.lexer = LexerGenerator() def _add_token(self): self.lexer.add('PRINT', r'print') …
Rodion
  • 1
-2
votes
1 answer

NoneType object is not callable (Making lexer)

My Code: import re from collections import namedtuple class TokenDef(namedtuple("TokenDef", ("name", "pattern", "value_filter"))): def __repr_(self): return "TokenType." + self.name class TokenType(object): _defs = [ …
MarulForFlask
  • 1
  • 4
-2
votes
1 answer

Regex for parameters

Could anybody explain me how a regex should look like if I want to match from: section(test) only the string test? Condition is that the string section has to be in front of (test) Thanks for any help
CrustyCheese
  • 49
  • 1
  • 5
-2
votes
1 answer

OCaml file reader

i am defining a language in ocaml with ocamllex and ocamlyacc. The input for this language is a stream of ints from a file, for example: 1 2 3
kkk
  • 33
  • 5
-2
votes
1 answer

OCaml Parsing String into String*Int list

type unit_t = (string * int) list How would I go about creating an expression to parse a string into this form? Do I need to implement code in a parser.mly format in order to handle this?
Alphonse Koffi
  • 1
-2
votes
1 answer

Ruby attrib_reader for lexer

Ive got a token class, and a lexer class. I want to have a get_token_value method in the lexer class and I can't seem to get it. The Token and Lexer classes are below with only relevant methods. This is the code used to try to get the values of all…
ol bassud
  • 1
  • 1
-2
votes
1 answer

How to make a parser after building parser.java file using ANTLR4?

I am planning to build a parser, that parse only particular format as below. Dec 17 14:00:00 IP firewall,info FFFW forward: in: out:sfp-sfpplus1.vlan113, proto TCP (ACK,PSH), IP:PORT->IP:PORT, NAT (IP:PORT->IP:PORT)->IP:PORT, len…
Avid Learner
  • 67
  • 1
  • 6
-3
votes
1 answer

which source file in rust language source refers_to/parse language keywords

I want to change keyword spellings in Rust language. For example, for -> phur, so where should I look in rustc source code? Thanks in advance.
Vnti Jiro
  • 3
  • 2
-3
votes
1 answer

Writing a lexer : lexer doesnt find "

I am writing a small programming language that has only one keyword which is write. I haven't done a parser but lexer. It works fine till it gets to " token. main.cpp: #include #include #include #include…
Gameplayer Xd
  • 9
  • 6
-3
votes
2 answers

How to simulate a symbol table in Common Lisp

i am learning Common LISP at the moment and i encountered a huge roadblock. I've had an assignment that we were to learn how to create a parser in Common LISP. I have managed to implement everything from the grammar rules to the lexer with alot of…
Joo223
  • 1
  • 1
-3
votes
1 answer

C++ array and find functions

I am writing a lexer and I am using an array to strore the keywords and reserved words: string keywords[20] = { "function", "if", "while", "halt", }; I am trying to use : bool isKeyword(string s) { return find( keywords.begin(),…
user3599420
  • 59
  • 8
-3
votes
1 answer

logical, Finite State Machine

I am trying to draw out a finite state machine (start, next state etc.) how can I draw this using only 7 states? I wrote out what the table looks like, this inputing characters into tokens Table: e.g. if the user types in a one a T for true should…
Homer Homer
  • 1
  • 1
-4
votes
1 answer

How can I find all language specific tokens/lexicons for javascript?

I need basically all language-specific tokens, symbols for javascript. Basically, all the keywords, all the identifiers, all the punctuators, all the logical operation symbols and etc. Where can I actually find it?
Berkay Berabi
  • 1,933
  • 1
  • 10
  • 26
-5
votes
1 answer

Can we consider preprocessor directives as token?

for the following code can we consider '#define' as token? #define int char main(){ int i=65; // i is initialized printf("sizeof(i)=%d",sizeof(i)); }
Dash_25
  • 1
  • 1
  • 6
1 2 3
…
69
70