-1

Recently I've decided to try to implement a very tiny language just to see what I can do. Over the past few hours I managed to write a lexical analyzer for my language that works quite nicely. So after reading theory and stuff I understand the next step is to write a parser. The job of the parser is still a little bit unclear to me, (I'm writing a tiny interpreted language for a start so things will be done directly in code) does it go through tokens, understand meaning, then do what the code says/generate code (for compiler)? OR I've read about abstract syntax tree's, is this something I'm supposed to construct with the parser? I'm not quite sure what an AST is even for really... So I suppose my question is, what's my next step after a lexer? What's the job of the parser? And last but not least what's the job of an abstract syntax tree?

Thanks for any help!

APott
  • 318
  • 4
  • 11
  • The best thing you can do is go get any book on compilers and read it. Getting some vague idea of what the pieces do and trying to put together a working tool won't be very effective and you won't learn what those guys building compilers spent 50 years figuring out. The Aho/Ullman/Sethi book on Compilers is *the* standard reference, and it will answer your questions directly. – Ira Baxter Aug 19 '13 at 02:29

1 Answers1

1

I'm not sure how you wrote you lexer, but a standard way to approach this is by writing the lexer with flex (previously lex) and to write the parser with yacc. The combination of the two makes it very easy to implement a wide variety of languages.

vy32
  • 28,461
  • 37
  • 122
  • 246
  • Yes, but that's not very challenging, the point of this is to challenge myself. My lexer is hand written! And assuming the job of a parser is to determine semantics of the tokens provided by the lexer I have written a basic working parser as well, I'm just not 100% sure on what the parser should do. – APott Aug 19 '13 at 01:30
  • @APott The challenge the way you've posed it is not going to be helpful. If you try to build a giant edifice on foundations of sand, you will likely fail but not for interesting reasons. You are starting by making assumptions and you're not even checking them, or even using the words properly. Skipping basic education before you start isn't a good idea. – Ira Baxter Aug 19 '13 at 02:32
  • 1
    The tokenizer breaks input into tokens, the lexer analyzes those tokens and identifies what kind of token they are. The parser looks at what those types are and what order they come in to determine what the program "means". The parser turns those types into an "abstract syntax tree", which is basically an intermediate representation of the program being compiled. Then the AST is transformed into the actual executable code. – Matt G Aug 19 '13 at 02:51
  • Oh wow I've been reading the wrong sources apparently... I was using the terms incorrectly, thanks for the comments guys! – APott Aug 19 '13 at 03:35
  • @Apott, it is plenty challenging to write a compiler or interpreter using flex & yacc. – vy32 Aug 19 '13 at 20:59
  • @vy32 I understand this I meant I wanted the challenge of writing the lexer and parser myself. – APott Aug 22 '13 at 23:45