3

The question is rather simple, I've written a lexer, using boost::spirit, however I can't seem to find a way, to generate an EOF token. - So how would one go about doing this?

Skeen
  • 4,614
  • 5
  • 41
  • 67

1 Answers1

5

What is an EOF token?

Historically, some platforms have associated special 'EOF' (ascii 26, e.g.) characters with text files. Like the use of 0x15 as newline character, such uses are now largely defunct. The end of a file is better defined as the absense of further input, in other words: it is a stream state, not a character.

The token iterators Spirit Lex signal 'EOF' by returning the end iterator.

Both the tokenizer API (lex::tokenize(...)) as well as Spirit Qi understand this behaviour (by exiting the tokenizing loop (lex) and/or by making the qi::eoi parser succeed match).

E.g. if you need to assert that parsing reached the end of the input, you'd just say

 myrule = subrule1 >> subrule2 > qi::eoi;

Or if you want to assert the presence of something (say, a closing ;) unless at end of input:

 myrule = subrule1 >> subrule2 >> (qi::eoi | ';');

Did I miss something about the question that isn't addressed like this?

sehe
  • 374,641
  • 47
  • 450
  • 633
  • Well it seems like all I ever needed was the `qi::eoi`! However, if I use this in my grammar, then can I assume that all input has been consumed, if the `tokenize_and_parse` function returns true? - That is, can I assume that `(begin == end)`, where begin and end are my input iterators. – Skeen Sep 04 '13 at 08:40
  • 1
    @Skeen If you look at [the header file](http://www.boost.org/boost/spirit/home/qi/auxiliary/eoi.hpp) you can see that all `eoi` does is use the skipper and then simply return whether `first` and `last` are equal (`qi::skip_over(first, last, skipper);return first == last;`). So yes I think you can assume that, if they are not equal the parsing would fail. – llonesmiz Sep 04 '13 at 09:37
  • $cv_and_he: Oh wow, I really should have checked the header file, before asking stupid questions.. Anyway thanks :) – Skeen Sep 04 '13 at 09:47
  • @sehe: Shouldn't there be a `>>` instead of a `>`, in your first code example? – Skeen Sep 04 '13 at 10:36
  • 1
    @Skeen I chose `>` which is the ['expectation operator'](http://www.boost.org/doc/libs/1_54_0/libs/spirit/doc/html/spirit/qi/reference/operator/expect.html) (as opposed to [`>>` the 'sequence operator`](http://www.boost.org/doc/libs/1_54_0/libs/spirit/doc/html/spirit/qi/reference/operator/sequence.html)) because I imagined a requirement _`if you need to assert that parsing reached the end`_. Expectation points fail with an `expectation_failure` if they don't match (instead of 'soft-failing' which allows the calling parser to back-track or attempt alternatives) – sehe Sep 04 '13 at 10:39