Ragel - how to return one token at a time

Question

I want to build a one-token-per-call ragel grammar / thing.

I'm relatively new to Ragel (but not new to compilers, etc).

I've written a grammar for a json-like notation (three levels deep). It emits C code.

My input comes in complete strings (no need to cross buffer boundaries).

I want to call my grammar with the input string, have the grammar return one token. Then call it again and have it return the next token and so on. Until end of string. Then, call again with a new string.

One would think that a state machine would be perfectly suited to this kind of behaviour, but I haven't yet been able to figure how to accomplish this in Ragel.

Answer 1

Your best bet is probably to call fbreak after each token, then call the machine again without re-initializing p or cs.

From the (Ragel 6.9) manual:

fbreak; – Advance p, save the target state to cs and immediately break out of the execute loop. This statement is useful in conjunction with the noend write option. Rather than process input until pe is arrived at, the fbreak statement can be used to stop processing from an action. After an fbreak statement the p variable will point to the next character in the input. The current state will be the target of the current transition. Note that fbreak causes the target state’s to-state actions to be skipped.

Note that you don't actually need the noend option. That option is for ignoring pe, which is probably not what you want to do in this case, since you want the parser to be able to detect the end of the string it's parsing.