How to deal with overlapping character groups in different tokens in an EBNF grammar?

Question

I'm using an LL(k) EBNF grammar to parse a character stream. I need three different types of tokens:

CHARACTERS

  letter = 'A'..'Z' + 'a'..'z' .
  digit = "0123456789" .
  messageChar = '\u0020'..'\u007e' - ' ' - '(' - ')' .

TOKENS

  num = ['-'] digit { digit } [ '.' digit { digit } ] .
  ident = letter { letter | digit | '_' } .
  message = messageChar { messageChar } .

The first two token declarations are fine, because they don't share any common characters.

However the third, message, is invalid because it's possible that some strings could be both num and message (such as "123"), and other strings could be both an ident and a message (such as "Hello"). Hence, the tokenizer can't differentiate correctly.

Another example is differentiating between integers and real numbers. Unless you require all real numbers to have at least one decimal place (meaning 1 would need to be encoded as 1.0, which isn't an option for me) then I can't get support in the grammar for the differences between these two numeric types. I've had to go for all values being expressed as reals and doing the checking after the point. That's fine, but sub-optimal. My real problem is with the message token. I can't find a workaround for that.

So the question is, can I do this with an LL(k) EBNF grammar? I'm using CoCo/R to generate the parser and scanner.

If I can't do it with LL(k) EBNF, then what other options might I look into?

EDIT This is the output I get from CoCo/R:

Coco/R (Apr 23, 2010)
Tokens double and message cannot be distinguished
Tokens ident and message cannot be distinguished
...
9 errors detected

Answer 1

Try this:

CHARACTERS

    letter = 'A'..'Z' + 'a'..'z' .
    digit = "0123456789" .
    messageChar = '\u0020'..'\u007e' - ' ' - '(' - ')'  .

TOKENS

    double = ['-'] digit { digit } [ '.' digit { digit } ] .
    ident = letter { letter | digit | '_' } .
    message = messageChar { messageChar } CONTEXT (")") .

Oh, I have to point out that '\u0020' is the unicode SPACE, which you are subsequently removing with "- ' '". Oh, and you can use CONTEXT (')') if you don't need more than one character lookahead. This does not work in your case seeing as all the tokens above can appear before a ')'.

FWIW: CONTEXT does not consume the enclosed sequence, you must still consume it in your production.

EDIT:

Ok, this seems to work. Really, I mean it this time :)

CHARACTERS
    letter = 'A'..'Z' + 'a'..'z' .
    digit = "0123456789" .
//    messageChar = '\u0020'..'\u007e' - ' ' - '(' - ')'  .

TOKENS

    double = ['-'] digit { digit } [ '.' digit { digit } ] .
    ident = letter { letter | digit | '_' } .
//    message = letter { messageChar } CONTEXT (')') .

// MessageText<out string m> = message               (. m = t.val; .)
// .

HearExpr<out HeardMessage message> =
    (.
        TimeSpan time; 
        Angle direction = Angle.NaN; 
        string messageText = ""; 
    .)
    "(hear" 
    TimeSpan<out time>
        ( "self" | AngleInDegrees<out direction> )
//         MessageText<out messageText>
    {
        ANY (. messageText += t.val; .)
    }
    ')'
    (. 
        message = new HeardMessage(time, direction, new Message(messageText)); 
    .)
    .

ANY will read character until it hits ')' or whitespace. I put it in a loop concatenating each value, but you may not want to do that. You may want to have it in a loop though so that it doesn't return "over" when it sees "over here", but "here". You can do a simple length check on messageText, or other validity checks such as adding t.val to a List and checking the count. Anything really. You can also do a test with a RegEx to make sure it complies with whatever pattern you need to check against.

EDIT (8 Apr 2011): Example using Coco/R with integers and reals

COMPILER Calculator
CHARACTERS
    digit       = "0123456789".

TOKENS
    intNumber    = ['-'] digit { digit } .
    realNumber   = ['-'] { digit } "." digit { digit } 
                         [("e" | "E") ["+" | "-"] digit {digit}] .

PRODUCTIONS
    Calculator  = { Expression "=" } .
    Expression  = Term { "+" Term | "-" Term }.
    Term        = Factor { "*" Factor | "/" Factor }.
    Factor      = intNumber | realNumber .

END Calculator.

EDIT (9 Apr 2011)

Factor<out double value>
    (. value = 0.0; .)
= 
    ( 
        intNumber 
        (. value = Convert.ToDouble(t.val); .)
        | 
        realNumber 
        (. value = Convert.ToDouble(t.val); .)
    ) 
    | "(" Expression<out value> ")"         
.

or

Factor<out double value>
    (. value = 0.0; .)
=
    ( intNumber | realNumber ) 
    (. value = Convert.ToDouble(t.val); .)
    | "(" Expression<out value> ")"
.

Answer 2

You may want to look into a PEG generator which has context sensitive tokenization.

http://en.wikipedia.org/wiki/Parsing_expression_grammar

I cannot think of a way you will get around this using COCO/R or similar, as each token needs to be unambiguous.

If messages were surrounded by quotes, or some other way of disambiguating then you would not have a problem. I really think PEG may be your answer, as it also has ordered choice (first match).

Also take a look at:

http://tinlizzie.org/ometa/

Answer 3

Despite the title, this all seems to relate to the scanner, not the parser. I haven't used CoCo/R, so I can't comment on it directly, but in a typical (e.g., lex/Flex) scanner, rules are considered in order, so the rule/pattern that's chosen is the first one that matches. Most scanners I've written include a '.' (i.e., match anything) as their last pattern, to display an error message if there's some input that doesn't match any other rule.