How do I figure out ANTLR grammar failure to parse input, special timestamp in logfile

Question

INPUT:

Mar 9 10:19:07 west info tmm1[17280]: 01870003:6:
/Common/mysaml.app/mysaml:Common:00000000: helloasdfasdf asdfadf vgnfg

GRAMMAR:

grammar scratch;
lines :       datestamp hostname level proc msgnum  module msgstring;
datestamp:    month day time;
//month :       MONTH;
day  :        INTEGER;
time :        INTEGER ':' INTEGER ':' INTEGER;
hostname :    STRING;
level :       ALPHA;
proc:         procname '[' procnum ']' ':';
procname :    STRING;
procnum :     INTEGER;
msgnum :      INTEGER ':' DIGIT':';
module :      '/' DOTSLASHSTRING ':' PARTITION ':' SESSID ':';
PARTITION:     STRING;
sessid :      HEX;
msgstring:      MSGSTRING;
DOTSLASHSTRING : [a-zA-Z./]+;
SESSID :      HEX;
INTEGER :     [0-9]+;
DIGIT:        [0-9];
STRING :      [a-zA-Z][a-zA-Z0-9]*;
HEX :         [a-f0-9]+;
//ALPHA:        [a-zA-Z]+;
ALPHA:         ('['|'(') .*? (']'|')');
MSGSTRING :   [a-zA-Z0-9':,_(). ]+ [\r];
 //         |   'Agent' MSGSTRING;
month : 'Jan' | 'Feb' | 'Mar' | 'Apr' | 'May' | 'Jun' | 'Jul' | 'Aug' | 'Sep' | 'Oct' | 'Nov' | 'Dec' ;
WS :          [ \t\r\n]+ -> skip;

PROBLEM: the parse tree shows that the month is populated properly, but the next item, day is not. In the parse tree, it shows day is set to the entire rest of the input. Don't see how this is possible.

Error from parser is:

line 1:4 mismatched input '9' expecting INTEGER

Answer 1

The parser (i.e. rules starting with lowercase letter) and lexer (uppercase first letter) behave in a slightly different way:

• The parser knows what token it expects and tries to match it (except when it has multiple alternatives - then it looks at the next tokens to see which alternative to choose)
• The lexer however knows nothing about the parser rules - it matches whatever it can match to the current input. When multiple lexer rules can match a prefix of the input:
  • It will match (and emit token for) the rule that matches the longest sequence
  • If multiple rules can match the same sequence, the rule earlier in the file (closer to the top) wins.

So your input would most likely be tokenized to^*:

MONTH          Mar
(WS)
SESSID         9  - SESSID matches and is higher up than INTEGER
(WS)
SESSID         10
':'            :
SESSID         19
':'            :
SESSID         07
(WS)
PARTITION      west  - same as STRING but higher up - STRING will never be matched
(WS)
PARTITION      info
(WS)
PARTITION      tmm1
ALPHA          [17280]  - matches longer sequence than just '[' in rule "proc"
':'            :
(WS)
SESSID         01870003
':'            :
SESSID         6
':'            :
(WS)
DOTSLASHSTRING /Common/mysaml.app/mysaml  - longer than just '/' in rule "module"
MSGSTRING      :Common:00000000: helloasdfasdf asdfadf vgnfg  - the rest can be matched to this rule

As you can see, these are quite different tokens than your parser expects.

The bottom line is, you have too much logic in your lexer rules, namely you tried to put semantic meanings into lexer. It's not suited to that task. If a single input sequence can mean different things (like 123 might be an integer number, hex number or session ID), that distinction needs to go into parser since it can only be decided based on context (where in the sentence it occurred), and not by the content of the 123 itself. Similarly, if [17280] can either be ALPHA (whatever that is) or an INTEGER in brackets, that decision needs to go into parser because it cannot be decided solely by looking at [17280] (it's now in lexer due to the ALPHA rule).

---

* The likely tokenization is based on the input from your screenshot which is all on one line, while the input in question itself is on two lines - not sure whether that is intentional or a result of line wrap.