1

I am trying to setup ELK for a Java application. The tomcat logs are produced using log4j. To write a test a pattern, I am using Grok Debugger. But on the debugger it always shows

Compile ERROR

My log sample:

YYYY-MM-DD HH:MM:SS,SSS INFO : [so-me-uni-que-id] com.xx.xx.xx.xx.xx - log message here

My grok filter:

filter {   if [type] == "tomcat" {     grok {       match => { "message" => "%{TOMCATLOG}" }     }     date {       match => [ "timestamp", "yyyy-MM-dd HH:mm:ss,SSS" ]     } } }

My pattern:

TOMCATLOG %{TOMCAT_DATESTAMP:timestamp} \| %{LOGLEVEL:level} \| %{UNIQUEID:uniqueid}\| %{JAVACLASS:class} - %{JAVALOGMESSAGE:logmessage}
mathakoot
  • 1,692
  • 2
  • 14
  • 26
  • Why do you have escaped pipes in your pattern when your input has no pipes? – Alain Collins Jul 09 '15 at 20:37
  • I'm fairly new to this. So I read on some blog; somewhere but I cannot recollect the link. I suppose I don't truly understand writing patterns. – mathakoot Jul 09 '15 at 20:42
  • It's a big topic, but you might start here: http://svops.com/blog/introduction-to-logstash-grok-patterns/ – Alain Collins Jul 09 '15 at 21:33
  • I was afraid, I'd get some harsh criticism for such a basic question. But answers/comments like these are indeed encouraging. Thanks @AlainCollins. – mathakoot Jul 09 '15 at 21:55

2 Answers2

1

The basic issue is that your pattern doesn't match your input. Look at the beginning:

YYYY-MM-DD HH:MM:SS,SSS INFO : [so-me-uni-que-id]

%{TOMCAT_DATESTAMP:timestamp} \| %{LOGLEVEL:level} \| %{UNIQUEID:uniqueid}\|

Your pattern has escaped pipes ("|"), but the input doesn't use them.

I also don't see that TOMCAT_DATESTAMP is in the default patterns, but maybe it's buried somewhere.

Start at the left side, matching one piece at a time in the debugger.

%{TIMESTAMP_ISO8601} %{WORD:level} : \[%{GREEDYDATA:uniqueid}\]

Then keep working your way across, grabbing more stuff into your pattern. Note that literals (":" and the escaped "[") become part of your pattern.

Good luck!

Alain Collins
  • 16,268
  • 2
  • 32
  • 55
  • Between the time that you made comments on the questions, I dig up so more on the net and trial and error, the way you said: one piece at a time and I could do it. Keep up!! So I got this pattern working on the Grok debugger: Pattern: %{TOMCAT_DATESTAMP:timestamp} %{LOGLEVEL:level} : %{GREEDYDATA:coorelationid} %{JAVACLASS:class} - %{GREEDYDATA:logmessage} Custom pattern: TOMCAT_DATESTAMP %{YEAR:year}-%{MONTHNUM:month}-%{MONTHDAY:day} %{HOUR:hour}:?%{MINUTE:minute}(?::?%{SECOND:second}) – mathakoot Jul 10 '15 at 02:59
  • I see that you have used %{WORD:level} for the log level. Just for the information, there is an default logstash grok pattern. You can check it out [here](http://grokdebug.herokuapp.com/patterns#). – mathakoot Jul 10 '15 at 19:54
  • TMTOWTDI! With the ":" and "[" delimiters, I didn't need the added complexity. – Alain Collins Jul 10 '15 at 20:24
0

I also had problems with Tomcat. Also don't need to forget that %LOGLEVEL pattern doesn't contains all levels for Tomcat (CONFIG, FINE, FINER, FINEST). It could be

TOMCAT_LOGLEVEL ([A-a]lert|ALERT|[T|t]race|TRACE|[D|d]ebug|DEBUG|[N|n]otice|NOTICE|[I|i]nfo|INFO|[W|w]arn?(?:ing)?|WARN?(?:ING)?|[E|e]rr?(?:or)?|ERR?(?:OR)?|[C|c]rit?(?:ical)?|CRIT?(?:ICAL)?|[F|f]atal|FATAL|[S|s]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?|CONFIG|FINE|FINER|FINEST)

I added all grok patterns in one place (Nginx, Tomcat, Spring): https://gist.github.com/petrov9/4740c61459a5dcedcef2f27c7c2900fd

Hope it will save your time

Anton
  • 604
  • 2
  • 11
  • 22