Logstash pattern matching more than it should

Question

I have a log line (multiline match) that looks like the following:

3574874    14/Jul/2016 20:42:37 +0000  ERROR [http-bio-0.0.0.0-8443-exec-128] error_jsp            _jspService    > could not lock: [com.myCompany.myProject.bean.scheduling.Area#6306]; SQL [/* UPGRADE lock com.myCompany.myProject.bean.scheduling.Area */ select Area_id from Area_ID where Area_id =? and Area_Version =? for update]; nested exception is org.hibernate.exception.LockAcquisitionException: could not lock: [com.myCompany.myProject.bean.scheduling.Area#6306]
org.springframework.dao.CannotAcquireLockException: could not lock: [com.myCompany.myProject.bean.scheduling.MyClass#6306]; SQL [/* UPGRADE lock com.myCompany.myProject.bean.scheduling.MyClass */ select Area_ID from Area where Area =? and Area =? for update]; nested exception is org.hibernate.exception.LockAcquisitionException: could not lock: [com.myCompany.myProject.bean.scheduling.Area#6306]
    at org.springframework.orm.hibernate3.SessionFactoryUtils.convertHibernateAccessException(SessionFactoryUtils.java:639)
    at org.springframework.orm.hibernate3.HibernateExceptionTranslator.convertHibernateAccessException(HibernateExceptionTranslator.java:89)
    at org.springframework.orm.hibernate3.HibernateExceptionTranslator.translateExceptionIfPossible(HibernateExceptionTranslator.java:68)
    at org.springframework.dao.support.ChainedPersistenceExceptionTranslator.translateExceptionIfPossible(ChainedPersistenceExceptionTranslator.java:58)
    at org.springframework.dao.support.DataAccessUtils.translateIfNecessary(DataAccessUtils.java:213)
    at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:163)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:633)
    at com.myCompany.myProject.dal.hibernate.Impl$$EnhancerBySpringCGLIB$$535be625.lock(<generated>)
    at com.myCompany.myProject.scheduling.Area.AreaClose(MyClass.java:1265)
    at com.myCompany.myProject.scheduling.Area.handleProviderDisconnect(MyClass.java:1190)
    at com.myCompany.myProject.scheduling.Area$$FastClassBySpringCGLIB$$220c3d67.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:700)

My pattern file lays things out like this:

    # AW Tomcat formatting
    TOMCAT_DATE %{MONTHDAY}[./-]%{MONTH}[./-]%{YEAR}
    TOMCAT_TS %{BASE10NUM}
    #%{BASE10NUM}
    TOMCAT_TIME %{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9])
    TOMCAT_THREAD \[(.+?)\]
    TOMCAT_CLASS [A-Za-z0-9]+
    TOMCAT_CLASS_METHOD %{NOTSPACE}
    TOMCAT_TIMESTAMP %{TOMCAT_DATE}[\s]+%{TOMCAT_TIME}
    TOMCAT_MESSAGE .+
    TOMCAT_IP %{IP}
    TOMCAT_DATA %{NOTSPACE}

    TOMCAT_LOG (?:%{TOMCAT_TS:log_ts})[\s]+(?:%{TOMCAT_TIMESTAMP:log_timestamp})[\s]+%{INT}[\s]+(?:%{LOGLEVEL:log_level})[\s]+(?:%{TOMCAT_THREAD:thread})[\s]+(?:%{TOMCAT_CLASS:class_name})[\s]$

When I look at this in logstash it seems like the code for thread is matching a lot more than just the thread... it's matching all the way to [com.myCompany.myProject.bean.scheduling.MyClass#6306];

When I take JUST the thread and the regex for space after and toss it on regexr or regex101 I can't recreate this using:

\[(.+?)\][\s]+

Does anyone have insight on why the regex is working everywhere but logstash? Also, this works on about 99.5% of my incoming tomcat logs as well....

score 1 · Accepted Answer · answered Jul 15 '16 at 08:20

1

You have a + in your log. You need to match it with \+ or make it optional with \+?:

TOMCAT_LOG (?:%{TOMCAT_TS:log_ts})\s+(?:%{TOMCAT_TIMESTAMP:log_timestamp})\s+\+?%{INT}\s+(?:%{LOGLEVEL:log_level})\s+(?:%{TOMCAT_THREAD:thread})\s+(?:%{TOMCAT_CLASS:class_name})\s*$
                                                                             ^^^

answered Jul 15 '16 at 08:20

Wiktor Stribiżew

607,720
39
448
563

That's 2 in a row that you've helped me out with, so first off, thank you so much. Secondly, do you have any resources that would help me in troubleshooting my regex issues that arise with grok / logstash in general? Thanks, – A_Elric Jul 18 '16 at 18:55
Oniguruma regex is actually possible to test at http://rubular.com. http://grokdebug.herokuapp.com/ is fine to test built-in and custom patterns. I think that is quite enough, just you should have some knowledge of regex to be in full control of what the pattern does. – Wiktor Stribiżew Jul 18 '16 at 20:03

Logstash pattern matching more than it should

1 Answers1