Software Stack
| Software | Version |
|---|---|
| OpenShift | 4.4 |
| Fluent-Bit | 1.7.0-rc4 |
| ElasticCloudStack | 7.10.2 |
I have a DaemonSet for fluent-bit pods to read OpenShift logs from /var/log/containers on the worker nodes.
Logs are coming to Elastic and viewable on Kibana.
Everything work well until we have to parse a Java stacktrace on the logs. Each line is sent a separate log entry.
How can I ensure that the stacktrace is read as one?
I have the following CRI-O parser:
[PARSER]
# http://rubular.com/r/tjUt3Awgg4
Name cri
Format regex
Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<log>.*)$
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L%z
and the following Merge_Parser for Spring Boot log entry
[PARSER]
Name springboot
Format regex
Regex /^(?<date>[0-9]+-[0-9]+-[0-9]+\s+[0-9]+:[0-9]+:[0-9]+.[0-9]+)\s+(?<log_level>[Aa]lert|ALERT|[Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nfo|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rr?(?:or)?|ERR?(?:OR)?|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?)\s+(?<pid>[0-9]+)\s+---\s+\[(?<thread>.*)\]\s+(?<class_name>.*)\s+:\s+(?<message>.*)$/
Time_Key time
Time_Format %Y-%m-%
The CRI-O log entry is as below for the log entries with and without excpetion
2021-02-09T11:19:04.815514933+00:00 stdout F 2021-02-09 11:19:04.814 DEBUG 1 --- [nio-8080-exec-8] x.x.x.service.DocumentService : retrieved: []
2021-02-09T11:19:04.817387066+00:00 stdout F 2021-02-09 11:19:04.816 ERROR 1 --- [nio-8080-exec-4] x.x.x.exceptions.RestExceptionHandler : 422 Status Code - EntityNotFoundException - XXXXXXXXXXXXXXXXXXXXXXXXX
2021-02-09T11:19:04.817387066+00:00 stdout F
2021-02-09T11:19:04.817387066+00:00 stdout F xxx.xxx.microservices.exceptions.EntityNotFoundException: XXXXXXXXXXXXXXXXXXXXXXXXX
2021-02-09T11:19:04.817387066+00:00 stdout F at xxx.xxx.microservices.service.XXXXXXX.lambda$getXXXData$3(XYXYXYXYX.java:139) ~[classes!/:na]
2021-02-09T11:19:04.817387066+00:00 stdout F at java.base/java.util.Optional.orElseThrow(Optional.java:408) ~[na:na]
2021-02-09T11:19:04.817387066+00:00 stdout F at xxx.xxx.microservices.rest.internal.XXXXXXX.getXXXX(XXXXXXXXXXX.java:101) ~[classes!/:na]
2021-02-09T11:19:04.817387066+00:00 stdout F at jdk.internal.reflect.GeneratedMethodAccessor854.invoke(Unknown Source) ~[na:na]
2021-02-09T11:19:04.817387066+00:00 stdout F at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
2021-02-09T11:19:04.817387066+00:00 stdout F at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
2021-02-09T11:19:04.817387066+00:00 stdout F at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190) ~[spring-web-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]
I have read this article (https://coralogix.com/log-analytics-blog/parsing-multiline-logs-the-complete-guide/) but it however only address the Spring Boot only before the cluster metadata is added.
