Distributing tracing and elastic stack visualisation

Question

2019-06-03 10:45:00.051  INFO [currency-exchange,411a0496b048bcf4,8d40fcfea92613ad,true] 45648 --- [x-Controller-10] logger                                   : inside exchange

This is the log format in my console. I am using spring cloud stream to transport my log from application to logstash.This is the format for log parsing in logstash

grok {
              match => { "message" => "%{TIMESTAMP_ISO8601:timestamp}\s+%{LOGLEVEL:severity}\s+\[%{DATA:service},%{DATA:trace},%{DATA:span},%{DATA:exportable}\]\s+%{DATA:pid}\s+---\s+\[%{DATA:thread}\]\s+%{DATA:class}\s+:\s+%{GREEDYDATA:rest}" }
       }

This is my logstash.conf

input { kafka { topics => ['zipkin'] } } filter { # pattern matching logback pattern grok { match => { "message" => "%{TIMESTAMP_ISO8601:timestamp}\s+%{LOGLEVEL:severity}\s+\[%{DATA:service},%{DATA:trace},%{DATA:span},%{DATA:exportable}\]\s+%{DATA:pid}\s+---\s+\[%{DATA:thread}\]\s+%{DATA:class}\s+:\s+%{GREEDYDATA:rest}" } } } output { elasticsearch {hosts => ['localhost:9200'] index => 'logging'} stdout {} }

and this is my output in log-stash console . which is parsing exception

{ "message" => "[{\"traceId\":\"411a0496b048bcf4\",\"parentId\":\"8d40fcfea92613ad\",\"id\":\"f14c1c332d2ef077\",\"kind\":\"CLIENT\",\"name\":\"get\",\"timestamp\":1559538900053889,\"duration\":16783,\"localEndpoint\":{\"serviceName\":\"currency-exchange\",\"ipv4\":\"10.8.0.7\"},\"tags\":{\"http.method\":\"GET\",\"http.path\":\"/convert/1/to/4\"}},{\"traceId\":\"411a0496b048bcf4\",\"parentId\":\"411a0496b048bcf4\",\"id\":\"8d40fcfea92613ad\",\"name\":\"hystrix\",\"timestamp\":1559538900050039,\"duration\":34500,\"localEndpoint\":{\"serviceName\":\"currency-exchange\",\"ipv4\":\"10.8.0.7\"}},{\"traceId\":\"411a0496b048bcf4\",\"id\":\"411a0496b048bcf4\",\"kind\":\"SERVER\",\"name\":\"get /convert\",\"timestamp\":1559538900041446,\"duration\":44670,\"localEndpoint\":{\"serviceName\":\"currency-exchange\",\"ipv4\":\"10.8.0.7\"},\"remoteEndpoint\":{\"ipv6\":\"::1\",\"port\":62200},\"tags\":{\"http.method\":\"GET\",\"http.path\":\"/convert\",\"mvc.controller.class\":\"Controller\",\"mvc.controller.method\":\"convert\"}}]", "@timestamp" => 2019-06-03T05:15:00.296Z, "@version" => "1", "tags" => [ [0] "_grokparsefailure" ] }

score 0 · Answer 1 · answered May 31 '19 at 23:42

When I use the Grok Debugger that is built into Kibana (under Dev Tools) I get the following result from your sample log and grok pattern:

{
  "severity": "DEBUG",
  "rest": "GET \"/convert/4/to/5\", parameters={}",
  "pid": "35973",
  "thread": "nio-9090-exec-1",
  "trace": "62132b44a444425e",
  "exportable": "true",
  "service": "currency-conversion",
  "class": "o.s.web.servlet.DispatcherServlet",
  "timestamp": "2019-05-31 05:31:42.667",
  "span": "62132b44a444425e"
}

That looks correct to me. So what is the missing part?

Also the logging output you are showing contains "ipv4":"192.168.xx.xxx"},"remoteEndpoint": {"ipv6":"::1","port":55394},"tags": ..., which is not in the sample log. Where is that coming from?

I dont know from where these are comming these are not present in my console log.I share my logstash conf and output on logstash console. — jss, Jun 03 '19 at 05:32
But this is also not part of the logstash.conf. Could this come in with these attributes from another source? — xeraa, Jun 03 '19 at 15:29

Distributing tracing and elastic stack visualisation

1 Answers1