0

I've got a lot of applications, each one have her own log. I'm typically in a distributed logs problematic.

To resolve this problematic, I'm using :

  • Brave / Spring Sleuth to generate trace and span ids
  • Elastic/Logstash/Kibana to centralize and search in my logs

This way, I'm able to see all the application logs in the order they where written (by their timestamp). But the fact is that I can't trust the timestamp written in the logs. All the server's clocks are not synchronized.

So I need something which will help me to sort all my logs.

I've thought at generating a number at each span :

app1 : sequenceId = {}
     app2 : sequenceId = {}
     app3 : sequenceId = {}
             app4 : sequenceId = {}
app5 : sequenceId = {}
     app6 : sequenceId = {}
     app7 : sequenceId = {}
             app8 : sequenceId = {}

which would expect something which will order the logs in this order : app1, app2, app3, app4, app5, app6, app7, app8

Does someone knows a technic which adresses this problematic ?

Akah
  • 1,890
  • 20
  • 28

1 Answers1

1

We have an issue for that in Sleuth (https://github.com/spring-cloud/spring-cloud-sleuth/issues/275). The easiest thing would be to add a tag / entry in the log that would increment with each child span. You can reuse the Sleuth's internals in such a way that you override the DefaultTracer's behaviour to add an additional tag on child span creation. Then in the Slf4jLogger you would reuse that tag to set an additional element in the MDC context

Marcin Grzejszczak
  • 10,624
  • 1
  • 16
  • 32
  • Thank you for your answer. Yes, I'm already passing additional tags for other needs (user-auth-id for example). But I need app2 and app6 to have different values, and app5 > app2 for example. I have difficulties to find a way assuring consistance (Other than generating a list of numbers, growing at each span). – Akah Dec 18 '17 at 12:37
  • That was our initial approach to but we didn't give it a lot of thought though – Marcin Grzejszczak Dec 18 '17 at 12:39