0

Logs get created in our context in several systems.

For example:

  1. Job12345 gets created via a web service.
  2. Then it gets processed in a message queue
  3. Then it gets stored in a database
  4. Later it gets send to a third party system.

In this example four or more systems create logs for Job12345.

Up to now we use simple file based logging.

My goal: A interface where I can query for all logs which contain "Job12345". It would be great if they are linked like hyperlinks in HTML.

I hope you understand what I want. Please ask if something is not clear.

How could this be solved?

guettli
  • 3,591
  • 17
  • 72
  • 123
  • Related paper: http://research.google.com/pubs/pub36356.html "Dapper, a Large-Scale Distributed Systems Tracing Infrastructure " – guettli Jul 18 '16 at 21:17

1 Answers1

1

Not sure if linking is possible, but I'm using Elasticsearch (to store the data and make it searchable) and Graylog (to query and actual search for the data; it provides a Webinterface and an API) for similar tasks.

I would recommend to have a look at booth tools at least; it's of great help to have all your logs in one place and be able to search through them.

gxx
  • 5,591
  • 2
  • 22
  • 42
  • Up to now we use the system default rsyslogd. If I understood you, then rsyslogd needs to store its logs into elasticsearch. Graylog connects to elasticsearch for searching. The admin uses the graylog web interface to search. Is this the way you use it? To have maximum useability we could configure the search results of graylog, to make some key-value mappings a hyperlink to an other graylog next search. – guettli Jul 21 '16 at 07:33