3

I have multiple microservices written in node and microservices are installed into the docker container and we are using Mesos+Marathon for clustering.

How can I aggregate the logs of all the containers(microservices) on different instance.?

Rico
  • 58,485
  • 12
  • 111
  • 141
Balwant Singh
  • 103
  • 6

3 Answers3

3

We're using Docker+Mesos as well and are shipping all logs to a log analytics service (it's the service the company I work for offers, http://logz.io). There a couple of ways to achieve that:

  • Have a log shipper agent within each docker - an agent like rsyslog, nxlog, logstash, logstash-forwarder - that agent would ship data to a central logging solution
  • Create a Docker that is running the shipper agent (like rsyslog, nxlog, logstash, logstash-forwarder) and that agent reads logs from all Dockers on each machine and ships them to a central location - this is the path we're taking
Kirill Bulygin
  • 3,658
  • 1
  • 17
  • 23
Tomer Levy
  • 357
  • 1
  • 4
  • Thanks for reposne but the problem is the configuration, we are using winston to collect the logs and dumping into the log file so what should be the configuration in case of winston...? – Balwant Singh Jul 30 '15 at 05:24
  • In this case I would use Winstons File Transport (possibly to a dedicated shared Docker volume container for data persistency) and then pick up the log files with one of the above mentioned shippers for delivery to the ELK provided either by Logz.io or on-premise. – Tomer Levy Aug 01 '15 at 12:46
2

This is a broad question but I suggest you setup an Elastic Search, Logstash, Kibana stack (ELK)

https://www.elastic.co/products/elasticsearch

https://www.elastic.co/products/logstash

https://www.elastic.co/products/kibana

Then on each one of your containers you can run the logstash forwarder/shipper to send logs to your logstash frontend.

Logs get stored in Elastic Search and then you search for them using Kibana or the Elastic Search API

Hope it helps.

Rico
  • 58,485
  • 12
  • 111
  • 141
1

I am also doing some Docker + Mesos + Marathon work so, I guess, I am going to face same doubt that you have.

I don't know if there's any native solution yet. But there's a blog by the folks at elastic.io on how they went about solving this issue.

Here's the link - Log aggregation for Docker containers in Mesos / Marathon cluster

Dharmit
  • 5,498
  • 2
  • 27
  • 30
  • Your article link is broken. Use http://www.elastic.io/log-aggregation-for-docker-containers-in-mesos-marathon-cluster/ – David Morales Jul 20 '16 at 21:33