2

Our team decided to switch to Prometheus monitoring. So I wonder how to setup highly available fault tolerant Prometheus installation. We have a bunch of small projects, running on AWS ECS, almost all services are containerized. So I have some questions.

Should we containerize the Prometheus?

That means to run 2 EC2 instances with one Prometheus container per instance and one NodeExporter per instance. And run highly available Alert Manager in the container with Wave Mesh per instance in separate instances.

Or just install Prometheus binary and other stuff on EC2 and forget about containerizing them?

Any ideas? Are some best practices exist for highly available Prometheus setup?

heartwilltell
  • 86
  • 1
  • 9

2 Answers2

2

Don't run node_exporter inside of a container as you'll greatly limit the number of metrics exposed.

There is also a HA guide in relation to Prometheus setups that may be of use to you.

Also this question would be better suited to the Prometheus user mailing list

brian-brazil
  • 31,678
  • 6
  • 93
  • 86
Conor
  • 3,279
  • 1
  • 21
  • 35
  • `Don't run node_exporter inside of a container as you'll greatly limit the number of metrics exposed` - Any explanation will be appreciated – heartwilltell Aug 03 '17 at 08:21
  • 1
    @heartwilltell The node exporter exports machine level metrics. Containerising the node exporter abstracts it away from the host system it's running on and thus limits the amount metrics available to you. This is further addressed in the README for the node exporter https://github.com/prometheus/node_exporter#using-docker This thread may also prove useful https://github.com/prometheus/node_exporter/issues/474 Hope that helps! – Conor Aug 03 '17 at 09:31
1

Running Prometheus inside a container works if you configure some additional options, especially for the node_exporter. The challenges of this approach relate to the fact that node_exporter gathers metrics from what it sees as the local machine - a container in this case - and we want it to gather metrics from the host instead. Prometheus offers options to override the filesystem mount points from which this data is gathered.

See "Step 2" in https://www.digitalocean.com/community/tutorials/how-to-install-prometheus-using-docker-on-ubuntu-14-04 for detailed instructions.

talonx
  • 1,762
  • 1
  • 15
  • 33