3

I am struggling with discovering and monitoring dynamic (i.e. moving) containers across multiple physical servers and affiliating the results with the container service, not the server on which the agent that runs it is running.

I have 2 servers: A and B; I have one container: C. C can run on either A or B, my orchestration engine (Swarm, Kubernetes, Rancher, etc.) is responsible for ensuring it runs in at least one place.

I have a zabbix agent on A and B, so I can monitor CPU, filesystem, memory, all the usual stuff.

I want to monitor 2 things:

  1. Availability of C. I don't care where, but I want to know it is running.
  2. Checks of processes within C. I have a script that checks status.

How do I configure Zabbix and the agents so it reports on both the state of C and its processes, independent of where it is running?

My assumptions are:

  1. I treat C as a host, rather than A or B.
  2. I assign discovery of processes within C and their state as Items of C.

My questions are:

  1. How do I set C to be auto-discovered as a Host from A or B? I can use my own script or https://github.com/monitoringartist/Zabbix-Docker-Monitoring with filters to pick up only the containers I care about.
  2. How do I set the process check to run on both A and B agents?

In short, how do I set it up to run container auto-discovery as host, and then process check for each of those, such that they run on every server with an agent in a given group/pattern, but then affiliate the output to C container rather than the A or B server on which the agent is running?

EDIT: Thanks to first responder, I get the "meta-host" idea. But that creates new issues:

  1. How do I create a C "meta-host" if I have to add an IP when I create it? It could be the IP of A or B!
  2. How do I automatically create C by discovery, based on rules?
  3. Since Zabbix reaches out to each of A and B to say, "run these tests", how does it know whether to ask A to run them or B to run them?

This would be much easier if I could just say, "run discovery on all hosts in the 'Docker' group", which would discover all C (and D, etc.) containers and add them as hosts. And then also say, "run processes check script on all containers that were auto-discovered", perhaps by knowing which agent currently has access to the container (i.e. where it is running now).

I am starting to get the feel more and more that Zabbix is great for monitoring applications that are tied to a particular server, less so at monitoring apps that move around? Or am I misunderstanding it?

deitch
  • 585
  • 1
  • 4
  • 15

2 Answers2

3

Disclaimer: I'm the author of https://github.com/monitoringartist/Zabbix-Docker-Monitoring

Set up standard Zabbix-Docker-Monitoring on A and B.

Edit Docker template - filter discovered container as you need + remove any trigger prototypes.

Create new calculated items, which will aggregate each C related item from A and B into new C calculated items (you can create C "metahost" in the Zabbix) - set up new triggers on top of these new C metrics.

Update: Use calculated items for aggregation - for example aggregate sum(docker.up[cid]) from A and B - then trigger condition for "Container cid is not running" will be sum(docker.up[cid])<1. Pls read Zabbix doc for correct syntax.

LLD discovers where is your container running atm and it will update items/triggers accordingly. If you want to eliminate any false alerts, don't forget to tune timings of LLD/triggers.

Jan Garaj
  • 879
  • 1
  • 7
  • 15
  • yep, we like your plugin, we have started using it. But by default, all of the metrics are affiliated with the host in which the agent runs. Some of it makes a lot of sense, but others are hard. My missing info is how I would do the last paragraph: how do I aggregate the C-related items from A and B into new C? And similarly, if I have my own script to check processes in C, how do I aggregate them? – deitch Feb 03 '16 at 18:30
  • I know you are helping, so thank you. I still am confused. I get that putting C as a host moves it beyond the A or B, and thus doesn't matter where it runs. But I have 2 missing pieces: how do I add a discovered container C to be added as a host? how do I add processes returned by a script run on the host as items? – deitch Feb 04 '16 at 11:23
  • Editing question above. – deitch Feb 04 '16 at 12:26
0

Since Zabbix version 5 a new Zabbix agent 2 is available. It adds docker monitoring support natively in Zabbix. This agent is more or less a drop in replacement for the old agent. When installed add the user zabbix to the docker group on the monitored system to make the agent is able to access the docker server. Test docker availability to the agent with zabbix_get. This has to be done from that server that is allowed access from. Typically the zabbix server or a proxy.

$ zabbix_get -s some.host.tld -k docker.info

This should return a json output with docker.info data.

After a successful test head over to your configuration on the Zabbix server. You have two options to add the discovery - either by adding a template to your host or adding a template to one of your own templates (adding docker discover to all hosts that are using your template). Let's just talk about adding the template to a host: Open the configuration of your host. Add the Docker by Zabbix agent 2 Template in Templates/Applications to your host. If the template is not installed on your Zabbix server, head over to the link and download the raw .yaml file and import the template. The template might not be available if you have an older Zabbix server and upgraded to Version 5 or later. This is, because Zabbix will never upgrade your templates.

After you added the template, new items will be collected. Check Latest Data for your host.

itsafire
  • 468
  • 3
  • 15