I am creating a virtual environment initially using docker containers locally on my Ubuntu machine. Eventually, I will take it to the HPC environment and run it there in much larger scales. The scenario is as follows: I am running a small program inside the docker container; in order to measure the performance, I should create multiple samples of containers with different resources (cpu usage, memory, cache) and let them run to see the performance. I also have installed ganglia monitoring system for instrumenting the environment. The goal is to write the data collected by the ganglia (or Prometheus) to a time series database such as influxdb and then process and parse that data for machine learning model training.
Since I will run the environment on the HPC, I would like to run them with singularity module. For the above scenario, I am thinking of creating a singularity network (similar to docker network), but singularity doesn't seem to have a network option. My containers are including my program, influxdb, and ganglia. In the future I will run much heavier programs. I need to get them to communicate well!
What I did? I ran docker containers of influxdb and my program. Still, figuring out the monitoring part. Also I can wrote to influxdb using a python script. I still need to connect the database to ganglia monitoring system(container image? or natively?).
For ganglia, since ganglia doesn't have an API or CLI to interact, I am looking to pull and run the ganglia container from docker hub. I was wondering if there was an alternative solution for the monitoring part. Please let me know if you had an idea to address the containerized monitoring agent.
For influxdb CLI, I haven't been able to make it work for Version 2.x. However, I have set up the API, authentication, and my script for writing to influxdb works fine. Do I need to setup the influxdb 2.x CLI at this point? How do I do that? I would appreciate it if you would comment on my scenario and how to set up the CLI.
Thank you!
