As a DevOps engineer, there are a few different ways to instrument a large C++ application for Prometheus monitoring:
Automate instrumentation: One approach is to automate the instrumentation process as much as possible, using tools like Prometheus' client libraries, or third-party tools like OpenCensus. This can help to ensure that the instrumentation is consistent across the entire application, and can reduce the burden on individual developers.
Use a Service Mesh: If your application is running in a containerized environment, consider using a service mesh like Istio or Linkerd that can automatically instrument your services for Prometheus monitoring.
Use a Middleware: Another approach is to use a middleware that can automatically instrument your SOAP and DWM services for Prometheus monitoring.
Use a Wrapper: Another approach is to use a wrapper that can automatically instrument your GUI and TENENT services for Prometheus monitoring.
Create a centralized instrumentation system: Create a centralized system that can be used to instrument all the services of the application. This can be done by creating a library that developers can include in their code.
Plan your workflow: Once you have identified the best approach for your application, create a detailed plan that outlines the steps that need to be taken to instrument the application, and identify any potential roadblocks or challenges that may arise.
Consider the migration: When migrating from Icinga to Prometheus, make sure you have a plan in place to ensure that all of your existing monitoring data is retained and that there is minimal disruption to your services.
Integrate Grafana: Integrate Grafana with Prometheus to give you a complete monitoring and visualization solution, this will help you to identify issues and anomalies in your application quickly.
Finally, it is important to communicate with the development team and to make sure they understand the benefits of instrumenting the application for Prometheus monitoring and how it will help them in their work.