I have several GCP projects, belonging to different people. I have all accesses on them. None of them have an Organization set.
I would like to monitor those projects (metrics from VM, logs etc) and display a centralized dashboard with relevant information and filters.
What I have currently is a GCP project with an Elastic stack installed and running (using this repo). My idea would be that all the projects send metrics continuously to the Elastic URL of my project and using Kibana I can display a health status and metrics of all the projects on a dashboard.
I found this tutorial using data-flows but I have trouble making it works. Here is what I did:
- Create a Pub/Sub topic, a sink and subscription on the project I want to monitor
- Create a VPC and a NAT in the project because I need to know what address to allow on my Elastic server.
- Create a firewall rule on my Elastic server to allow the NAT to access port 9200 and contact Elastic
- Create a data-flow on the project to monitor.
But when I launch the data-flow job, I get this error:
Error message from worker: java.lang.RuntimeException: org.apache.beam.sdk.util.UserCodeException: java.lang.IllegalArgumentException: Cannot get Elasticsearch version
Data-flow failing job
Is my idea correct ? If so Am I doing it the right way and why is it not working ? If not what would you suggest ?
Thank you !
