I'm running a test application and Jmeter on one cluster and Prometheus in a separate Kubernetes cluster, all created on the same bare metal. Through LB, I can access my application from company intranet as "host ip:8080" and access prometheus as "host ip:9090". My Jmeter is using the prometheus listener and runs in a pod. It is pushing the metrics to localhost:9270.
I have edited values.yaml (helm prometheus values) to add jmeter job with target as localhost:9270 but the target always showing down. I'm not sure what/how prometheus from the other cluster can scrape the jmeter metrics from the other cluster. Looking for some guidance.
EDIT: Some more context: In the remote server, I created prometheus cluster like this: sudo k3d cluster create prometheus -p "9090:80@loadbalancer" --k3s-arg "--no-deploy=traefik@server:" and I can access it on my local with the server ip and port 9090. The other cluster in the same server where my application and jmeter run, I created the cluster like this sudo k3d cluster create myapp -p "8080:80@loadbalancer" --k3s-arg "--no-deploy=traefik@server:". I can access my application with server port:8080.
Kubectl get pods -A: (Please see the job-executor-service pod in jes-keptn namespace. This pod creates a new pod in run time that executes jmeter and when test finishes that run time pod changes to Completed status. When that jmeter pod runs, I have logged into it and seen the Jmeter metrics at localhost:9270 inside the pod)
Kubectl get svc -nkeptn (from application/jmeter cluster)
This is my other cluster running promethus:
