I am researching on how to scrape application level metrics as well as the K8s host cluster metrics and forward it to a gateway collector, all using OpenTelemetry. I am referring to the OTel's Github page - https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver.
There is some confusion on which receiver to use to achieve the above use-case and how forward the metrics/logs from this cluster to another otel collector.
Any suggestions here is much appreciated. TIA.
Capturing Kubernetes metrics using the kubeletstats and hostmetrics receivers, needs to have the Collector deployed in Daemonset mode. Capturing Kubernetes events, and cluster level data is done using the k8sevents and k8scluster receivers with the Collector in Deployment mode. See the respective GitHub readme for each receiver to learn more on their specific settings.
I recommend you use the presets within the Collector Helm chart to install all of this. If you can't use Helm in your environment to install, I would still use it with helm template ... to generate the manifests so you can understand all the RBAC and service implications to get it all configured correctly.
From a telemetry pipeline perspective, It's a good idea to also enable the k8sattributes processor for metrics/traces/logs on your Daemonset Collectors and have all applications send their telemetry through the local Daemonset collector.
