Setup statsd-exporter as daemon on Kubernetes and send metrics to it from pods

Question

I want to setup statsd-exporter as DaemonSet on my Kubernetes cluster. It exposes a UDP port 9125, on which applications can send metrics using statsD client library. Prometheus crawlers can crawl this exporter for application or system metrics. I want to send the metrics to the UDP server running in the exporter on port 9125. I have two options:

1. Expose a service as ClusterIP for the DaemonSet and then configure the statsD clients to use that IP and Port for sending metrics

2. Make the statsd-exporter run on hostNetwork, and somehow enable the pods to send metrics to exporter running on the same node.

Somehow, option 2 seems better, since my pods will be sending metrics to an exporter running on the same node, but I am not able to send metrics to the local pod of statsd-exporter since I don't have the IP of the node the pod is running on.

Can you please compare the pros and cons of both methods, and suggest how can I know the IP address of Node on which the pod is running along with the exporter.

EDIT 1

I am able to get the Node IP by adding the environment variable.

- name: NODE_IP
  valueFrom:
    fieldRef:
      fieldPath: status.hostIP

I still need clarity on which will be better approach to setting this up. Exposing a service with type ClusterIP and then using the HOST:PORT from environment variable in the pod or use hostNetwork: true in pod spec and then access it using NODE_IP environment variable. Does clusterIp guarantees that the packet will be routed to the same node as the pod sending the packet?

EDIT 2

I explored headless service and I think this comes closest to what I want. But I am not sure that the DNS resolution will return the local node IP as the first result in nslookup.

Answer 1

I would suggest one of the below two approaches, both have their pros and cons.

• Fast, may not be entirely secure.

Daemon set using hostPort. It would be fast as both pods would be on the same node, but the statsd port would be exposed. (You need to secure statsd by some other way)

• Not as fast as hostPort but secure

Exposing a service and using the service dns to connect (servicename.namespace.svc.cluster.local). Not as fast as the hostPort as there is no way to reach specific pod, but secure as no one from outside the cluster can hit the statsd.

More details: https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/#communicating-with-daemon-pods