0

I'm trying to get Heapster/InfluxDB/Grafana set up on a Kubernetes cluster. It looks like Heapster is generating data but not sending it to InfluxDB. When I look at the container logs for the eventer container within the Heapster pod, the logs look something like this:

E1020 18:43:20.006608      52 influxdb.go:147] Failed to create infuxdb: failed to ping InfluxDB server at "monitoring-influxdb:8086" - Get http://monitoring-influxdb:8086/ping: dial tcp: lookup monitoring-influxdb on 10.254.0.10:53: read udp 172.16.23.2:43533->10.254.0.10:53: i/o timeout
I1020 18:43:30.000165      52 manager.go:95] Exporting 88 events
I1020 18:44:00.000173      52 manager.go:95] Exporting 29 events
W1020 18:44:20.000451      52 manager.go:108] Failed to events data to sink: InfluxDB Sink
E1020 18:44:20.075039      52 influxdb.go:147] Failed to create infuxdb: failed to ping InfluxDB server at "monitoring-influxdb:8086" - Get http://monitoring-influxdb:8086/ping: dial tcp: lookup monitoring-influxdb on 10.254.0.10:53: read udp 172.16.23.2:60660->10.254.0.10:53: i/o timeout
I1020 18:44:30.000211      52 manager.go:95] Exporting 26 events
I1020 18:45:00.000169      52 manager.go:95] Exporting 23 events
W1020 18:45:20.000568      52 manager.go:108] Failed to events data to sink: InfluxDB Sink
E1020 18:45:20.004656      52 influxdb.go:147] Failed to create infuxdb: failed to ping InfluxDB server at "monitoring-influxdb:8086" - Get http://monitoring-influxdb:8086/ping: dial tcp: lookup monitoring-influxdb on 10.254.0.10:53: read udp 172.16.23.2:53401->10.254.0.10:53: i/o timeout
I1020 18:45:30.000568      52 manager.go:95] Exporting 30 events
I1020 18:46:00.000188      52 manager.go:95] Exporting 26 events
W1020 18:46:20.000504      52 manager.go:108] Failed to events data to sink: InfluxDB Sink
E1020 18:46:20.074697      52 influxdb.go:147] Failed to create infuxdb: failed to ping InfluxDB server at "monitoring-influxdb:8086" - Get http://monitoring-influxdb:8086/ping: dial tcp: lookup monitoring-influxdb on 10.254.0.10:53: read udp 172.16.23.2:49101->10.254.0.10:53: i/o timeout
I1020 18:46:30.000267      52 manager.go:95] Exporting 39 events
I1020 18:46:35.052332      52 influxdb.go:199] Created database "k8s" on influxDB server at "monitoring-influxdb:8086"
I1020 18:47:00.000173      52 manager.go:95] Exporting 64 events
I1020 18:47:30.000212      52 manager.go:95] Exporting 52 events
I1020 18:48:00.000234      52 manager.go:95] Exporting 43 events
I1020 18:48:30.070486      52 manager.go:95] Exporting 52 events
I1020 18:49:00.000340      52 manager.go:95] Exporting 40 events
I1020 18:49:30.000170      52 manager.go:95] Exporting 40 events
I1020 18:50:00.000205      52 manager.go:95] Exporting 57 events
I1020 18:50:30.000211      52 manager.go:95] Exporting 94 events

After that there's just a series of events being exported. My first question is did Heapster successfully connect to the InfluxDB service or is it just sending data to http://monitoring-influxdb:8086 assuming that something will be collected? If it did connect, why is the k8s database in my InfluxDB empty?

I deployed my cluster using the ansible script in kubernetes/contrib/ansible and the yaml files were downloaded from somwhere (I don't know where). I pushed them up to a git repository here if you want to take a look.

Ryan
  • 1,131
  • 3
  • 13
  • 17
  • How do you install heapster/influxDB? What do the yaml definitions look like? Or do you install via the k8s addon-manager? – fishi0x01 Oct 22 '17 at 09:57
  • I just updated the question, thank you. I used an ansible script to deploy kubernetes, and the script downloaded a set of yaml files. I then applied these yaml files with `kubectl apply -f` – Ryan Oct 22 '17 at 17:12

1 Answers1

1

The easiest way (IMO) to determine if it's collecting data is to look at the logs for the influxdb. In my environment (deployed from https://github.com/kubernetes/heapster/tree/master/deploy/kube-config/influxdb), the pods are all in the kube-system namespace:

kubectl get pods -n kube-system
NAME                                    READY     STATUS    RESTARTS   AGE
heapster-603813915-tggbr                1/1       Running   0          34d
kubernetes-dashboard-3313488171-mpn5b   1/1       Running   2          43d
monitoring-grafana-2175968514-zxgn1     1/1       Running   0          34d
monitoring-influxdb-1957622127-92r2w    1/1       Running   0          34d

and the logs:

  • kubectl logs monitoring-influxdb-1957622127-92r2w -n kube-system

    [httpd] 10.244.1.18 - root [23/Oct/2017:21:13:05 +0000] "POST /write?consistency=&db=k8s&precision=&rp=default HTTP/1.1" 204 0 "-" "heapster/v1.4.0" f5e74e91-b836-11e7-84d0-000000000000 121606 [httpd] 10.244.1.18 - root [23/Oct/2017:21:14:05 +0000] "POST /write?consistency=&db=k8s&precision=&rp=default HTTP/1.1" 204 0 "-" "heapster/v1.4.0" 19a8a32c-b837-11e7-84d1-000000000000 137213 [httpd] 10.244.1.18 - root [23/Oct/2017:21:15:05 +0000] "POST /write?consistency=&db=k8s&precision=&rp=default HTTP/1.1" 204 0 "-" "heapster/v1.4.0" 3d6eb415-b837-11e7-84d2-000000000000 125065 [httpd] 10.244.1.18 - root [23/Oct/2017:21:16:05 +0000] "POST /write?consistency=&db=k8s&precision=&rp=default HTTP/1.1" 204 0 "-" "heapster/v1.4.0" 6133e847-b837-11e7-84d3-000000000000 128300

The 204 responses are pretty clear that it's accepting data in this case. If your heapster isn't communicating with influxdb for some reason, then the answer is likely to be within the logs for the influxdb pod as well.

heckj
  • 7,136
  • 3
  • 39
  • 50
  • Thank you. After poking around, it looks like influxdb is getting the data as a series, but not saving it to the database. It's something I'll keep looking into – Ryan Oct 26 '17 at 16:53