0

We are running several microservices on Google Cloud Platform (GKE using master version v1.12.9-gke.15) and have been using the Stackdriver integration for logging and monitoring (https://cloud.google.com/monitoring/kubernetes-engine/installing). Unfortunately, we also need to have custom log parsing and it is currently not possible to have custom log parsing and Stackdriver monitoring at the same time (discussed also here: https://stackoverflow.com/questions/57528039/gke-stackdriver-kubernetes-engine-monitoring-and-custom-log-format). It seems that we have to turn off the automatic integration with StackDriver for both logging and monitoring.

My question is: What do I need to apply to our cluster to supplement the automatic StackDriver monitoring integration?

I am specifically looking for descriptions on what particular deployments achieve and official deployment descriptors that can be applied directly to the cluster.

From what I can see the difference between enabling and disabling the "Stackdriver Kubernetes Engine Monitoring" are the monitoring related deployments:

  1. event-exporter-v0.2.4
  2. heapster-v1.6.1
  3. stackdriver-metadata-agent-cluster-level

Here is what I found so far:

  1. The event-exporter enables events' gathering from the cluster (information: https://github.com/Stackdriver/kubernetes-configs and https://github.com/GoogleCloudPlatform/k8s-stackdriver/tree/master/event-exporter). I am missing the official YAML descriptor to be able to deploy it manually (the YAML descriptors in the referenced links are not up-to-date).
  2. The metrics-server-v0.3.1 (which is active regardless of what is enabled) and AFAIK is replacing the heapster deployment (source: https://github.com/kubernetes-retired/heapster/blob/master/docs/deprecation.md ). Should it still be deployed?
  3. I have found some references to the stackdriver-metadata-agent-cluster-level here: https://cloud.google.com/gke-on-prem/docs/concepts/logging-and-monitoring#stackdriver_gkeop and there are some descriptors here: https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/metadata-agent/stackdriver/metadata-agent.yaml but they describe a daemonset and a deployment and settings appear to differ from what I have.

And, finally, a sub-question: is this list complete to fully enable monitoring the Kubernetes cluster?

(Note: I have omitted logging related deployments and daemonsets. Supplementing them with our custom deployed configurations is sufficiently documented on the web. Besides, we do require customizations for the logs' parsing.)

Prerok
  • 1
  • What exactly you meant by 'supplement the automatic StackDriver monitoring integration'? – Digil Oct 17 '19 at 19:24
  • @Digil, I meant that we are happy with the current monitoring functionality provided when setting the "Stackdriver Kubernetes Engine Monitoring" (SKEM) to "Enabled". We would be happy to keep it but we cannot because we need custom logs' parsing. I opened a Feature Request to allow separate setting for logging and for monitoring just as it was for Legacy Stackdriver. Right now, though, we have to disable the monitoring functionality as well. I am looking for a way to manually enable the same *monitoring* functionality as is provided when setting the SKEM to "Enabled". – Prerok Oct 18 '19 at 07:52
  • Looks like this kind of tweaking is not possible if you are using [Stackdriver Kubernetes Engine Monitoring](https://cloud.google.com/kubernetes-engine-monitoring/). As explained [here](https://cloud.google.com/monitoring/kubernetes-engine/#version), only [Legacy Stackdriver](https://cloud.google.com/monitoring/kubernetes-engine/#lkm-howto) lets you disable Stackdriver Logging while still using Stackdriver Monitoring – Digil Oct 18 '19 at 21:33
  • @Digil, understood. I am aware that we have to disable the monitoring mechanism. We are looking for a way to replace it with some sort of manual deployments with as little hassle and/or cost of maintenance. – Prerok Oct 21 '19 at 08:54
  • If you didn't find any workaround that suits your needs, I would strongly recommend you to use the [issue-tracker](https://cloud.google.com/support/docs/issue-trackers) to create one feature request with [Stackdriver](https://issuetracker.google.com/savedsearches/559785) monitoring. While opening the feature request, try to explain your current situation and also how exactly you want to improve it by implementing new feature with Stackdriver. – Digil Oct 25 '19 at 15:04

0 Answers0