2

I have configured a working EFK(Elasticesearch,Fluentd,Kibana) in one of my kubernetes cluster builded in GCP. I have two more clusters and installed the same EFK in remaining too. Now If I want to monitor the logs of each cluster environment,then I need to check all the three kibana console. Please let me know is it possible to centralize the all EFK builded in three clusters, So that I can manage to see the pod logs from all my clusters in a single Kibana console. Any help or suggestion will be helpful.

manu thankachan
  • 433
  • 3
  • 9
  • 19
  • Have you looked into [cross cluster search](https://www.elastic.co/guide/en/elasticsearch/reference/6.3/modules-cross-cluster-search.html) ? – Crou Jul 10 '18 at 10:54

1 Answers1

1

In fact Kibana only draws and allows to sort/manage data which exists in Elasticsearch. Let's say, you have 3 k8s clusters. Consequently, you have 3 DaemonSet of Fluentd. All you should do - is configure all Fluentd deployments to send data to the one and only Elasticsearch endpoint, to which the Kibana is connected.

Konstantin Vustin
  • 6,521
  • 2
  • 16
  • 32
  • Thank you for the advice. I will try this. – manu thankachan Jul 10 '18 at 12:27
  • As per your update I have added the Elasticesearch URL from one of my cluster to a kibana yaml file. Now I have two entries for ELASTICSEARCH_URL. One is from the cluster itself and other is from another cluster. But the issue is the logs from remote cluster is visualising in kibana but the logs from local cluster is not. Can you identify the issue? – manu thankachan Jul 10 '18 at 13:43
  • without a config files, logs and a brief description of your architecture nobody could help you – Konstantin Vustin Jul 11 '18 at 03:20
  • Hi, I have changed my plan and I configured Elasticesearch and kibana in a separate server and the fluentd pods are running on my cluster. For getting connection between fluentd pod and remote elasticesearch, I have created an endpoint for fluentd from cluster. But now my fluentd pod is not able to reach my remote elasticesearch cluster. Please tell me where can I add my elasticesearch url in fluentd yaml. I have followed the kb https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch – manu thankachan Jul 12 '18 at 10:08
  • I don't really know much about `fluentd`, but from your link it's clear that "elasticsearch URL" is "under" the *elasticsearch-logging* `Service`. You can create `Service` resource with the `externalName` type and connect similar way. More info: https://akomljen.com/kubernetes-tips-part-1/ – Konstantin Vustin Jul 13 '18 at 03:29
  • Thank you @Konstantin Vustin for your update. I think I have done it. I have added my elasticesearch server IP in the fluentd configuration from the fluentd-es-configmap.yaml described in the link https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch. Now my fluentd is sending the logs to my remote elasticesearch server. :) – manu thankachan Jul 13 '18 at 06:32