winston logs to EFK Stack on GKE

Question

I'm writing my nodejs logs to a file using winston js and storing them on /tmp/logs volume on my host node on GKE. Now, I'm trying to add the EFK Stack to store them and visualize them but my problem is that I'm new to the whole logging process and I'm facing troubles understanding the configuration of fluentd. I've tried to configure the stack on my local cluster with the help of this tutorial and did visualize the stdout and stderr logs of my containers but those of my winston files are not shown.

To do so:

1. How should I tell fluentd to read *.log files that I'm storing under /tmp/logs on the cluster?
2. What is the best way to install EFK stack on GKE cluster (ECK....) since I already have Elasticsearch and Kibana installed for production purposes. I'm thinking of isolating the new ones under a logging namespace but I also found that I can install the monitoring stack on a GCE machine directly.
3. In terms of resource management, does the EFK stack require a lot of RAM, CPU...?

NB: GCP used to have Elastic gke logging on the marketplace that I wanted to try but it's not there anymore.

Answer 1

fluentd config can be entered into ConfigMap, for example:

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-config
data:
  fluentd.conf: |
    <source>
      type tail
      format none
      path /tmp/logs/1.log
      pos_file /tmp/logs/1.log.pos
      tag count.format1
    </source>

    <source>
      type tail
      format none
      path /tmp/logs/2.log
      pos_file /tmp/logs/2.log.pos
      tag count.format2
    </source>

    <match **>
      type google_cloud
    </match>    

There are also complete tutorials on configureing EFK on GKE here and here.