0

I deployed a logstash by statefulset kind with 3 replicas in k8s. Using filebeat to send data to it.

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: logstash-nginx
spec:
  serviceName: "logstash"
  selector:
    matchLabels:
      app: logstash
  updateStrategy:
    type: RollingUpdate
  replicas: 3
  template:
    metadata:
      labels:
        app: logstash
    spec:
      containers:
      - name: logstash
        image: docker.elastic.co/logstash/logstash:7.10.0
        resources:
          limits:
            memory: 2Gi
        ports:
          - containerPort: 5044
        volumeMounts:
          - name: config-volume
            mountPath: /usr/share/logstash/config
          - name: logstash-pipeline-volume
            mountPath: /usr/share/logstash/pipeline
        command: ["/bin/sh","-c"]
        args:
          - bin/logstash -f /usr/share/logstash/pipeline/logstash.conf;
      volumes:
        - name: config-volume
          configMap:
            name: logstash-configmap
            items:
              - key: logstash.yml
                path: logstash.yml
        - name: logstash-pipeline-volume
          configMap:
            name: logstash-configmap
            items:
              - key: logstash.conf
                path: logstash.conf

Logstash's service

---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: logstash
  name: logstash
spec:
  ports:
    - name: "5044"
      port: 5044
      targetPort: 5044
  selector:
    app: logstash

Filebeat's daemonset configmap

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    ...

    output.logstash:
      hosts: ["logstash.default.svc.cluster.local:5044"]
      loadbalance: true
      bulk_max_size: 1024

When run real data. Most data went to the second logstash's pod. Sometimes data also can go to the first and the third pods but very little occurs.

Use another way to set LB from filebeat

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    ...

    output.logstash:
      hosts: ["logstash-nginx-0.logstash.default.svc.cluster.local:5044", "logstash-nginx-1.logstash.default.svc.cluster.local:5044", "logstash-nginx-2.logstash.default.svc.cluster.local:5044"]
      loadbalance: true
      bulk_max_size: 1024

Logstash's configmap

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: logstash-configmap
data:
  logstash.yml: |
    http.host: "0.0.0.0"
    path.config: /usr/share/logstash/pipeline
    xpack.monitoring.enabled: false

From the filebeat pod, it can access URIs by curl:

  • logstash-nginx-0.logstash.default.svc.cluster.local:5044
  • logstash-nginx-1.logstash.default.svc.cluster.local:5044
  • logstash-nginx-2.logstash.default.svc.cluster.local:5044

But the data can't be sent by filebeat to logstash's 3 pods at all. No traffic in the logstash's output logs. Where is wrong?

iooi
  • 453
  • 2
  • 10
  • 23
  • What GKE version are you using. Can you check pod logs and verify if there is something suspicious in logs? – PjoterS Dec 16 '20 at 12:16
  • 1
    Do you need 3 replicas? I saw that you are using `sts` with `selectors`. As per [sts limitations docs](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#limitations) `StatefulSets currently require a Headless Service to be responsible for the network identity of the Pods. You are responsible for creating this Service.` Filebeat provides configuration options that you can use to fine tune load balancing when sending events to [multiple hosts](https://www.elastic.co/guide/en/beats/filebeat/6.8/load-balancing.html). – PjoterS Dec 16 '20 at 15:21
  • You can try to set-up output logstash with [Stable Network ID](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#stable-network-id) for stst pods. `The domain managed by this Service takes the form: $(service name).$(namespace).svc.cluster.local, where "cluster.local" is the cluster domain.` It would look like: `logstash-nginx-0.logstash.default.svc.cluster.local, logstash-nginx-1.logstash.default.svc.cluster.local` – PjoterS Dec 16 '20 at 15:22
  • @PjoterS Thank you for your advice. I tried your method. It seems should work but can't run. I posted the information to the end of the question. – iooi Dec 17 '20 at 01:54

0 Answers0