2

I am collecting logs from a kubernetes cluster using fluentbit, having an output that connect to loki to send them there.

This is my loki configuration at fluentbit configmap file

Since loki is deployed at loki namespace, and fluentbit at fluentbit namespace I am using to contact loki: host loki.loki.svc.cluster.local 

apiVersion: v1
data:
  custom_parsers.conf: |
    [PARSER]
        Name docker
        Format json
        Time_Key time
        Time_Format %Y-%m-%dT%H:%M:%S %z
  fluent-bit.conf: |
    [SERVICE]
        Daemon Off
        Flush 1
        Log_Level info
        Parsers_File parsers.conf
        Parsers_File custom_parsers.conf
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port 2020
        Health_Check On
    [INPUT]
        Name tail
        Path /var/log/containers/*.log
        multiline.parser docker, cri
        Tag kube.*
        Mem_Buf_Limit 100MB
        Skip_Long_Lines On
    [INPUT]
        Name systemd
        Tag host.*
        Systemd_Filter _SYSTEMD_UNIT=kubelet.service
        Read_From_Tail On
    [FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log On
        Keep_Log Off
        K8S-Logging.Parser On
        K8S-Logging.Exclude On
    [OUTPUT]
        Name stdout
        Match kube.*
        Format json
        Json_date_key timestamp
        Json_date_format iso8601
    [OUTPUT]
        Name loki
        Match kube.*
        host loki.loki.svc.cluster.local 
        port 3100
        tenant_id ""
        Labels {job="fluent-bit"}
        auto_kubernetes_labels false
        line_format json
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: fluent-bit
    meta.helm.sh/release-namespace: fluent-bit
  creationTimestamp: "2021-10-21T13:53:14Z"
  labels:
    app.kubernetes.io/instance: fluent-bit
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: fluent-bit
    app.kubernetes.io/version: 1.8.8
    helm.sh/chart: fluent-bit-0.19.1
  name: fluent-bit
  namespace: fluent-bit

But I got this error in my fluentbit logs.

[2021/10/21 14:59:59] [error] [output:loki:loki.1] loki.loki.svc.cluster.local:3100, HTTP status=400 Not retrying.
1:2: parse error: unexpected left brace '{'

Looks like that is not the correct format, and sometimes I got this another message with the same configuration (weird):

[2021/10/21 14:59:59] [error] [output:loki:loki.1] loki.loki.svc.cluster.local:3100, HTTP status=400 Not retrying.
1:2: parse error: unexpected left brace '{'

Like I would have to explicitly specify the POST endpoint on loki to push logs there, this one /loki/api/v1/push

But in general terms, I am getting the most the 400 bad syntax error. How can I contact loki from fluentbit configuration?

bgarcial
  • 2,915
  • 10
  • 56
  • 123

1 Answers1

1

You should not use curly braces for the labeling, this would do:

[OUTPUT]
    ...
    Labels job="fluent-bit"
    ...

See the example here: https://docs.fluentbit.io/manual/pipeline/outputs/loki

Mikhail Chuprynski
  • 2,404
  • 2
  • 29
  • 42