Logstash Error | lumberjack protocol error

Question

We are trying to push our logs from JournalBeat to Logstash. We see some data populated in logstash after the journalbeat service is started. But, not sure why our jounrnalbeat logs are continuously giving the below error message.

2020-07-07T11:52:33.506Z        INFO    [publisher_pipeline_output]     pipeline/output.go:152  Connection to backoff(async(tcp://logstash-hostname:5044)) established
2020-07-07T11:52:33.562Z        ERROR   [logstash]      logstash/async.go:280   Failed to publish events caused by: lumberjack protocol error
2020-07-07T11:52:33.562Z        INFO    [publisher]     pipeline/retry.go:221   retryer: send unwait signal to consumer
2020-07-07T11:52:33.562Z        INFO    [publisher]     pipeline/retry.go:225     done

journalbeat.yml config is as below for the output.

    output.logstash:
      hosts: ["logstash-hostname:5044"]
      protocol: "http"
      index: "journal-${ENVIRONMENT}-system-%{[agent.version]}-%{+YYYY.MM.dd}"
      indices:
        - index: "journal-${ENVIRONMENT}-k8s-%{[agent.version]}-%{+YYYY.MM.dd}"
          when.has_fields:
          - 'kubernetes.namespace'

Below is the beats.conf included in the Logstash configuration.

input {
 beats {
    type => "beats"
    host => "0.0.0.0"
    port => "5044"
  }
}
output {
    stdout {}
}

Could you please suggest on how can this be fixed. Thanks in Advance.

Try to remove the `protocol` line from your journalbeat config, this option does not exist for the logstash output, it does not use `http`. The `indices` option also do not exist, you will probably need to apply those filters in your logstash pipeline. Take a look at the [documentation](https://www.elastic.co/guide/en/beats/journalbeat/current/logstash-output.html) for the existing options. — leandrojmp, Jul 07 '20 at 12:25
I tried removing indices & index, protocol and just having the hosts alone. Still observing the same error. — Vamshi Siddarth, Jul 07 '20 at 12:34

score 1 · Accepted Answer · answered Jul 07 '20 at 13:03

The issue is fixed now. I missed adding the task to copy the beats.conf file to my logstash instance and that was causing the issue. Also, the index, protocol and indices attributes are not required by logstash. Remove these as well and it works absolutely fine now. Thanks leandrojmp :)

Logstash Error | lumberjack protocol error

1 Answers1