Filebeat TCP Input Usage

Question

Questions:

• Do TCP inputs manage harvesters (i.e. do you send a file path to the TCP input and then a harvester starts ingesting that file)?
• Can TCP inputs accept structured data (like the json configuration option on the log input)?
• Does the TCP input expect the data sent over the TCP connection to be in a specific format?

From the filebeat documentation (https://www.elastic.co/guide/en/beats/filebeat/current/how-filebeat-works.html#input):

What is an input? An input is responsible for managing the harvesters and finding all sources to read from.

The documentation is very unclear on how to use TCP inputs and what format of data these TCP inputs expect (https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-tcp.html). I ended up reading the source code for the TCP input, but it is still unclear to me how to use TCP inputs.

Answer 1

TL;DR: It expects a JSON message, what I saw was sent from logstash to filebeat.

I have the same doubt as you. However, I spent a whole day trying to use filebeat somehow. What I wanted to achieve was adding stacktrace to kibana, and many websites said that, as a good practice, you shouldnt apply the multiline conecpt in logstash and use it in filebeat.

So, I tried adding filebeat to my ELK stack.

This is the previous communication between my application and the stack:

SpringApp > Logstash > ElasticSearch > Kibana

But I changed it adding filebeat with TCP communication

SpringApp > Logstash > Filebeat > ElasticSearch > Kibana

I achieved it with the following configuration:

logstash.conf

input {
    tcp {
        port => 5000
        codec => multiline {
            pattern => "\tat\s"
            what => "previous"
        }
    }
}

## Add your filters / logstash plugins configuration here

output {
    tcp {
        host => "filebeat"
        port => "5044"
        mode => "client"
        reconnect_interval => "2"
        codec => "line"
    }
    stdout { }
}

filebeat.yml

filebeat.inputs:

- type: tcp
  host: "0.0.0.0:5044"

multiline.pattern: '^\t'
multiline.negate: false
multiline.match: after

output.elasticsearch:
  hosts: ["http://elasticsearch:9200"]

logging.level: debug
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0644
logging.metrics.enabled: false

Note 1: Logstash is receiving and sending the communication via TCP
Note 2: Other logstash codecs for the output were not recognized by filebeat
Note 3: I'm using kubernetes and the containers for the communication between all