Sending Filebeat logs to AWS Kinesis

Question

My Problem

I have a log pipeline in which logs are written to files and shipped to ElasticSearch using Filebeat. I would like to switch from ElasticSearch to AWS Kinesis, and I wonder what's the right way to configure Filebeat for the new output.

My Options

Google for a solution - found kinesisbeat, but it is not very well documented or used.
Running a custom script using the exec plugin. However, since "The exec output will run a command for each event received", it is impractical for high volume log streams.
Writing my own plugin. Possible, but I would not like to reinvent the wheel.

My Question

How do I send logs from Filebeat to AWS Kinesis?

I'm facing the same problem, I'm currently looking at the firehose java agent http://docs.aws.amazon.com/firehose/latest/dev/writing-with-agents.html — Daniel Dror, Dec 27 '17 at 10:40
I am working on a custom output for Firehose & Kinesis for filebeat, but requires compling from source since they do not accept new plugins anymore — Mustafa, Feb 06 '18 at 11:22

score 3 · Accepted Answer · answered Feb 12 '18 at 07:57

3

You can check https://github.com/s12v/awsbeats, which I'm working on at the moment. However, it requires Go 1.10 which is still a RC.

answered Feb 12 '18 at 07:57

Sergey

351
3
11

Perfect. Please update your answer when it's fully functional. – Adam Matan Feb 13 '18 at 09:38

Sending Filebeat logs to AWS Kinesis

My Problem

My Options

My Question

1 Answers1