1

I'm running Fluent Bit in a warehouse environment where internet connection drops quite often. Still, I want to upload data from fluent bit to my logstash server (HTTP output).

I ran some tests by disconnecting from the internet, sending the data, wait a minute and reconnecting. Unfortunately, on kibana, I can see the timestamp sent by Fluent bit is one after this minute, instead of the one before. This doesn't matter when sending log files but when tracking performance over time, or CPU activity, it really messes up the data.

I thought of having a central device in the warehouse where I could forward the data and this one would send it to logstash with Fluentd but i was wondering if there was a less complex solution. I am also not sure which timestamp will be taken into account here.

Also, I took a look at collectd but it is not as easy to use as Fluent Bit.

Am I missing any obvious solution?

David Bensoussan
  • 2,887
  • 2
  • 38
  • 55
  • One simple solution would be to use Kafka between fluentbit and logstash. You would need a Kafka broker in your warehouse environment, then your fluentbit would send data to this broker and your logstash would pull the data and deal with retries if the connection is dropped. – leandrojmp Aug 10 '20 at 12:12
  • Is there any reason to use kafka (and add some more tools) instead of going with fluent? – David Bensoussan Aug 10 '20 at 15:16
  • 1
    I'm not familiar with FluentBit but there is an Time_Key option for the es output plugin. Using it should render the original timestamp into the event regardless of the time of transport... Did i've missed sth? – ibexit Aug 10 '20 at 18:15

1 Answers1

0

As pointed by ibexit, the timestamp is sent properly but in my case the logstash conversion is failing. If i send the data directly to elasticsearch the timestamp is fine. So, the answer is to use the full storage system provided by fluentbit while sending directly to logstash.

In warehouse environments, forwarding the data to another unit, in a centralized ways has other benefits and can be seen better for security constraints but this depends on your business case.

David Bensoussan
  • 2,887
  • 2
  • 38
  • 55