2

I have a Graylog installation that can handle well over 25.000 msg/sec.

I am now experimenting with setting up a dedicated Logstash cluster. So something like this: Filebeat (n hosts) -> Logstash Cluster -> Graylog

The biggest issue now is that I can not send more than ~2.500 msg/sec from one host to Graylog. No matter how big the host is (4, 8, 16, … CPUs), no matter how small the messages are, there is always this 2.500 msg/sec limit.

I’ve even created messages with only 1 Byte payload, the limit here is ~2.800 msg/sec.

I can send messages from, for example, 4 different hosts. Resulting in 10.000 msg/sec (4 x 2.500). Graylog can handle this easily.

So my question is: Why can one host send no more than ~2.500 msg/sec via GELF UDP?

Frizz
  • 2,524
  • 6
  • 31
  • 45
  • 1
    The size of the network pipe maybe? A "logstash cluster" means several instances, how many do you have on that single host? – Val Aug 28 '18 at 02:56
  • @Val: I have one Logstash instance per Host. – Frizz Aug 29 '18 at 08:19
  • Some more information: I wrote a small Python script to send UDP messages in GELF format to Graylog. I easily managed to send 15.000 msg/sec. So the bottleneck is most probably the Logstash GELF Output Plugin, or the combination Logstash & GELF Output Plugin, or something like. Most probably its not an OS / Network limitation. – Frizz Aug 29 '18 at 08:21
  • How many Logstash hosts in your cluster then? At this point, it'd be useful to show some relevant filebeat/logstash configuration – Val Aug 29 '18 at 08:21
  • @Val: Number of Logstash instances is dynamic (-> autoscaled). But I set a minimum of 3. Also tried with minimum 5 or 6 or xyz. Same result. I also pre-filled a Redis host with Log messages and configured this as Input for Logstash (so that I can read messages as fast as possible). Still it can send only ~2.500 msg/sec to Graylog. – Frizz Aug 29 '18 at 08:24
  • @Val: Logstash config is pretty simple: File input (or Redis, for testing), no filters, no processing, output { gelf { host => "172.12.19.24" port => 12201 sender => "%{host}" } } – Frizz Aug 29 '18 at 08:26
  • interesting... have you tried to play with the `chunk_size` in the gelf output ? – Val Aug 29 '18 at 08:31
  • Hi @Frizz, did you find a solution to your problem? And what caused the issue? – Nico Mar 09 '21 at 14:51

0 Answers0