1

We are planning to use 3 kafka-brokers and a zookeeper node. We have kept one partition for each topic. But we have a stringent requirement of latency under 5ms. We are using reactive-kafka client library https://github.com/akka/reactive-kafka.

e.g. We have a producer that produces 128 byte(approx) data at every 30ms and Consumer consumer at every 50 ms.

We tried with 

linger.ms=0

compression.type=none

acks=0

on producer side.

Polling on consumer side at 50ms

But as soon as we increase the frequency of producing to 1 second, the latency starts becoming unpredictable with 1ms to 70ms.

So, what should if we always require the latency under 5ms what tuning needs to be done at kafka-broker, producer and consumer level.

Suneel Kumar
  • 1,650
  • 2
  • 21
  • 31
Saloni Vithalani
  • 323
  • 1
  • 4
  • 13
  • 2
    Did you consider, what really means latency <5ms? I mean e.g. ping between 2 machines in the same network is often about 1-2ms and you want to guarantee latency 5ms, when 1-2ms are already used just by network. BTW increase of polling below 50ms makes more noise in TCP channels, what is also hidering data transport itself. For latencies like "5ms", I would consider dedicated p2p messaging, which does not poll (e.g. YAMI4). – Seweryn Habdank-Wojewódzki Jan 25 '18 at 09:24
  • @SewerynHabdank-Wojewódzki Thank you for your comment. We have an assumption that network latency will be in microseconds. This is something that our clients will take care of, as it will be an on premise deployment. With that in mind we were testing with kafka server and zookeeper running on same machine where publisher and subscriber is running just to get the best possible latency result. But those results were also becoming unpredictable as mentioned above. But your comment helps to incline towards something else from polling mechanism. So, we will keep that in mind. – Saloni Vithalani Feb 03 '18 at 05:15

0 Answers0