0

I have 120 data producers with Kakfa Producer, they send messages each second and some have sensible data (i mean, we have to send it yes or yes), also these producers sometimes could shutdown or lose internet connection, so I need fault tolerance in the producer. I know there are manners to do send it, syncronous and asyncronous and they persist trying to send data and keep it in memory when does not, but if i want to keep it in disk instead of buffer memory, how could we do it. How could we do to persist in disk the messages if they are not sent? How could we control when Kafka producer throw exceptions?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
O. Azofeifa
  • 1
  • 1

3 Answers3

0

You can insert messages to a database table in local transaction. After that by using a thread you can send messages and after getting ack from Kafka you can update sent column in the messages table as true. So, your thread should continually read messages from the messages table which sent field is false and send messages to Kafka. (if you have more than one instance, leader instance should be responsible to send messages to Kafka to avoid duplication)

Note: You can also use a separate service to send messages to Kafka as shown below.

enter image description here

For more information you can check this.

H.Ç.T
  • 3,335
  • 1
  • 18
  • 37
  • Interesting pattern, thanks, it's a good idea. At the end, I implemented another workaround, I used sync kafka method within a Thread (it does'nt seem to make sense since an async method already exists) and I configured max.block.ms to force a TimoutException, when this happens I save it. By the other side, I used an AtomicBoolean as a flag to coordinate the restoring process. – O. Azofeifa Mar 06 '20 at 06:55
0

I'd suggest writing to files immediately instead.

Use tools like Filebeat, Fluentbit, Flume, etc running on each machine to then funnel data into Kafka

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
0

Your use case for Kafka is wrong in this scenario.I suggest to save data in a NOSQL database instead of sending to Kafka.

MihaiGhita
  • 137
  • 6