1

I build a topo to get messages from by kafka, and then grep some keyword, if fit, write to local file.

I use OpaqueTridentKafkaSpout of storm-kafka to ensure the tuple will not miss or repeat, but consider one situation: when writing message to local file, and some error occur (for example, not enough space). At this moment, some messages have written to local file, and others not, if the spout resend the message, the message will be write twice.

How to handle that?

Matthias J. Sax
  • 59,682
  • 7
  • 117
  • 137
jinhong_lu
  • 238
  • 1
  • 2
  • 11
  • if the message is sent again, it will processed again. Your concern should be about what happens to the tuples in case of failure. You have to look at whatever is writing to the files to understand how the failure is handled (if the tuples are failed or acknowledged). If tuples are failed the spout will read them again from kafka, if they are acknowledged they are "consumed" and the spout will not attempt to read them. – SQL.injection Jul 03 '15 at 11:15

2 Answers2

1

It's simple. The code that writes to the file needs to do the following:

1) Ack the tuple - Only if the write to a file is successful. 2) Fail the tuple - If the write to a file was NOT successful.

For all tuples that were ack'd, Kafka spout will NOT resend them. Failed tuples will be reset by the spout.

redTiger
  • 312
  • 4
  • 5
0

You must design your anchoring strategy for this purpose. I suggest that you can decrease batch size from kafkaspoutconfig and stores your selected messages in list. When all messages in batch is processed, you can write list content to local file .

As you know, Trident processes stream in batch, if your system throw any error while processing any of tuple in stream, all batch will be discarded.

In your case, you can surround by try catch your code block which is responsible for write to local file, and in catch block you must throw backtype.storm.topology.ReportedFailedException. By this way, you can ensure exactly one semantic.

Also you must use transactional kafka spout to ensure exactly one semantic.

serkan kucukbay
  • 623
  • 7
  • 15