7

I am currently wondering how to handle application errors in Apache Flink streaming applications. In general, I see two cases:

  1. Transient errors, where you want the input data to be replayed and processing might succeed on second try. An example would be a dependency on an external service, which is temporarily unavailable.
  2. Permanent errors, where repeated processing will still fail; for example invalid input data.

For the first case, it looks like the common solution is to just throw some exception. Or is there a better way, e.g. a special kind of exception for more efficient handling such as FailedException from Apache Storm Trident (see Error handling in Storm Trident topologies).

For permanent errors, I couldn't find any information online. A map() operation, for example, always has to return something so one cannot just silently drop messages as you would in Trident.

What are the available APIs or best practices? Thanks for your help.

F30
  • 1,036
  • 1
  • 10
  • 21
  • 2
    FYI: [How to handle errors in custom MapFunction correctly?](http://stackoverflow.com/questions/36085646/how-to-handle-errors-in-custom-mapfunction-correctly) – Maxim Apr 28 '16 at 10:38

1 Answers1

1

Since this question was asked, there has been some development:

This discussion holds the background of why side outputs should help, key extract:

Side outputs(a.k.a Multi-outputs) is one of highly requested features in high fidelity stream processing use cases. With this feature, Flink can

  • Side output corrupted input data and avoid job fall into “fail -> restart -> fail” cycle
  • Side output sparsely received late arriving events while issuing aggressive watermarks in window computation.

This resulted in jira: FLINK-4460 which has been resolved in Flink 1.1.3 and above.

I hope this helps, if an even more generic solution would be desireable, please think a bit on your usecase and consider to create a jira for it.

Dennis Jaheruddin
  • 21,208
  • 8
  • 66
  • 122
  • I have to admit that it's been a while since I wrote Flink quote and thought about this question. From my impression, Side Outputs do in fact look useful. However, they are only available for the generic `ProcessFunction` and its relatives. There still appears to be no good way of error handling for a simple `Map`. – F30 Jul 31 '20 at 10:43