0

I have a problem with the Apache Flink Streaming API.

I could manage to set up the whole CEP-Environment with a custom DataSource and when using a standard sink on that source like "print()", everything works fine.

This is what my sink looks like now:

@RequiredArgsConstructor
public class EventDataConsumer extends RichSinkFunction<EventData>{

private final transient Consumer<EventData> consumer;

    @Override
    public void invoke(EventData eventData) throws Exception {
        consumer.accept(eventData);
    }
}

What I try to achieve ist, to pass a method reference to this SinkFunction, which shall be executed for each element in my DataStream.

This is how I initialize the SinkFunction:

EventDataConsumer consumer = new EventDataConsumer(someService::handleEventData);
outStream.addSink(consumer);

My Problem is, that when I set a breakpoint in the "invoke" method of my custom sink, the consumer appears to be null even though I call the constructor explicitly, which assigns the consumer.

H. Ayguen
  • 33
  • 5

1 Answers1

1

As the Sink is distributed to as many instances as the parallelism of the sink it should be serializable. When executing on cluster the Sink is serialised sent to TaskManagers where it is deserialised.

In your example the consumer field is transient, that is why after serialisation it becomes null.

Dawid Wysakowicz
  • 3,402
  • 17
  • 33
  • Thanks, that really helps understanding the problem. Any idea on how to make a method reference serializable? – H. Ayguen Aug 24 '17 at 10:13
  • I meant the Consumer-Field, sorry. I tried creating an SerializableConsumer-Interface which extends the Consumer and Serializable-Interface but that doesn't seem to work. – H. Ayguen Aug 24 '17 at 10:25
  • Hmm it should work this way. Are you sure you removed the `transient` keyword? Also make sure the method is really serializable. Remember that lambdas usually have some closure that also needs to be serializable. In your example `someService` also needs to be serializable. – Dawid Wysakowicz Aug 25 '17 at 13:35