Why sink operation execute multiple times in my flink program?

Question

I have a flink program with source from kafka, and i opened three windowedStream：seconds, minutes,hours.Then sending window result to others by AsyncHttpSink extends RichSinkFunction.But i found that same window，one kafka message, same result may invoke AsyncHttpSink.invoke() function multiple times which aroused my curiosity.Shouldn't it happen just once in same window，one kafka message, same result?

hourOperator.addSink(httpSink(WindowType.h));
minuteOperator.addSink(httpSink(WindowType.m));
secondOperator.addSink(httpSink(WindowType.s));


/**
 * http sink
 */
public class AsyncHttpSink extends RichSinkFunction<Tuple3<String, Long, Map<String, Tuple2<XXX, Object>>>> {


    public AsyncHttpSink(WindowType windowType) {
        this.windowType = windowType;
    }

    @Override
    public void open(Configuration parameters) throws Exception {
        httpClient = HttpAsyncClients.custom()
                .build();
        httpClient.start();
    }

    @Override
    public void close() throws Exception {
        httpClient.close();

    }



    @Override
    public void invoke(Tuple3<String, Long, Map<String, Tuple2<XXX, Object>>> tuple3, Context context) throws Exception {
        
        httpClient.execute(httpPost, new FutureCallback<HttpResponse>() {
            @Override
            public void completed(HttpResponse response) {
                try {
                    logger.info("[httpSink]http sink completed.");
                } catch (IOException e) {
                    logger.error("[httpSink]http sink completed. exception:", e);
                }
            }

            @Override
            public void failed(Exception ex) {
                logger.error("[httpSink]http sink failed.", ex);
            }

            @Override
            public void cancelled() {
                logger.info("[httpSink]http sink cancelled.");
            }
        });
    }
}

How do you know that it's the same result being invoked multiple times? — David Anderson, Apr 20 '22 at 13:53
log4j record the result in http method，every window has it's own endtime line, eg:9144_102_1649584800000 . And the result of every window should be Sequential grow. — Flory Li, Apr 21 '22 at 01:36

score 0 · Answer 1 · answered Apr 21 '22 at 08:45

0

If this is a keyed window, then each distinct key that has results for a given window will report its results separately.

And you may have several parallel instances of the sink.

answered Apr 21 '22 at 08:45

David Anderson

39,434
4
33
60

Yes, it's a keyed window, and each keyed window has it's own sink instance. I build sink instance like : secondOperator.addSink(new AsyncHttpSink()). Did I do anything wrong？ I have a guess: multiple different keys have the same window. As long as one key window triggers, other keys window will also trigger sink, Is it possible？ – Flory Li Apr 22 '22 at 06:41
Yes, there is just one instance of each operator in each task slot. Flink multiplexes each instance, using it for all of the keys belonging to all of the key groups assigned to that slot. – David Anderson Apr 22 '22 at 06:56
Thanks, but i need three keyed window different by time span(hour, minute, second), how can i solve this problem? – Flory Li Apr 22 '22 at 07:11
That seems like a different question than this one. Please ask another question, and share what you've already tried. But first see if https://stackoverflow.com/questions/71937236/how-to-do-multiple-aggregate-with-single-window-in-flink answers your question. – David Anderson Apr 22 '22 at 07:15
I read it and my question has a little different after a survey. I did a local test : First I opened a one hour window （before that i opened three window）, then first message with key1 step into the window, and the second message with key2 step into the window. The result is that the sink was triggered twice when the second message step into, but it triggered only once if the second message had the same key with the first message. – Flory Li Apr 26 '22 at 09:15
Yes, that's exactly what I'd expect. Keyed windows aggregate on a per-key basis, so you'll get a separate result for each window for each key. – David Anderson Apr 26 '22 at 09:29

Why sink operation execute multiple times in my flink program?

1 Answers1