Flink SQL Job runs out of heap space

Question

I am running a query to join a stream and a table as below. It is running out of heap space. Even though it has enough heap space in flink cluster (60GB * 3)

Is there an eviction strategy needed for this query ?

SELECT sourceKafka.* FROM sourceKafka INNER JOIN DefaulterTable ON sourceKafka.CC=DefaulterTable.CC;

David Anderson · Answer 1 · 2019-09-29T08:02:19.890

0

If the table that you want to join with the stream is static, then you could implement your own join that would be much better behaved.

One way to do this would be to use the state processor API to create a savepoint that has the static table in Flink state, and then implement the join as a KeyedProcessFunction.

If you need to periodically update the "static" table, then it should be workable to use a KeyedCoProcessFunction, and stream in the occasional update.

edited Sep 29 '19 at 08:02

answered Sep 26 '19 at 14:29

David Anderson

39,434
4
33
60

Thanks David. Looking into it – Nishant Gupta Sep 30 '19 at 14:37

Flink SQL Job runs out of heap space

1 Answers1