MapValue use in Flink

Question

I have a question about using MapValue in Flink because I need to save a map as part of the state, as you know the state needs to be deserializable/serializable, so i extend my class from MapValue because MapValue is an abstract class.

public class HashMapValue<K extends Value, V extends Value> extends MapValue<K, V> {
    public HashMapValue() {
        super();
    }
}

When i try to use this class, i can’t initiate the class

HashMapValue<IntValue, BooleanValue> hashMapValue = new HashMapValue<IntValue, BooleanValue>();

the error message is as follows:

java.lang.AssertionError
    at org.apache.flink.util.ReflectionUtil.getTemplateTypes(ReflectionUtil.java:141)
    at org.apache.flink.util.ReflectionUtil.getSuperTemplateTypes(ReflectionUtil.java:98)
    at org.apache.flink.util.ReflectionUtil.getTemplateType(ReflectionUtil.java:44)
    at org.apache.flink.util.ReflectionUtil.getTemplateType1(ReflectionUtil.java:54)
    at org.apache.flink.types.MapValue.<init>(MapValue.java:55)

I step into the code and apparently in getTemplateTypes function, it gets K as the parameter instead of IntValue, which I clearly passed in as generic type.

I can’t find online or in the flink source code anything related to MapValue as example. Can anyone help me by pointing out how to use MapValue in Flink?

the reason why it doesn’t work is because there is a check for K,V type, which expects K, V as class. because the HashMapValue I created is still a generic class, K,V are not class, so the check fails. — user2289345, Sep 07 '22 at 21:05
Why do you think you need to extend `MapValue`, versus using it as-is? — kkrugler, Sep 08 '22 at 14:25
@kkrugler you are right, I changed my implementation and it is resolved. — user2289345, Sep 09 '22 at 17:57

David Anderson · Answer 1 · 2022-09-08T20:29:35.350

1

Flink's MapState is intended for just this purpose. See https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/learn-flink/event_driven/ for an example from the tutorials that are included in the documentation.

edited Sep 08 '22 at 20:29

answered Sep 08 '22 at 20:11

David Anderson

39,434
4
33
60

sorry, I was not clear on the initial question, my state has a field which is a map, so in order to serialize/deserialize the state, the field needs to implement Value. So it is not a pure MapState. But I got it resolved, thank you. – user2289345 Sep 09 '22 at 17:59
It could turn out to be dramatically more efficient to split that field out as a separate MapState, rather than having it be a field of a ValueState. – David Anderson Sep 11 '22 at 18:12

score 0 · Answer 2 · answered Sep 08 '22 at 00:33

0

MapValue is not for this purpose. Flink state supports Java Map natively, so we can use like HashMap as state value directly.

answered Sep 08 '22 at 00:33

BrightFlow

1,294
8
13

MapValue use in Flink

2 Answers2