0

Do we need to statically type/declare the data type of variable when is going to be serialized? Does it improve any performance while serializing?

I'm creating a flink project for batch processing. I wrote a custom input reader, which is going to read from database through jdbc and returns a record as Hashmap, containing column name and value. I know Flink serialize the objects between each subtasks. So, my question is, since I have hashmap where values of type object, does it have any impact on serialization performance?

Flink uses kyro serializer by default

Keshore Durairaj
  • 27
  • 5

2 Answers2

1

Serialization and deserialization of a hashmap is expensive. If you could make this work with Tuples or Rows, for example, that would perform better. Or you might consider implementing a custom Table source, and then take advantage of the Table/SQL API and its optimizations.

There's a recent article on the Apache Flink blog with detailed information about serialization, including a section on performance. I recommend the whole article, but for performance results, see https://flink.apache.org/news/2020/04/15/flink-serialization-tuning-vol-1.html#performance-comparison. Your results will certainly be different from those shown in that article -- serialization performance varies a lot depending on the details of what you're doing -- but the general patterns shown there are worth paying attention to.

Exactly how much you have to gain from worrying about all this is hard to say. You'd have to do your own benchmarking to know for sure. As for your question about static typing -- I'm not sure, but my guess is that with Kryo it won't make much difference.

David Anderson
  • 39,434
  • 4
  • 33
  • 60
0

Thanks @David Anderson, the artical is very useful. I used rows to achieve a better performance, it internally uses object []. There is no performance difference on static typed vs type object. As with kryo, it dosent make any difference

Keshore Durairaj
  • 27
  • 5