Stack Overflow
Stack Overflow

Questions tagged [kryo]

Kryo is a fast and efficient object graph serialization framework for Java. The goals of the project are speed, efficiency, and an easy to use API. The project is useful any time objects need to be persisted, whether to a file, database, or over the network.

431 questions
10
votes
2 answers

Using an Collections$UnmodifiableCollection with Apache Flink

While using Apache Flink with the following code: DataStream> result = source.window(Time.of(1, TimeUnit.SECONDS)).mapWindow(new WindowMapFunction>() { @Override public void mapWindow(Iterable…
Robert Metzger
  • 4,452
  • 23
  • 50
10
votes
2 answers

kryo list serialization

I am trying to serialize a List of List of some objects (of a customized class: List> ), using Kryo. list2D; // List> which is already produced. Kryo k1 = new Kryo(); Output output = new Output(new…
MAZDAK
  • 573
  • 1
  • 4
  • 16
9
votes
1 answer

Kryo serialization refuses to register class

I'm trying to use kryo serialization with: kryo.setRegistrationRequired(true); I keep getting the following error saying that a certain class is not registered: java.lang.IllegalArgumentException: Class is not registered:…
kbirk
  • 3,906
  • 7
  • 48
  • 72
8
votes
2 answers

Spark Kryo register for array class

I am trying to register a class with array (Spark Java with Kryo activated), log shows a clear message: Class is not registered: org.apache.spark.sql.execution.datasources.InMemoryFileIndex$SerializableBlockLocation[] I have written several…
MrElephant
  • 302
  • 4
  • 26
8
votes
1 answer

Understanding Kryo serialization buffer overflow error

I am trying to understand the following error and I am running in client ode. org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow. Available: 0, required: 61186304. To avoid this, increase spark.kryoserializer.buffer.max…
soupybionics
  • 4,200
  • 6
  • 31
  • 43
8
votes
2 answers

Spark with Avro, Kryo and Parquet

I'm struggling to understand what exactly Avro, Kryo and Parquet do in the context of Spark. They all are related to serialization but I've seen them used together so they can't be doing the same thing. Parquet describes its self as a columnar…
Matthew Jones
  • 944
  • 9
  • 17
8
votes
1 answer

Handling case classes in twitter chill (Scala interface to Kryo)?

Twitter-chill looks like a good solution to the problem of how to serialize efficiently in Scala without excessive boilerplate. However, I don't see any evidence of how they handle case classes. Does this just work automatically or does something…
Urban Vagabond
  • 7,282
  • 3
  • 28
  • 31
8
votes
1 answer

How do you deserialize immutable collection using Kryo?

How do you deserialize immutable collection using Kryo ? Do I need to register something in addition to what I've done ? Here is my sample code import com.esotericsoftware.kryo.Kryo import com.esotericsoftware.kryo.io.Input import…
expert
  • 29,290
  • 30
  • 110
  • 214
7
votes
0 answers

Problem with Java Iterator created by Jackson breaks Scala Flink application

There is a Scala Flink application where I am parsing JSON using Jackson library. Parsing is handled by a custom method and it uses the lazy initiation concept to keep it fast. Now, for whatever reason, passing model with lazy values further in…
Atais
  • 10,857
  • 6
  • 71
  • 111
7
votes
1 answer

KryoException: Unable to find class with spark structured streaming

1-The Problem I have a Spark program that make use of Kryo but not as part of the Spark Mechanics. More specifically I am using Spark Structured Streaming connected to Kafka. I read binary values coming from Kafka and decode it on my own. I am…
MaatDeamon
  • 9,532
  • 9
  • 60
  • 127
7
votes
2 answers

Flink 1.4 AvroUtils error

I try to submit a job on Flink 1.4 and getting the following exception. Any idea how to solve the problem? Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed. at…
Nick Toker
  • 101
  • 4
7
votes
1 answer

Kryo Serialization for Spark 2.x Dataset

Is Kryo serialization still required when working with the Dataset API? Because Datasets use Encoders for or serialization and deserialization: Does Kyro serialization even work for Datasets? (Provided the right config is passed to Spark, and…
y.mazari
  • 374
  • 2
  • 8
7
votes
2 answers

Kryo: deserialize old version of class

I need to modify a class by adding two new parameters. This class is serialized with Kryo. I'm currently persisting the information related to this class, among other things, as an RDD, every time I stop my stream. When I restart the stream I load…
Alessandro Grassi
  • 165
  • 2
  • 12
7
votes
2 answers

How to register byte[][] using kryo serialization for spark

I am trying to fully utilize kryo serialization for spark. Setting .set("spark.kryo.registrationRequired", "true") This will let me know which classes need to be registered. I have registered about 40 classes, some of my classes and some of spark's…
John Engelhart
  • 279
  • 4
  • 13
7
votes
1 answer

Kryo serialization dependent on Java version?

When deserializing a serialized object (from a file) using Kryo, I get the following exception: java.lang.ExceptionInInitializerError (...) Caused by: com.esotericsoftware.kryo.KryoException: (...) Serialization trace: (...) at…
Tom
  • 1,375
  • 3
  • 24
  • 45
1
2
3
…
28 29