Kryo is a fast and efficient object graph serialization framework for Java. The goals of the project are speed, efficiency, and an easy to use API. The project is useful any time objects need to be persisted, whether to a file, database, or over the network.
Questions tagged [kryo]
431 questions
5
votes
1 answer
Hazelcast 3.6.1 "There is no suitable de-serializer for type" exception
I am using Hazelcast 3.6.1 to read from a Map. The object class stored in the map is called Schedule.
I have configured a custom serializer on the client side like this.
ClientConfig config = new ClientConfig();
SerializationConfig sc =…
Ayub Malik
- 2,488
- 6
- 27
- 42
5
votes
1 answer
How to serialize Joda-Time's LocalDate with Kryo
I am trying to serialize with Kryo a class containing a LocalDate from Joda-Time.
It looks like I'll need a custom-made serializer.
I found some Joda-Time serializers in https://github.com/magro/kryo-serializers but none for LocalDate.
Any help?…
Hugo Zaragoza
- 574
- 8
- 25
5
votes
0 answers
Why is InputStream constructed from Channel much faster than one constructed from FileDescriptor?
When constructing an InputStream on a RandomAccessFile in order to have Kryo deserialize objects from it, it seems it makes a HUGE difference for performance whether one constructs the mediating InputStream through the file's Channel (gives good…
Jimmy Jam
- 71
- 3
5
votes
0 answers
How to get the serialized size fast?
Is there any way to find out how big will be the byte array returned after serializing an object with Kryo? Or do I have to actually serialize it and then check the array's length?
For instance, with LZ4, before compressing a byte array, I can get a…
Eduardo Bezerra
- 1,923
- 1
- 17
- 32
4
votes
1 answer
Increase buffer size in server/client in Kryonet
How do I increase the buffer size in Server/Client constructor in Kryonet?
nebula
- 3,932
- 13
- 53
- 82
4
votes
0 answers
Task Not Serializable - Serialization behavior Change between Spark 2.4 and Spark 3.x
We are migrating our spark jobs from 2.4.5 to 3.1.1 (scala 2.11.11 to scala 2.12.13)
We are facing the following exception
org.apache.spark.SparkException: Task not serializable
at…
cnemri
- 454
- 4
- 14
4
votes
3 answers
Kryo encoder v.s. RowEncoder in Spark Dataset
The purpose of the following examples is to understand the difference of the two encoders in Spark Dataset.
I can do this:
val df = Seq((1, "a"), (2, "d")).toDF("id", "value")
import org.apache.spark.sql.{Encoder, Encoders, Row}
import…
jack
- 1,787
- 14
- 30
4
votes
1 answer
Configure function/lambda serialization in Spark
How do I configure Spark to use KryoSerializer for lambdas? Or have I found a bug in Spark? We have no issue with the serialization of the data elsewhere, just these lambdas where it uses the default instead of Kryo.
Here's the…
Mohannad
- 124
- 7
4
votes
1 answer
How to register kryo classes in the spark-shell
The SparkConf has the method registerKryoClasses:
def registerKryoClasses(classes: Array[Class[_]]): SparkConf = { .. }
However it is not available/exposed in the RuntimeConfiguration facade provided by the SparkSession.conf() attribute
@transient…
WestCoastProjects
- 58,982
- 91
- 316
- 560
4
votes
1 answer
Registering Classes with Kryo via SparkSession in Spark 2+
I'm migrating from Spark 1.6 to 2.3.
I need to register custom classes with Kryo. So what I see here: https://spark.apache.org/docs/2.3.1/tuning.html#data-serialization
val conf = new…
kmh
- 1,516
- 17
- 33
4
votes
1 answer
Spark dataset encoders: kryo() vs bean()
While working with datasets in Spark, we need to specify Encoders for serializing and de-serializing objects. We have option of using Encoders.bean(Class) or Encoders.kryo(Class).
How are these different and what are the performance…
Abhay Dubey
- 549
- 2
- 7
- 18
4
votes
1 answer
How to register InternalRow with Kryo in Spark
I want to run Spark with Kryo serialisation. Therefore I set spark.serializer=org.apache.spark.serializer.KryoSerializer and spark.kryo.registrationRequired=true
When I then run my code I get the error:
Class is not registered:…
Joha
- 935
- 12
- 32
4
votes
0 answers
How can TypeTag be serialized using kryo and chill?
I use chill-scala for serialization. I want to serialize a generic class that stores type parameter information so I pass TypeTag[T] as an implicit parameter:
import com.twitter.chill.{Output, ScalaKryoInstantiator}
import…
gadubishe
- 65
- 1
- 5
4
votes
1 answer
Does kryo serialization work on non serializable class and class have non serializable attributes?
I'm trying to use Kryo library to convert any given object to byteArray and store in a data store or queue for later use. But is it possible to serialize any given object or only object implementing serializable interface can be converted.
Sharath
- 407
- 5
- 16
4
votes
1 answer
Spark, Kryo Serialization Issue with ProtoBuf field
I am seeing an error when running my spark job relating to Serialization of a protobuf field when transforming an RDD.
com.esotericsoftware.kryo.KryoException: java.lang.UnsupportedOperationException
Serialization trace:
otherAuthors_…
eboni
- 883
- 2
- 10
- 25