Kryo is a fast and efficient object graph serialization framework for Java. The goals of the project are speed, efficiency, and an easy to use API. The project is useful any time objects need to be persisted, whether to a file, database, or over the network.
Questions tagged [kryo]
431 questions
0
votes
1 answer
Spark Serializer Kryo setRegistrationRequired(false)
I'm using weka.mi.MISVM in a Scala/Spark program and need to serialize my kernels to reuse them later.
For this I use Kryo as this :
conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
conf.registerKryoClasses(
…
Heleo
- 25
- 8
0
votes
1 answer
Why does Apache Flink stall in method org.apache.flink.api.java.typeutils.runtime.kryo.Serializers.getContainedGenericTypes?
While implementing my algorithm I have created a long chain of operators in Apache Flink with for loop. Starting from some length processing stalls in method
org.apache.flink.api.java.typeutils.runtime.kryo.Serializers.getContainedGenericTypes for…
Vladimir Protsenko
- 1,046
- 2
- 14
- 21
0
votes
1 answer
How can I cache DataFrame with Kryo Serializer in Spark?
I am trying to use Spark with Kryo Serializer to store some data with less memory cost. And now I come across a trouble, I cannot save a DataFram e(whose type is Dataset[Row]) in memory with Kryo serializer. I thought all I need to do is to add…
Raul
- 866
- 7
- 4
0
votes
1 answer
kryo serialization - Error during Java serialization
My application has bunch of domain objects which get serialized into Redis store through spring-session. I'm trying to use Kryo (4.0.0) for automatic serialization without making objects explicitly serializable.
I'm getting the following error…
Charith De Silva
- 3,650
- 4
- 43
- 47
0
votes
1 answer
Akka Remoting: Custom Kryo serializer for org.opencv.core.Mat
I'm having trouble to implement a Kryo serializer for a org.opencv.core.Mat object representing e.g. a frame of a video file. The idea is that an Akka ActorSystem A sends a video frame in greyscale to a system B in order to detect objects in it…
Toaditoad
- 254
- 2
- 12
0
votes
1 answer
Hive - GenericUDTF - runQuery fails due to kryo stackoverflow exception
HiveServer2 ; Hive version: 1.2
I'm trying to run a query which contains a custom UDF class (that implements GenericUDTF).
The UDF class contains a tree object which it uses for its calculations.
When the tree is small, the query runs properly.…
urk
- 31
- 3
0
votes
1 answer
Kryo serialisation in Chronicle Map - slow byte reading
I'm using Chronicle Map extensively in Scala and recently decided to experiment with Kryo serialisation. I added custom marshallers (code below) and while it reduced the size of my stores by 14G (roughly 62%), and everything still works, the speed…
Anton
- 3,006
- 3
- 26
- 37
0
votes
1 answer
Kryo causing buffer underflow exception on read
I am serializing with:
private byte[] serialize()
{
KryoPool pool = new KryoPool.Builder(factory).softReferences().build();
Kryo kryo = pool.borrow();
Output output = new Output(0, 1024);
kryo.writeClassAndObject(output,…
raycons
- 735
- 12
- 26
0
votes
0 answers
Kryo Serialization Error in spark
I want to register kryo Serialization, for that i have written below code,
First Class (Select.Class)
var SparkConf = new SparkConf()
.set("spark.kryo.registrationRequired", "true")
.set("spark.kryo.registrator",…
Darshan Manek
- 155
- 1
- 2
- 12
0
votes
1 answer
Kryo java.lang.UnspportedOperationException
I am trying to use Kryo in spark for the SAMRecord class, but getting the error shown at runtime. I don't understand what this error means, can anyone explain? What do I need to change for Kryo to work with this? The problem is with the member…
pythonic
- 20,589
- 43
- 136
- 219
0
votes
1 answer
Incorrect column names in spark dataset
I have next case class:
case class Data[T](field1: String, field2: T)
I'm using kryo serializer with next implicits for it:
implicit def single[A](implicit c: ClassTag[A]): Encoder[A] = Encoders.kryo[A](c)
implicit def tuple2[A1, A2](implicit e1:…
Cortwave
- 4,747
- 2
- 25
- 42
0
votes
0 answers
Using kryo to custom serialize java object in kafka
I'm trying to send java object using kryo object serializer. I'm running into couple of issues. First, I'm not able to extend kryo to serialize a java bean, which is my payload. Here's the custom serializer.
package…
Shamik
- 1,671
- 11
- 36
- 64
0
votes
1 answer
com.esotericsoftware.kryo cross version compatibility
I have a few apps in data pipeline which use kafka as a queuing system.
If a producer app on java7 using kryo 2.22 produces to a kafka (java7) then would a consumer app on java8 using kryo 4.0 be able do deserialize the data ?
I short data…
Urvishsinh Mahida
- 1,440
- 16
- 23
0
votes
1 answer
How to read serialized object with a method taking generic type by Scala/Kryo?
Use Kryo to read serialized object is easy when I know the specific class type, but if I want to create a method that takes simple generic type, how to do it?
I have code that can not be compiled:
def load[T](path: String): T = {
val…
Stephen Kuo
- 1,175
- 3
- 11
- 19
0
votes
1 answer
Issue on using kryo serialization for spark
I am transforming a existing package to make it run on spark, in order to serialize the class in the third-party tools, I used the following code:
SparkConf conf = new SparkConf().setAppName("my.app.spark").setMaster("local").set("spark.serializer",…
MIn
- 3
- 3