Stack Overflow
Stack Overflow

Questions tagged [kryo]

Kryo is a fast and efficient object graph serialization framework for Java. The goals of the project are speed, efficiency, and an easy to use API. The project is useful any time objects need to be persisted, whether to a file, database, or over the network.

431 questions
7
votes
1 answer

How to let Spark serialize an object using Kryo?

I'd like to pass an object from the driver node to other nodes where an RDD resides, so that each partition of the RDD can access that object, as shown in the following snippet. object HelloSpark { def main(args: Array[String]): Unit = { …
Tao Xiao
  • 259
  • 1
  • 4
  • 10
6
votes
2 answers

Does Kryo help in SparkSQL?

Kryo helps improve the performance of Spark applications by the efficient serialization approach. I'm wondering, if Kryo will help in the case of SparkSQL, and how should I use it. In SparkSQL applications, we'll do a lot of column based operations…
user6502167
  • 731
  • 9
  • 18
6
votes
0 answers

Kryo deserialization Buffer underflow when class adds new field

I have a java class serialized using kryo-shaded 4.0.1 class definition : class A{ private int c1; private Map c2; } After serialization when I deserialized its fine,But if If after serilization using existing definition and…
user1846749
  • 2,165
  • 3
  • 23
  • 36
6
votes
1 answer

Why does Kafka Direct Stream create a new decoder for every message?

I have a Spark streaming app written in Java and using Spark 2.1. I am using KafkaUtils.createDirectStream to read messages from Kafka. I am using kryo encoder/decoder for kafka messages. I specified this in Kafka properties-> key.deserializer,…
scorpio
  • 329
  • 1
  • 18
6
votes
3 answers

spark - How to reduce the shuffle size of a JavaPairRDD?

I have a JavaPairRDD on which I want to perform a groupByKey action. The groupByKey action gives me a: org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output location for shuffle which is practically an…
vefthym
  • 7,422
  • 6
  • 32
  • 58
6
votes
2 answers

Kryo serializer causing exception on underlying Scala class WrappedArray

Two questions, the answer to the general one will guide me on how minimal I can make a MVCE. 1) How can I know to register WrappedArray up front, (and every other class in Scala I might use)? Is it normal to have to register classes from libraries…
JimLohse
  • 1,209
  • 4
  • 19
  • 44
6
votes
1 answer

java.lang.StackOverflowError when using Kryo to serialize objects with references to each other

I have a graph-like object that I'm sending from server to client that contains nodes that have adjacencyLists. I have something similar to this: Clearing c1 = new Clearing(1, 134, 151); Clearing c6 = new Clearing(6, 250,…
Abe Fehr
  • 729
  • 9
  • 23
6
votes
0 answers

Class not found Kryo exception in hive 0.13 - Hadoop

I have a GenericUDF (see code below) that was running fine on Hadoop-1 and Hive-0.12. But when testing the same GenericUDF using Hive-0.13 + Hadoop-2, I am getting the below error. Vertex failed, vertexName=Map 12,…
user3072054
  • 339
  • 2
  • 6
  • 17
6
votes
2 answers

How can I use Kryo to serialize an object and deserialize it again?

How can I use Kryo to serialize an object and deserialize it again? I am working in Kryo 2.23.0
mcfly soft
  • 11,289
  • 26
  • 98
  • 202
6
votes
1 answer

Java serialization, Kryo and the object graph

Lets say I have an array arr of objects of type A in memory, each of which has a reference field pointing to the same object B. Illustration: A_1 A_2 A_3 ... A_N | | | | | | V | \--->\--> B <-----/ Note that the…
axel22
  • 32,045
  • 9
  • 125
  • 137
6
votes
2 answers

Generic Java serialization/deserialization using Kryo

I am trying to serialize and deserialize objects of a custom class (say, SomeClass, having a default no-args constructor) to a byte[] array, using Kryo 2.19 and the default serializer (FieldSerializer). Serialization seems to work OK, but I get…
PNS
  • 19,295
  • 32
  • 96
  • 143
5
votes
2 answers

Serialize multiple objects into a single file using Kryo

As far I know, Kryo serialization / deserialization happens per object. Is it possible to serialize multiple objects into a single file?. One of workaround suggested in another similar SO question was to use an array of objects. Considering a huge…
Harsha Hulageri
  • 2,810
  • 1
  • 22
  • 23
5
votes
1 answer

Spark: Dataframe Serialization

I have 2 questions regarding Spark serialization that I can simply find no answers to by googling. How can I print out the name of the currently used serializer; I want to know whether spark.serializer is Java or Kryo. I have the following code…
user1888243
  • 2,591
  • 9
  • 32
  • 44
5
votes
1 answer

How to set Unmodifiable collection serializer of Kryo in Spark code

I am using Kryo serialization in Spark (v1.6.1) in Java and while serializing a class which has a collection in its field, it throws the following error - Caused by: java.lang.UnsupportedOperationException at…
Pooja Mazumdar
  • 223
  • 2
  • 14
5
votes
2 answers

How to change default Serializer for an Akka application?

I read the akka serialization page, where they talk about serialization-bindings as serialization-bindings { "java.lang.String" = java "docs.serialization.Customer" = java "com.google.protobuf.Message" = proto …
daydreamer
  • 87,243
  • 191
  • 450
  • 722
1 2
3
…
28 29