Stack Overflow
Stack Overflow

Questions tagged [kryo]

Kryo is a fast and efficient object graph serialization framework for Java. The goals of the project are speed, efficiency, and an easy to use API. The project is useful any time objects need to be persisted, whether to a file, database, or over the network.

431 questions
4
votes
1 answer

Serializing and Deserializing Object Arrays in Kryo

I've been testing Kryo for serialization and deserialization recently and have generally been happy with it, however it is not clear to be how to handle the (de)serialization of a class which contains an object array. The class contains final…
Charles A
  • 404
  • 3
  • 10
4
votes
5 answers

Kryonet disconnects just after connecting

I followed this Youtube tutorial covering the basics of Kryonet. Basically it is a Kryonet Hello World, it explains how to setup a basic Server and a Client, allowing the Client to send Packets to the Server and have very basic communication. A link…
TMichel
  • 4,336
  • 9
  • 44
  • 67
4
votes
3 answers

How do I detect "End of File" in Kryo?

I'm reading an unknown number of objects from a gzipped file in Kryo: Input input = new Input(new GZIPInputStream(Files.newInputStream(inputFile))); The problem is: How do I detect when I've read the last object from the file? When I'm writing the…
sanity
  • 35,347
  • 40
  • 135
  • 226
3
votes
5 answers

How can I pass messages to another thread without using a blocking queue?

I have a pretty straight forward server (using kryonet). The clients are only storing the current state of the car (x,y,angle etc) and are sending requests of acceleration and turning. The server is receiving the requests and adding them to a…
Ólafur Waage
  • 68,817
  • 22
  • 142
  • 198
3
votes
0 answers

Spark Kryo Serialization

We have a Spark Structured Streaming application that consumes from a Kafka topic in Avro format. The payload is part of the state object in the mapGroupWithState function. Given that we enforce FULL compatibility for our Avro schemas, we generally…
Matthew Formosa
  • 468
  • 3
  • 9
3
votes
0 answers

Difference between Encoders.kryo[T] and Encoders.product[T] in Spark SQL

In Spark, I can do both: case class MyObj(val i: Int, val j: String) import org.apache.spark.sql.{Encoder, Encoders} implicit val myEncoder: Encoder[MyObj] = Encoders.product[MyObj] val ds = spark.createDataset(Seq(new MyObj(1, "a"),new MyObj(2,…
jack
  • 1,787
  • 14
  • 30
3
votes
1 answer

Impossible to operate on custom type after it is encoded? Spark Dataset

Say you have this (solution of encoding custom type is brought from this thread): // assume we handle custom type class MyObj(val i: Int, val j: String) implicit val myObjEncoder = org.apache.spark.sql.Encoders.kryo[MyObj] val ds =…
jack
  • 1,787
  • 14
  • 30
3
votes
1 answer

TinkerPop Gremlin How to serealize Predicate passed in Until step

I have a traversal where I need to repeat till no more new edges are found. In gremlin console/groovy I achieve this by passing a lambda that always returns false. g.V(7). repeat(out().not(hasLabel('region', 'business')).simplePath()). …
sas1138
  • 338
  • 5
  • 16
3
votes
1 answer

Spark Internal class Kryo registration

I'm new to Spark I'm using 2.4.4 with kryo. The spark job will write around 100 part files and then fails by throwing the following exception Caused by: java.lang.IllegalArgumentException: Class is not registered:…
Raj
  • 401
  • 6
  • 20
3
votes
1 answer

Question regarding kryo and java encoders in datasets

I am using Spark 2.4 and referring to https://spark.apache.org/docs/latest/rdd-programming-guide.html#rdd-persistence Bean class: public class EmployeeBean implements Serializable { private Long id; private String name; private Long…
Dev
  • 13,492
  • 19
  • 81
  • 174
3
votes
1 answer

Flink throws Kryo error due to Avro array types

I'm getting the following error from the getProducedType method in my Flink deserializer: com.esotericsoftware.kryo.KryoException: java.lang.NullPointerException Serialization trace: values (org.apache.avro.generic.GenericData$Record) at…
moku
  • 4,099
  • 5
  • 30
  • 52
3
votes
1 answer

Set spark.serializer to kryo with kryo.KryoException org.hibernate.LazyInitializationException

I get some entity from db with hibernate3 using Dao method(after the entity return, session is closed), and wrap it into a broadcast for all the executors. After setting spark.serializer to org.apache.spark.serializer.KryoSerializer, exception…
Fan Fan
  • 41
  • 3
3
votes
0 answers

Spark 2.2.0 intermittently produces `org.apache.spark.SparkException: Failed to register classes with Kryo` error

I'm running a Spark application in cluster mode via spark-submit, and I have a custom Kryo registrator. The JAR is built with sbt "+assembly". Occasionally, after letting the job run for a while, some of the executors will produce errors like the…
jbrock
  • 53
  • 7
3
votes
0 answers

spark custom kryo encoder not providing schema for UDF

When following along with How to store custom objects in Dataset? and trying to register my own kryo encoder for a data frame I face an issue of Schema for type com.esri.core.geometry.Envelope is not supported There is a function which will parse a…
Georg Heiler
  • 16,916
  • 36
  • 162
  • 292
3
votes
1 answer

Meaning of registering a class with kryo serialization

I don't understand what does it mean to register a class for kryo serialization. To give some background the link here says that Kryo does not support all Serializable types and requires you to register the classes you’ll use in the program in…
hesk
  • 317
  • 3
  • 11
1 2 3
…
28 29