0

I am looking to use the Kryo serialization library (https://github.com/EsotericSoftware/kryo) with Pivotal Gemfire 8.x onwards. However, there is not much information available on how to do that. If anyone has a code/config snippet to show how to integrate Kryo serialization with Pivotal Gemfire or Apache Geode that will be very useful.

Thanks.

jcools
  • 71
  • 4

2 Answers2

1

One option is just to store serialized bytes in a geode region, eg

 byte[] bytes = serializeWithKyro(myObject)
 region.put("key", bytes)

Another option would be to register a DataSerializer that can serialize with Kyro:

DataSerializer.register(KyroSerializer.class)

public class KyroSerializer extends DataSerializer {
    public boolean toData(Object o, DataOutput out) {
        if(!(o instanceof MyClass)) {
            return false;
        }
        writeWithKyro(o, out);
    }
    ...

}

You can check out the DataSerializer javadocs for more info.

Dan Smith
  • 481
  • 2
  • 3
  • Thanks for your reply Dan. I am still not very clear, how this will get invoked by Gemfire when storing the Java objects in Regions? How does Gemfire knows that it is supposed to use the custom KryoSerializer class above? Also, does all the Java classes have to implement the DataSerializer interface now? If I simply override the "readObject" and "writeOjbect" API's (private methods invoked by Java Serialization), wouldln't that will work transparentlly? – jcools Nov 19 '17 at 19:44
  • Here is a link to another post where the original question talks about implementing Kryo serialization with Gemfire: https://stackoverflow.com/questions/22396122/gemfire-custom-serialization-not-helping But, I have been told to ask a new question there, so posted it here and linking to the above thread for reference. – jcools Nov 19 '17 at 19:50
  • When you put an object into a gemfire region, that object may need to be serialized and sent to other members. That serialization is controlled by gemfire. It will first look for any registered DataSerializers, like this KyroSerializer. If you have a registered DataSerializer like this, your object doesn't need to implement DataSerializable. Gemfire will just call that toData method on KyroSerializer, which will handle serializing the object. – Dan Smith Nov 28 '17 at 18:08
  • I read the [DataSerializer javadocs](https://geode.apache.org/releases/latest/javadoc/org/apache/geode/DataSerializer.html) and as per the API documentation, I have to specify which Custom classes I want to use Kryo in the API: public Class>[] getSupportedClasses(). This seems very cumbersome because I have almost 70 regions and if there was a way to simply provide a generic way for all Custom class types, then that would have been a lot more easier. – jcools Dec 15 '17 at 02:59
  • Also, strangely, the "getId()" returns a single integer - so I am unable to figure how that will correspond to the array of Class objects from getSupportedClasses() api? Any thoughts? – jcools Dec 15 '17 at 03:04
  • Gemfire/Geode provides no way to replace it's own serialization for several standard JDK classes, and therefore we end up having a mixed bag, i.e. Kryo and Gemfire DataSerialization. This results in compatibility issues with the serialized data. – jcools Dec 27 '17 at 23:44
0

Gemfire/Geode provides no way to replace it's own serialization for several standard JDK classes, and therefore we end up having a mixed bag, i.e. Kryo and Gemfire DataSerialization. This results in compatibility issues with the serialized data.

I tested by registering a few domain classes with Kryo serializer, and it started giving me issues during deserialization on client side. See my other thread on this issue.

I am not sure if anyone had success with integrating Kryo and Apache Geode/GemFire, but if you did, please let me know how you did it.

jcools
  • 71
  • 4