Pivotal GemFire: PDX serializer config in Spring Data GemFire

Question

I have created a GemFire cluster with 2 Locators, 2 cache servers and a "Customer" REPLICATE Region. (Domain object class is placed in classpath during server startup).

I am able to run a Java program (Peer) to load the "Customer" Region in the cluster. Now we want to move to Spring Data GemFire where I am not sure how to configure PDX serialization and getting...

com.gemstone.gemfire.InternalGemFireException: java.io.NotSerializableException: com.gemfire.poc.DomainObjects.Customer

cache.xml in simple Java program...

<?xml version="1.0" encoding="UTF-8"?><cache xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://schema.pivotal.io/gemfire/cache" xsi:schemaLocation="http://schema.pivotal.io/gemfire/cache http://schema.pivotal.io/gemfire/cache/cache-8.1.xsd" version="8.1" lock-lease="120" lock-timeout="60" search-timeout="300" is-server="false" copy-on-read="false">
 <pdx>
    <pdx-serializer>
      <class-name>
       com.gemstone.gemfire.pdx.ReflectionBasedAutoSerializer
      </class-name>
    <parameter name="classes">
      <string>com.gemfire.poc.DomainObjects.*</string>
    </parameter>
  </pdx-serializer>
 </pdx>
    <region name="Customer" refid="REPLICATE">
    <region-attributes refid="REPLICATE" scope="distributed-no-ack"> 
     <cache-loader>
        <class-name>com.citigroup.pulse.pt.gemfire.poc.clientserver.SimpleCacheLoader</class-name>
    </cache-loader>
</region-attributes>
    </region>
</cache>

spring-context.xml in Spring Boot app...

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:gfe="http://www.springframework.org/schema/gemfire"
       xmlns:util="http://www.springframework.org/schema/util"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="
                http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
        http://www.springframework.org/schema/gemfire http://www.springframework.org/schema/gemfire/spring-gemfire.xsd
                http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util.xsd
                http://www.springframework.org/schema/data/gemfire 
                           http://www.springframework.org/schema/data/gemfire/spring-data-gemfire.xsd">

    <util:properties id="gemfireProperties">
        <prop key="log-level">config</prop>
        <prop key="locators">hostA[10334],hostB[10334]</prop>
    </util:properties>


    <bean id="mappingPdxSerializer" class="com.gemstone.gemfire.pdx.ReflectionBasedAutoSerializer"/>

    <gfe:cache use-bean-factory-locator="false" properties-ref="gemfireProperties" use-cluster-configuration="true" pdx-serializer-ref="mappingPdxSerializer" />

    <gfe:replicated-region id="Customer" ignore-if-exists="true">       
    </gfe:replicated-region>


</beans>

Can someone help me fix the serialization issue?

Caused by: com.gemstone.gemfire.InternalGemFireException: java.io.NotSerializableException: com.gemfire.poc.DomainObjects.Customer
    at com.gemstone.gemfire.distributed.internal.DistributionManager.putOutgoing(DistributionManager.java:1954)
    at com.gemstone.gemfire.internal.cache.DistributedCacheOperation.distribute(DistributedCacheOperation.java:476)
    at com.gemstone.gemfire.internal.cache.AbstractUpdateOperation.distribute(AbstractUpdateOperation.java:65)
    at com.gemstone.gemfire.internal.cache.DistributedRegion.distributeUpdate(DistributedRegion.java:519)
    at com.gemstone.gemfire.internal.cache.DistributedRegion.basicPutPart3(DistributedRegion.java:500)
    at com.gemstone.gemfire.internal.cache.AbstractRegionMap.basicPut(AbstractRegionMap.java:3052)
    at com.gemstone.gemfire.internal.cache.LocalRegion.virtualPut(LocalRegion.java:5838)

Precisely, how to add "classes" parameter of ReflectionBasedAutoSerializer in spring-data-gemfire tags?

PDX deserialization exception while retrieving value from Region:

com.gemstone.gemfire.ToDataException: PdxSerializer failed when calling toData on class javax.management.Notification
    at com.gemstone.gemfire.internal.InternalDataSerializer.writePdx(InternalDataSerializer.java:3130)
    at com.gemstone.gemfire.internal.InternalDataSerializer.writeUserObject(InternalDataSerializer.java:1520)
    at com.gemstone.gemfire.internal.InternalDataSerializer.writeWellKnownObject(InternalDataSerializer.java:1416)
    at com.gemstone.gemfire.internal.InternalDataSerializer.basicWriteObject(InternalDataSerializer.java:2208)
    at com.gemstone.gemfire.DataSerializer.writeObject(DataSerializer.java:3181)
    at com.gemstone.gemfire.internal.util.BlobHelper.serializeToBlob(BlobHelper.java:50)
    at com.gemstone.gemfire.internal.util.BlobHelper.serializeToBlob(BlobHelper.java:38)
    at com.gemstone.gemfire.internal.cache.UpdateOperation$UpdateMessage.toData(UpdateOperation.java:492)
    at com.gemstone.gemfire.internal.InternalDataSerializer.invokeToData(InternalDataSerializer.java:2407)
    at com.gemstone.gemfire.internal.InternalDataSerializer.writeDSFID(InternalDataSerializer.java:1378)
    at com.gemstone.gemfire.internal.tcp.MsgStreamer.writeMessage(MsgStreamer.java:239)
    at com.gemstone.gemfire.distributed.internal.direct.DirectChannel.sendToMany(DirectChannel.java:458)
    at com.gemstone.gemfire.distributed.internal.direct.DirectChannel.sendToOne(DirectChannel.java:310)
    at com.gemstone.gemfire.distributed.internal.direct.DirectChannel.send(DirectChannel.java:696)
    at com.gemstone.gemfire.distributed.internal.membership.jgroup.JGroupMembershipManager.directChannelSend(JGroupMembershipManager.java:2929)
    at com.gemstone.gemfire.distributed.internal.membership.jgroup.JGroupMembershipManager.send(JGroupMembershipManager.java:3163)
    at com.gemstone.gemfire.distributed.internal.DistributionChannel.send(DistributionChannel.java:79)
    at com.gemstone.gemfire.distributed.internal.DistributionManager.sendOutgoing(DistributionManager.java:3907)
    at com.gemstone.gemfire.distributed.internal.DistributionManager.sendMessage(DistributionManager.java:3948)
    at com.gemstone.gemfire.distributed.internal.DistributionManager.putOutgoing(DistributionManager.java:1951)
    at com.gemstone.gemfire.internal.cache.DistributedCacheOperation.distribute(DistributedCacheOperation.java:476)
    at com.gemstone.gemfire.internal.cache.AbstractUpdateOperation.distribute(AbstractUpdateOperation.java:65)
    at com.gemstone.gemfire.internal.cache.DistributedRegion.distributeUpdate(DistributedRegion.java:519)
    at com.gemstone.gemfire.internal.cache.DistributedRegion.basicPutPart3(DistributedRegion.java:500)
    at com.gemstone.gemfire.internal.cache.ProxyRegionMap.basicPut(ProxyRegionMap.java:242)
    at com.gemstone.gemfire.internal.cache.LocalRegion.virtualPut(LocalRegion.java:5838)
    at com.gemstone.gemfire.internal.cache.DistributedRegion.virtualPut(DistributedRegion.java:387)
    at com.gemstone.gemfire.internal.cache.LocalRegionDataView.putEntry(LocalRegionDataView.java:118)
    at com.gemstone.gemfire.internal.cache.LocalRegion.basicPut(LocalRegion.java:5228)
    at com.gemstone.gemfire.internal.cache.LocalRegion.validatedPut(LocalRegion.java:1599)
    at com.gemstone.gemfire.internal.cache.LocalRegion.put(LocalRegion.java:1582)
    at com.gemstone.gemfire.internal.cache.AbstractRegion.put(AbstractRegion.java:327)
    at com.gemstone.gemfire.management.internal.ManagementResourceRepo.putEntryInLocalNotificationRegion(ManagementResourceRepo.java:169)
    at com.gemstone.gemfire.management.internal.NotificationHub$NotificationHubListener.handleNotification(NotificationHub.java:193)
    at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor$ListenerWrapper.handleNotification(DefaultMBeanServerInterceptor.java:1754)
    at javax.management.NotificationBroadcasterSupport.handleNotification(NotificationBroadcasterSupport.java:275)
    at javax.management.NotificationBroadcasterSupport$SendNotifJob.run(NotificationBroadcasterSupport.java:352)
    at javax.management.NotificationBroadcasterSupport$1.execute(NotificationBroadcasterSupport.java:337)
    at javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:248)
    at com.gemstone.gemfire.management.internal.beans.ManagementAdapter.handleRegionRemoval(ManagementAdapter.java:879)
    at com.gemstone.gemfire.management.internal.beans.ManagementListener.handleEvent(ManagementListener.java:123)
    at com.gemstone.gemfire.distributed.internal.InternalDistributedSystem.notifyResourceEventListeners(InternalDistributedSystem.java:2252)
    at com.gemstone.gemfire.distributed.internal.InternalDistributedSystem.handleResourceEvent(InternalDistributedSystem.java:506)
    at com.gemstone.gemfire.internal.cache.LocalRegion.basicDestroyRegion(LocalRegion.java:6642)
    at com.gemstone.gemfire.internal.cache.DistributedRegion.basicDestroyRegion(DistributedRegion.java:1957)
    at com.gemstone.gemfire.internal.cache.LocalRegion.close(LocalRegion.java:2219)
    at org.springframework.data.gemfire.RegionFactoryBean.destroy(RegionFactoryBean.java:529)
    at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:272)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:578)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:554)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:961)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:523)
    at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.destroySingletons(FactoryBeanRegistrySupport.java:230)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:968)
    at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1032)
    at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1008)
    at org.springframework.context.support.AbstractApplicationContext$2.run(AbstractApplicationContext.java:929)
Caused by: org.springframework.data.mapping.model.MappingException: Could not write value for property protected transient java.lang.Object java.util.EventObject.source
    at org.springframework.data.gemfire.mapping.MappingPdxSerializer$2.doWithPersistentProperty(MappingPdxSerializer.java:188)
    at org.springframework.data.gemfire.mapping.MappingPdxSerializer$2.doWithPersistentProperty(MappingPdxSerializer.java:173)
    at org.springframework.data.mapping.model.BasicPersistentEntity.doWithProperties(BasicPersistentEntity.java:309)
    at org.springframework.data.gemfire.mapping.MappingPdxSerializer.toData(MappingPdxSerializer.java:173)
    at com.gemstone.gemfire.internal.InternalDataSerializer.writePdx(InternalDataSerializer.java:3075)
    ... 56 more
Caused by: com.gemstone.gemfire.pdx.PdxFieldAlreadyExistsException: The field "source" already exists.
    at com.gemstone.gemfire.pdx.internal.PdxType.addField(PdxType.java:262)
    at com.gemstone.gemfire.pdx.internal.PdxWriterImpl.updateMetaData(PdxWriterImpl.java:858)
    at com.gemstone.gemfire.pdx.internal.PdxWriterImpl.updateMetaData(PdxWriterImpl.java:851)
    at com.gemstone.gemfire.pdx.internal.PdxWriterImpl.writeObject(PdxWriterImpl.java:303)
    at com.gemstone.gemfire.pdx.internal.PdxWriterImpl.writeField(PdxWriterImpl.java:705)
    at com.gemstone.gemfire.pdx.internal.PdxWriterImpl.writeField(PdxWriterImpl.java:625)
    at org.springframework.data.gemfire.mapping.MappingPdxSerializer$2.doWithPersistentProperty(MappingPdxSerializer.java:184)
    ... 60 more

John Blum · Accepted Answer · 2018-05-12T20:35:49.043

0

You have a couple of options here, along with a few suggested recommendations.

1) First, I would not use Pivotal GemFire's o.a.g.pdx.ReflectionBasedAutoSerializer. Rather SDG has a much more robust PdxSerializer implementation based on Spring Data's Mapping Infrastructure (i.e. the o.s.d.g.mapping.MappingPdxSerializer).

In addition, SDG's MappingPdxSerializer allows you to register custom PdxSerializer's on an entity field/property case-by-case basis. Imagine if your Customer class has a reference to a complex Address class and that class has special serialization needs.

Furthermore, SDG's MappingPdxSerializer can handle transient and read-only properties.

Finally, you don't have to mess with any fussy/complex Regex to properly identify the application domain model types that need to be serialized.

2) Second, you can leverage Spring's JavaConfig along with SDG's new Annotation-based configuration model to configure Pivotal GemFire PDX Serialization as simply as this...

@SpringBootApplication
@PeerCacheApplication
@EnablePdx(..)
class MySpringBootApacheGeodeApplication {
  ...
}

That is, using the SDG @EnablePdx annotation.

More details on #1 and #2 above are available here and here.

Of course, the later is more applicable when using Pivotal GemFire 9.x+ with Spring Data GemFire Kay (2.0+). Judging by the package in your configuration of the com.gemstone.gemfire.pdx.ReflectionBasedAutoSerializer from your XML config (i.e. the com.gemstone.gemfire package) it would appear you are using Pivotal GemFire 8.2.x with Spring Data GemFire Ingalls (or 1.9.x.RELEASE), perhaps?

However, if you insist on, or are required to use XML for your configuration, then you can do the following...

<beans ...>

  <bean id="mappingPdxSerializer" class="org.springframework.data.gemfire.mapping.MappingPdxSerializer"/>

  <gfe:cache pdx-serializer-ref="mappingPdxSerializer" .../>

</beans>

And, if you really want to use Pivotal GemFire's ReflectionBasedAutoSerializer, then you can find examples of it's use in the SDG test suite. For instance.

I also have a few more examples in my spring-gemfire-test project/repo (which is quite a mess and I don't maintain this repo much anymore, as a warning). Examples here, using Java configuration with GemFire's API here, here, here which also shows the use of SDG's MappingPdxSerializer as well (by comparison), and so on and so forth. Many examples riddle throughout my repos.

Hope this helps!

Cheers, -John

edited May 12 '18 at 20:35

answered May 11 '18 at 21:51

John Blum

7,381
1
20
30

Thanks much for your reply. I am using Gemfire 8.2.7. On the server side, i have two xmls cache.xml and spring-context where PDX is mentioned in cache.xml. Spring-data-gemfire version is 1.6 as it came along with 8.2.7. @EnablePDX is not available in SDG 1.6 i think. How does mappingPdxSerializer know which object to be serialized?. – Sudharsan May 12 '18 at 16:42
I tried with mappingPdxSerializer in both cluster and my java program, it worked. I am able to persist the data into Region from my java peer. Thanks much. But i am getting following exception while deserializing and retrieving , Can you explain why?. Caused by: com.gemstone.gemfire.pdx.PdxFieldAlreadyExistsException: The field "source" already exists. at com.gemstone.gemfire.pdx.internal.PdxType.addField(PdxType.java:262) – Sudharsan May 12 '18 at 17:21
Update the post with full stack trace of exception. All i have done is created a gemfire-config jar with domain objects (POJO without @Region ) and using the same in both server and peer. – Sudharsan May 12 '18 at 17:33
Spring Data for Pivotal GemFire 1.6 reached EOL some time ago. The latest/current version of SDG supporting Pivotal GemFire 8.2.x is SDG 1.9.12.RELEASE, which is based on the latest version of Pivotal GemFire 8.2, i.e. 8.2.11. What happens when you try the newest versions? – John Blum May 12 '18 at 20:15
You may also use the following reference as a guide for evaluating the latest/current versions in order to stay up-to-date... https://github.com/spring-projects/spring-data-gemfire/wiki/Spring-Data-GemFire-to-GemFire-Version-Compatibility. – John Blum May 12 '18 at 20:16
On a related note, I think you could possibly be running into this problem (https://issues.apache.org/jira/browse/GEODE-4822). NOTE: Apache Geode is the open source core of Pivotal GemFire. Since there currently is no capability in SDG's `o.s.d.g.mapping.MappingPdxSerializer` to filter on what types actually are handled and get de/serialized (i.e. all types passing through the serializer as determined by GemFire/Geode will be de/serialized) I have been contemplating adding support for filtering types, such as by the package where your application entities live, for instance. – John Blum May 12 '18 at 20:21
NOTE: the non-type filtering capabilities in SDG's `MappingPdxSerializer` is unlike GemFire/Geode's `ReflectionBasedAutoSerialzier`, which does accept a Regex pattern to filter types that the serializer is directly responsible for handling. I am also uncertain whether 8.2.11 has a fix for GEODE-4822, or even whether it will. Please test with these versions though, i.e. SDG 1.9.12.RELEASE and Pivotal GemFire 8.2.11 (which is pulled in by SDG 1.9.12). In the meantime, I have filed SGF-745 (https://jira.spring.io/browse/SGF-745) for your reference. – John Blum May 12 '18 at 20:35
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/170922/discussion-between-sudharsan-and-john-blum). – Sudharsan May 12 '18 at 20:52

Pivotal GemFire: PDX serializer config in Spring Data GemFire

1 Answers1