I have two gemfire (version 8.2) clusters (sender and receiver) connected through WAN. Sender cluster gets data from DB and transmits entries to Receiver through WAN gateway until both clusters are online. Sometime few cache nodes from Receiver cluster goes down due to environment issue that causes data loss in Receiver cluster and impacts my clients connected to it. We are using gemfire-toolkit to transmit all data from sender to receiver in case almost all data is missing, but this solution doesn't fit right if there is few data missing in Receiver cluster.
I have a requirement that if my client connected to receiver cluster is looking for an entry and that not available in cache, I want to load that entry from Sender cluster on the fly. Also I can't rely on gateway-sender as it will transmit data asynchronously according to configured batch-time-interval.
I am thinking to expose RESTful end-point in sender cluster and configure a cache loader in receiver to call that end-point, get data and put into the it's cluster.
Is there anyway I can ask Sender cluster to transmit required entry to receiver synchronously or immediately without exposing any web-service. Or is there any other best solution someone can propose me explore?