0

Hello and thanks in advance!

Question In short:

Is there embedded replicated map solution with 'read your writes' guaranties and async updates propagation of frequently changed data?

Description:

System comprised with two services (just for example):

  1. One 'importer' - receives data from external system, updates system's data by merging received with current, and propagates it to 'user_api' services.
  2. N 'user_api' services, that frequently provides data for users.

I'm tring improve response time of 'user_api' by holding up-to-date data (2GB on average) within in-memory replicated map (history data rarely requested and MySQL used). I need only put/get/remove operations.

I've tried Hazelcast's ReplicatedMap where each service embedded with Hazelcast instance. Also I add separate not embedded Hazelcast instance (using default docker image) to have ability restart entire cluster without data losing (preventing initial fillup from MySQL). And it works almost like a charm. BUT there seems to be one strong downside - it slowdown import process within 'importer' by blocking on write operations.

What I need it is speed of local Map read/writes with async updates propagation to other cluster nodes. I need 'read your writes' guaranties and async changes propagation to 'user_api' services.

I found discussion here: Replicated caching solutions compatible with AWS. The difference: the questioner was not interested in the case with frequent updates.

I read a bit about Ignite and seems like it works the same as Hazelcast for this case. About the Geode: could not understand whether he fits the described case by quickly going over the review - need more time to investigate.

Wrapping up questions:

  • Is there replicated map solution with 'read your writes' guaranties and async updates propagation of frequently changed data?
  • It also be cool if there is of-the-shef possibility of not loosing data when all apps restarted (for example, by holding separate not embedded instance by using default docker image or periodical disk backups).
  • Maybe I even miss some point from Hazelcast docs?
Timur Efimov
  • 358
  • 2
  • 10

1 Answers1

0

I think Ignite would do what you need. There’s a cache property called write synchronisation mode. The default is PRIMARY_SYNC, which writes synchronously to the primary and asynchronously to the backups. There’s also FULL_SYNC and FULL_ASYNC.

Ignite works the same way as Hazelcast, in the sense that you can have a separate cluster rather than embedding your nodes. Ignite also has persistence, so you could restart the cluster as well and still not lose data.

Stephen Darlington
  • 51,577
  • 12
  • 107
  • 152
  • Thanks. It sounds really cool about ability for persisting data to disk. A little question: As I understand your correctly, I can use `IgniteCache` with `CacheMode.REPLICATED` and `CacheWriteSynchronizationMode.FULL_ASYNC`. But at this case, do I have 'read your writes' guaranties within the same app (eg 'importer')? – Timur Efimov Nov 05 '20 at 16:05
  • Or may be there is ability to force bind primary partitions to 'importer' (and re-bind on restart)? And so we can use `PRIMARY_SYNC`. – Timur Efimov Nov 05 '20 at 16:09
  • You probably want `PRIMARY_SYNC`. Reads/writes typically go to the primary. Binding data to a specific node is possible, but generally not the right thing to do. – Stephen Darlington Nov 05 '20 at 17:29
  • The reason - almost all write operations performed by 'importer' (~99.9%). So, is seems like it is good to reduce writes cost in this case. And it is sounds good that it is possible! I will try to implement it as soon as possible and will come back. (Few days) – Timur Efimov Nov 05 '20 at 18:02
  • Hi! Perhaps you faced the problem of using the same classes from different packages? If so, please take a look: https://stackoverflow.com/questions/64728898/for-ignite-how-can-i-use-the-same-classes-in-different-packages-without-overrid – Timur Efimov Nov 07 '20 at 14:55
  • Ignite seems to be really perfect, but due to lack of time, unfortunately, I could not complete the version of the system using it. At the moment, I was left with a solution involving Hazelcast for 'user_api' and Redis for 'importer'. – Timur Efimov Nov 17 '20 at 06:10