-1

If I have a cache (IMemoryCache) Dictionary<string, List>. At some point in my app, I have get the List for some Id from Cache, then I use this List to insert it in the DB. Sometimes this List could be changed by other task prior to inserting - that's throwing an exception. To avoid this as soon as I get the list from Cache, I'm making a copy of this. But sometimes the object in the List can be changed by other task prior to inserting - that's throwing exception. To avoid this I could create a copy for each object in the list and the creating a new list from copied objects, but that's kind of unappealing.

Maybe other solution would be to get the List from cache and create a lock and then inside the lock finish the inserting to DB flow. Are there any other solutions, or best approaches to this kind of issues?

Some explanation: Well this is a long-chain problem. I’m using Cassandra DB for storage. The Cache from above as well as Table in DB is related to message queue. If I'm removing a lot from DB (Cassandra) it generates a lot of tombstones in a table, and at some point if I pass the threshold it fails to query the table. I solved this by changing the structure of my table from 1 row per unread message to 1 row per user with a column list. But that's now creating problems with the Cache. Why using Cache? Well because Cassandra is really fast on write operations, but not really for read operations (if I have 10.000 opened web sockets to 2 server instances and 1 cassandra node, read/write just hangs and my servers goes into thread pool starvation continuously getting more threads but not doing any job), so I read once on startup and then only write data to Cassandra using the Lists from Cache rather than those from DB to avoid rereading them from DB. And by doing all this I run in problems with consistency, weird cache layer & async mutations.

Any advice will be helpful.

dtln812
  • 41
  • 5

2 Answers2

2

The obvious thing to me which you already stated is to make a copy of the List object so it's not getting changed while you're iterating over the list.

As a side note, I disagree with the assertion that Cassandra is not fast with reads. I work with hundreds of companies who use Cassandra and a lot of them have a 95% SLA on read latency between 6-8 milliseconds. Full disclosure: I'm a Cassandra enthusiast at DataStax.

Cassandra is designed for internet scale running on dozens of commodity servers. Best practice is to deploy a cluster with a minimum of 3 nodes with a replication factor of 3 for high availability and optimum performance. 2 app instances with a single-node C* backend isn't going to cut it because you are effectively doing a DDoS attack on the database. Cheers!

Erick Ramirez
  • 13,964
  • 1
  • 18
  • 23
  • You think that is the reason of pool starvation? I mean I have 10.000 opened web sockets on my server, that are processing 100.000 messages in a matter of seconds (if we take out the cache that's causing problems, then all of those messages in some way interact with the DB directly, write to 1 table, read from it, then add to another) is causing an overload on Cassandra? But why then I don't get timeouts from Cassandra, instead I get a server instance that generates thread continously, without doing any work? – dtln812 Aug 19 '21 at 08:06
  • Pool starvation was caused by something else, now it's either timeout or 'All hosts tried to query failed' so I guess 100.000 write/reads for a single node is too much. – dtln812 Aug 19 '21 at 09:51
  • A lot of it depends on the data model, access patterns, disk IO, etc. But like I said, it's the equivalent of DDoSing a single node. Time to horizontally scale. Cheers! – Erick Ramirez Aug 19 '21 at 10:06
2

If i understand your question right, you have a Dictionary<string, List> to store something as cache in the app. And the problem was the List inside that got change by another task prior to you make some action to interact with it.

If that so, i suggest you to take another approach to this. Even a ConcurrentDictionary<string, List> is not safe for your use case

For example:

You have a List that contain object A and B 2 task can still take your list out one after another. Task 1 add object C then call .Set(), Task 2 add object D then call .Set() on the IMemoryCache instance.

The result is likely list of A,B,C or A,B,D.

Another approach:

Use a static HashSet<string> to store your key, then, store each List by a key in the IMemoryCache, using ConcurrentQueue. Then every task can safely enqueue. When take out the data, I suggest you to take a look at take all data from ConcurrentQueue atomicity, then do whatever you want with it.

Gordon Khanh Ng.
  • 1,352
  • 5
  • 12
  • Yeah, but Queue means that I'm sure that first message that was added to queue, will be the first message to be taken out. It's the best case scenario, but what if user sends acknowledgement for 2 messages, and one message had some delay, ack for the second message arrives first, but I'm getting from queue first message and setting it as read. – dtln812 Aug 19 '21 at 08:08
  • I'm not quite get it here... if you need to modify data in your object in a queue, we can use linq, not only the classic `Enqueue` and `TryDequeue` way, just like the way you use to find and modify an object in your list. (If there's a better way, could you share (XD)). And it's safer even with linq cause we also got some guarantee since the it use the private implement version of `Enumerate` – Gordon Khanh Ng. Aug 19 '21 at 11:57