4

I have a pool of persistent connections(Memcached clients). Data are being cached in the memcached server. If after restarting the memcached server, I try to get the cached data using the client from the pool, I m getting the below exception:

java.util.concurrent.ExecutionException: java.lang.RuntimeException: Cancelled
    at net.spy.memcached.MemcachedClient$OperationFuture.get(MemcachedClient.java:1662)
    at net.spy.memcached.MemcachedClient$GetFuture.get(MemcachedClient.java:1708)
    at com.eos.gds.cache.CacheClient.get(CacheClient.java:49)

I get this exception only for the first time after the restart when I try to get the cached data. I did a lot of search. But unable to find the exact reason for this.

Amit Daga
  • 127
  • 4
  • 13

5 Answers5

4

Spymemcached has a bunch of internal queues that operations are placed in before they are actually sent out to memcached. What is happening here is that you do an operation and then before that operation is sent over the wire or before a response is received from memcached, Spymemcached realizes that the connection has been lost. As a result Spymemcached cancels all operations in flight and then reestablishes the connection.

When you call get() on the Future then since the operation was cancelled by Spymemcached an exception is thrown. What I recommend doing here is catching all exceptions on every individual operation you do with Spymemcached and then, depending on the error, either retrying the operation of just forgetting about it. If it's a get for example and your cluster of memcached servers goes down then you can probably forget about it since the cache will be empty, but you will probably want to retry a set.

mikewied
  • 5,273
  • 1
  • 20
  • 32
  • I understood the part that Spymemcached maintains a queue. So, all the operations are cancelled by Spymemcached. I m also catching exceptions and cancelling future `future.cancel(false)` so that there's nothing in queue when the memcached server is up. But what about the operation which takes place just after the server is up? Is it giving exception because that persistent client lost the connection and needs to connect again. – Amit Daga Nov 10 '11 at 05:50
  • Hi Mikewied, could I know why sypmemecached cancels all operation in flight? should it establish the connection and try the request again? – Xilang Aug 10 '12 at 06:01
  • I believe you can set OperationState.RETRY in the connection factory builder if you want the operation to be sent again. Operations are likely cancelled by default because if an operation fails the user might not really care since doing the operation again would take too long for the application. – mikewied Aug 13 '12 at 02:50
  • @mikewied Do you know the how many times an operation is retried if we enable that setting on the ConnectionFactory? – saibharath May 21 '20 at 14:03
1

I ran into the exact same problem and fix it by handling the exception until success

while(true){
 try{
  memcacheclient.get(key);
  break;
 }
 catch(java.util.concurrent.CancellationException e ){
  log.info("cache cancelled");
 }
}
Long_Le
  • 76
  • 1
  • 4
  • 1
    this solution looks good and it works for this issue, but what is the underlying problem? network connection lost? unreliable connection ? is there a way to find out? – naresh goty Dec 08 '19 at 20:36
1

Run MemcachedClient.getStats() for each new client once and that will fix the cancelations issue.

AlexS
  • 519
  • 5
  • 11
0

I had the same issue. I am using Spymemcached client to connect with Memcache server.

I found this.

There must be a connection issue.

Ref: https://github.com/couchbase/spymemcached/blob/master/src/main/java/net/spy/memcached/internal/OperationFuture.java

Nitin
  • 2,701
  • 2
  • 30
  • 60
0

Been searching for days for a solution. Posting in case it helps someone else.

Our implementation of ServletContextListener was getting a new MemcachedClient(...) on contextInitialized, and would then call the MemacachedClient method shutdown() on contextDestroyed. I would always get a CancellationException or ExecutionException on the first request I would send. (The error messaging alluded to both, but an ExecutionException is what I was able to catch.)

Solution: switched from shutdown() to shutdown(1, TimeUnit.SECONDS)

Now the get call succeeds the very first time that it is run.

I cannot explain for sure how the contextDestroyed call was interfering with the regular handling of the request. My best guess is that spymemcached's single thread somehow gets shared between servlets, and so when a servlet was created to handle a request sent by a verification step of our build process, it would get destroyed prior to the first request I would send, and the MemcachedClient my request's servlet was using would then try to use that same thread and get hit with the exceptions from the shutdown.

(Our team had established the need to call shutdown a while back when we learned our web app had too many open connections to our memcached server.)

Laura
  • 525
  • 1
  • 4
  • 12
  • Keep in mind that closing and opening a connection for every request can be taxing for the server too if your tps(transactions per second) is too high. – saibharath May 21 '20 at 14:02