NoHostAvailableException with 1000 concurrent request to Cassandra with DataStax Java Driver

Question

The setup:

-2-node Cassandra 2.0.7.31 cluster
-replicas=1
-With default configuration
-Using DataStax java driver 1.0

Activity Simple insert query using QueyBuilder class

Result

com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.181.13.239 ([/10.181.13.239] Unexpected exception triggered))
            at com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:64)
            at com.datastax.driver.core.ResultSetFuture.extractCauseFromExecutionException(ResultSetFuture.java:214)
            at com.datastax.driver.core.ResultSetFuture.getUninterruptibly(ResultSetFuture.java:169)
            at com.jpmc.es.rtm.storage.impl.EventExtract.main(EventExtract.java:36)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:601)
            at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.181.13.239 ([/10.181.13.239] Unexpected exception triggered))
            at com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:98)
            at com.datastax.driver.core.RequestHandler$1.run(RequestHandler.java:165)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)

are you able to connect to the cassandra server using cassandra-cli ? — Vishal John, Jul 18 '14 at 12:18
yes. I am getting around 85 TPS for upsert for 1000 concurrent hit . But I have more expectation from Cassandra regarding TPS. I am confused is that the datastax driver which is slowing down the whole thing? — soumyadeep sarkar, Jul 19 '14 at 07:12

score 1 · Accepted Answer · answered Jul 26 '14 at 11:52

Problem was that at my end I have created one thread one connection model. Which is quite synchronous. But Datstax driver works asynchronously(its using netty for that i guess) and taking multiple request on single connection. so in my case I one session for each request and guess what? I have left with pool of open connection b/w Driver and Cassandra. Server got choked so does the Driver. Problem was solved simply by letting Driver manage its pool.

Datastax java driver by default maintain minimum amount of connection for handling Certain amount of simultaneous Request.
Spec says that driver have a limitation of handling at the most 128 request per connection.so When we driver find that more than 128 request is coming Then it only open connection. So Driver is beautifully managing connection pool.[ http://www.datastax.com/drivers/java/2.0/com/datastax/driver/core/PoolingOptions.html] This link was helpful

score 0 · Answer 2 · answered Jul 18 '14 at 12:14

0

Go to your cassandra.yaml file. It will be present in the conf folder of your cassandra installation.

For listen_address give the IP address of the cassandra server. eg.

listen_address: 10.181.13.239

and then restart the cassandra server. Also check the port for native transport is not commented. There should be an entry like

native_transport_port: 9042

answered Jul 18 '14 at 12:14

Vishal John

4,231
25
41

Hi john thanks for the reply but. Its working well upto 1000 client or less. – soumyadeep sarkar Jul 19 '14 at 07:00
Actually what i want to perform a benchmark on Cassandra. I am trying to configure Cassandra so that it could give me fine TPS. I am using dataStax java driver for as a client . – soumyadeep sarkar Jul 19 '14 at 07:06
I'm using a similar environment and can easily hit > 10,000 TPS using more complex transactions. You should check your CPU and memory usage on your Cassandra instances and see if you are running into problems there. Also, the `Datastax Driver` is at version 2.0.3 already, I'd seriously consider upgrading. – Baldy Jul 19 '14 at 11:32
HI @Baldy Actually I have not change much of Cassandra.yaml and Cluster name and IPS. I am using two server with configuration **Intel Xeon 2 * quad Core/64 GB RAM/800 GB*6 HDD.**
do i need to change any thing in the default configuration on Cassandra.yaml.
– soumyadeep sarkar Jul 21 '14 at 08:54
I haven't changed any performance related defaults at all. Without in-depth knowledge of your environment, it's pretty tough to diagnose performance issues. I would upgrade to the latest Datastax driver and re-run your tests. There is a `cassandra-stress` tool available that would help eliminate the driver as your performance bottleneck. – Baldy Jul 21 '14 at 11:57
@Baldy thanks for the Suggestion.I have taken follow on it. Bye the way I have downloaded dse-4.0.3 from datastax web site . is it ok with it.[link]http://www.datastax.com/download#dl-enterprise – soumyadeep sarkar Jul 21 '14 at 13:06
@Baldy Actually I am new to this "NOSQL" world. Don't have much knowledge But people like are helping me to getting wiser day by day. Anyway few more question do have any idea how to increase connection pool size and how to stop the retry feature of Cassandra – soumyadeep sarkar Jul 21 '14 at 13:09
I have not used Datastax Enterprise so I can't comment there. There's a pretty good guide that talks about connection pools and tuning [link](http://planetcassandra.org/blog/post/guide-to-cassandra-thread-pools/). Retry policies are set on the individual queries [link](http://www.datastax.com/drivers/java/2.0/com/datastax/driver/core/policies/RetryPolicy.html). **This thread is getting off-topic so if you need help with something other than the original topic you should ask a new question.** – Baldy Jul 21 '14 at 15:38
@Baldy i got your point regarding this "original topic" stuff. lemme back to the point.. here as you have asked be i have updated the dataStax driver version. when putting load on it by jmeter I find the same problem i.e after after sending 1500 concurrent request. after 4/5 iteration . datastax driver is getting choked. By doing net stat i have observer that all the TCP connection from My datastax driver to the cassandra are in TIME_WAIT state.what could be the reason ? Is cassandra is holding up all the connection and not sending Ack or data ? Please help. – soumyadeep sarkar Jul 22 '14 at 12:21
Great news!. Was there anything specific you changed besides the driver upgrade? Just curious as in your previous comment you stated that you had upgraded the driver and had a similar performance issue. Either way I'd like to post a proper answer to close out the thread in case anyone else stumbles on a similar problem. – Baldy Jul 24 '14 at 23:51
@baldy actually I have also fixed the maximum and minimum pool size per node at a time for cassandra – soumyadeep sarkar Jul 25 '14 at 05:45
And now I am Using one session per cluster (in my case there is only one cluster). Earlier I was using one session per request which had created havoc.But now by only 19 connection Cassandra could easily handle 50000+ transaction. – soumyadeep sarkar Jul 25 '14 at 05:55
Great. You should post the details of your solution as an answer and accept it. I suspect the change in the session handling is what solved your problem, not the driver change. – Baldy Jul 25 '14 at 14:39

NoHostAvailableException with 1000 concurrent request to Cassandra with DataStax Java Driver

2 Answers2