Astyanax not respecting configured socket timeout

Question

This is my setup:

4 nodes Cassandra 1.2.19
Astyanax 1.56.49

I am setting configuration like

AstyanaxContext<Keyspace> context = new AstyanaxContext.Builder()
    .forCluster(service.getClusterName())
    .forKeyspace(service.getKeySpaceName())
    .withAstyanaxConfiguration(new AstyanaxConfigurationImpl()
         .setDiscoveryType(NodeDiscoveryType.NONE)
         .setCqlVersion("3.0.0")
         .setDefaultReadConsistencyLevel(consistencyLevel.getAstyanaxValue())
         .setDefaultWriteConsistencyLevel(consistencyLevel.getAstyanaxValue())
    )
    .withConnectionPoolConfiguration(new ConnectionPoolConfigurationImpl("b2bConnectionPool")
         .setPort(service.getPort())
         .setMaxConnsPerHost(5)
         .setSeeds(StringUtils.join(hosts, ","))
         // increase default timeout for heavy operations (milliseconds)
         .setSocketTimeout(15000)
         .setSSLConnectionContext(sslContext)
         .setAuthenticationCredentials(credentials)
    )
    .withConnectionPoolMonitor(new CountingConnectionPoolMonitor())
    .buildKeyspace(ThriftFamilyFactory.getInstance());

Now there is a reproducible query that takes a long time and finally throws a OperationTimeoutException:

com.netflix.astyanax.connectionpool.exceptions.OperationTimeoutException: OperationTimeoutException: [host=myhost(myip):13260, latency=10001(40007), attempts=4]TimedOutException()
    at com.netflix.astyanax.thrift.ThriftConverter.ToConnectionPoolException(ThriftConverter.java:171) ~[astyanax-thrift-1.56.49.jar:na]
    at com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:65) ~[astyanax-thrift-1.56.49.jar:na]
    at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$1$2.execute(ThriftColumnFamilyQueryImpl.java:190) ~[astyanax-thrift-1.56.49.jar:na]
    at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$1$2.execute(ThriftColumnFamilyQueryImpl.java:182) ~[astyanax-thrift-1.56.49.jar:na]
    at com.netflix.astyanax.thrift.ThriftSyncConnectionFactoryImpl$ThriftConnection.execute(ThriftSyncConnectionFactoryImpl.java:151) ~[astyanax-thrift-1.56.49.jar:na]
    at com.netflix.astyanax.connectionpool.impl.AbstractExecuteWithFailoverImpl.tryOperation(AbstractExecuteWithFailoverImpl.java:119) ~[astyanax-core-1.56.49.jar:na]
    at com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:338) ~[astyanax-core-1.56.49.jar:na]
    at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$1.execute(ThriftColumnFamilyQueryImpl.java:180) ~[astyanax-thrift-1.56.49.jar:na]

The exception message says "latency=10001" and I thought this should be the socket timeout configured to 15000 ms but it's obviously not. How can I increase the timeout for a query operation in astyanax?

I couldn't reproduce the issue with cqlsh but I could reproduce the query with a small java program. I will try out the config changes mentioned in the answers and give feedback afterwards. — Steffen Harbich, Aug 16 '16 at 08:31

Ajay Deshwal · Answer 1 · 2016-08-15T13:32:02.060

I believe you are not encountering a socket timeout, rather an rpc timeout which i think cannot be controlled through Astyanax. In your stack trace, the exception wrapped inside OperationTimeoutException is TimedOutException which is thrown when an rpc timeout occurs (default 10 seconds). In case of a socket timeout, the SocketTimeoutException will be thrown.

Try executing your query with cqlsh and you will get a Request did not complete within rpc_timeout message.

Inside your <cassandra_home>/conf/cassandra.yaml try setting:

read_request_timeout_in_ms: 20000
rpc_timeout_in_ms: 20000

Restart your server and try the query execution again.

Thx for your answer. I will try it. – Steffen Harbich Aug 16 '16 at 08:46 — Steffen Harbich, Aug 16 '16 at 08:46

score 1 · Accepted Answer · answered Aug 15 '16 at 17:48

I was getting the similar timeout exception when trying to query nodes with bigger data. I modified these 4 values in cassandra.yaml and it resolved all the timeout errors.

How long the coordinator should wait for read operations to complete read_request_timeout_in_ms: 15000

How long the coordinator should wait for seq or index scans to complete range_request_timeout_in_ms: 30000

How long the coordinator should wait for writes to complete write_request_timeout_in_ms: 30000

How long the coordinator should wait for counter writes to complete counter_write_request_timeout_in_ms: 15000

Note: You need to do this in all the nodes in the cluster and also need to restart Cassandra in all of them.

Thx for your answer. I will try it. – Steffen Harbich Aug 16 '16 at 08:46 — Steffen Harbich, Aug 16 '16 at 08:46

Astyanax not respecting configured socket timeout

2 Answers2