4

I have 8 node Aerospike cluster. My Aerospike Go client (18 servers) sends x (~8k per seconds) batch request per second with batch size ~8-12. What will be Total TPS on my cluster and per Node?

I an getting frequent error in Go client:

command execution timed out: Exceeded number of retries. See Policy.MaxRetries. (last error: No available connections to the node. Connection Pool was empty, and limited to certain number of connections.)

I want to check if it's because of network limitation. what could be other reasons? Total storage used is ~12-15% and there are some ~3k per second writes overall as well.

Nishant Kumar
  • 2,199
  • 2
  • 22
  • 43
  • Does Aerospike limit the number of connections one can make? Have you tried increading the ulimit on the machine? Are you closing connections when you're done with them? Can you create a connection pool with a min max number of connections? – reticentroot Nov 16 '16 at 20:12

1 Answers1

2

Aerospike Go Client dev here.

The error you are getting means that the app is using all the connections available in the pool already, and no connections are available to connect to the nodes.

This behavior depends on the following attributes in the ClientPolicy when you're connecting to the database:

  • ConnectionQueueSize: determines how many connections will be pooled per node.
  • LimitConnectionsToQueueSize: Determines if an attempt should be made to connect to a node, in case the pool is empty when a connection is needed.

There is also the MaxRetries attribute on Read/WritePolicy, which determines the number of retries.

Which version of the go client are you using, and what are values you are passing for the above settings?

Khosrow Afroozeh
  • 141
  • 1
  • 1
    ConnectionQueueSize is 1500. I am not using LimitConnectionsToQueueSize. I am using commit ID : 7f3a312c3b2a60ac083ec6da296091c52c795c63 [Date: Fri Aug 19 19:47:49 2016 +0200] – Nishant Kumar Nov 17 '16 at 11:48
  • 1
    What are the recommended values for these attributes? What is the exact cause of this error? – ameykpatil Nov 29 '16 at 06:35
  • @NishantKumar Please use the latest released version. 1500 connections *per node* is excessive. Rarely should you need more than 512. Using `LimitConnectionsToQueueSize` prevents the node to try making a new connection when the queue is empty and all connections are already created and in use. – Khosrow Afroozeh Dec 04 '16 at 11:51
  • @KhosrowAfroozeh Sorry for late reply. I am still facing this issue. I ran go-client in debug mode and got this error: `logger: logger.go:102: Node BB9A4083AJJJ40C 10.57.115.42:3000: No available connections to the node. Connection Pool was empty, and limited to certain number of connections.` when i checked that particular node has high number of client connections (~15k) and other nodes has ~3-4k client connection. `proto-fd-max` in server config is 30k – Nishant Kumar Jun 20 '17 at 10:36
  • how many clients are connected to that particular node? We have observed that sometimes hardware failures, disproportionate scan/query load and hot keys can contribute to this problem. – Khosrow Afroozeh Jun 21 '17 at 12:31