4

We had multiple clients configured to talk to this cluster of aerospike nodes. Now that we have removed the configuration from all the clients we are aware of, there are still some read/write requests coming to this cluster, as shown in the AMC.

I looked at the log file generated in /var/log/aerospike/aerospike.log, but could not get any information.

Update

The netstat command as mentioned in the answer by @kporter shows the number of connections, with statuses ESTABLISHED, TIME_WAIT, CLOSE_WAIT etc. But, that does not mean those connections are currently being used for get/set operations. How do I get the IPs from which aerospike operations are currently being done?

Update 2 (Solved)

As mentioned in the comments to @kporter's answer, a tcpdump command on the culprit client showed packets still being sent to the aerospike cluster which was no more referenced in the config file. This was happening while even AMC of that cluster did not show any more read/write TPS.

I later found that this stopped after doing a restart of the nginx service on the client. Please note that the config file in the client now references a new aerospike cluster and packets sent to that cluster did not stop after the nginx restart. This is weird but it worked.

Sandeepan Nath
  • 9,966
  • 17
  • 86
  • 144

1 Answers1

5

Clients connect to Aerospike over port 3000:

The following command, when run on the server nodes, will show the addresses of hosts connecting to the server over port 3000.

netstat --tcp --numeric-ports | grep 3000
kporter
  • 2,684
  • 17
  • 26
  • Ok, that showed the number of connections, with statuses ESTABLISHED, TIME_WAIT, CLOSE_WAIT etc. But, that does not mean those connections are currently being used for get/set operations. How do I get the IPs from which aerospike operations are currently being done? – Sandeepan Nath Aug 08 '17 at 06:27
  • One of those connections is the IP youvare looking for. You could use tcpdump to identify the rogue client. – kporter Aug 09 '17 at 17:24
  • I ran this tcpdump command in one of the clients mentioning the IP of the server in a grep - `sudo tcpdump -i eth0 | grep ip-10-146-210-31`. Although it still showed some entries with that IP, the AMC console is now showing zero reads and writes. I am still not convinced with what I observed, because tcpdump shoud have not shown this IP anymore. Marking resolved still, because I would not be able to look into it anymore. Thanks! – Sandeepan Nath Aug 10 '17 at 13:10
  • Sample output of tcmpdump still containing the IP of the server node even though AMC shows zero TPS - `13:09:03.859301 IP ip-10-97-204-78.ec2.internal.58368 > ip-10-146-210-31.ec2.internal.hbci: Flags [P.], seq 86:129, ack 283, win 4079, length 43` – Sandeepan Nath Aug 10 '17 at 13:13
  • 1
    I later found that the packets sent to the cluster stopped after doing a restart of the `nginx` service on the client. Please note that the config file in the client now references a new aerospike cluster and packets sent to that cluster did not stop after the nginx restart. This is weird but it worked. – Sandeepan Nath Aug 11 '17 at 06:40