0

I am using Aerospike 3.7.3 and python client 2.x

I have two node cluster each node consuming 100G of data. Due to some issue, I have stopped one node and started with cold-start-empty=true.

  1. First issue is, it took around 9 hours to complete the migration. Is it usual to take 9 hours?

  2. Getting following error occationally (30% of the time) while connecting to cluster during migration. (While Starting one of the node in cluster)

        aerospike_config = {
      'hosts': [ ('212.16.290.10', 3000), ('212.16.290.11', 3000),],
      'policies': {
         'timeout': 10, # milliseconds
         'key': aerospike.POLICY_KEY_SEND
       }
    }


    client  = aerospike.client(aerospike_config).connect()

    

exception.ClientError: (-1L, 'Failed to seed cluster', 'src/main/aerospike/as_cluster.c', 417)

Kindly help.

Carbonrock
  • 457
  • 2
  • 15

1 Answers1

2

When you did a cold-start-empty, you created a situation where one node has 100% of the data, and the other that just joined had 0% of the data. You did not allow it to read the disk and recover the 50% it previously had, after which you'd have the data checked for being out of sync. At this point the length of migration depends on things such as the read speed of your disks, and the networking you have for shipping half of it to the opposite node, all while handling incoming work. Length of migration doesn't just depend on your hardware, but also on your configuration. See migrate-threads, migrate-xmit-hwm, migrate-xmit-lwm, migrate-xmit-priority, migrate-xmit-sleep, and other config parameters.

For your other question, you're setting the connection timeout limit down to 10ms from the default of 1s. I don't know much about your nodes or workload, but that is a very low connection timeout. If you're seeing that error 30% of the time it's set too low. Basically, not being able to seed a cluster means that the client cannot connect to either of the IPs you've provided in under 10ms, therefore it cannot acquires the seed node, the node from which it learns about the other nodes in the cluster, and grabs the partition table.

Ronen Botzer
  • 6,951
  • 22
  • 41