1

We are getting some weird load distribution on our cassandra (3.7) cluster. The cluster has 13 nodes, with 16 cores & 32G ram, and it is running behind Apache Usergrid (forked). When we put (artificial) stress on the application, same 3 nodes get almost all the CPU load and the rest of the nodes operate normally. We could not create the same scenario using cassandra-stress tool, with that all the load is distributed evenly. There is nothing different on the logs of these nodes either. gc logs also seem to be similar (running G1 gc for high heap memory). The queries originate from Usergrid, which we do not have much control. I need guidance for finding the problem. (So far nothing helpful from the internet.) Where should I look to diagnose the problem?

Eren Yilmaz
  • 1,082
  • 12
  • 21

1 Answers1

0

It looks like uneven distribution of data/keys/requests between nodes. Check that your partition key distributes data evenly, and you are not requesting same key.

nevsv
  • 2,448
  • 1
  • 14
  • 21
  • Data distribution is even actually: http://pastebin.com/a1RBBath . It's between 22-24%. – Eren Yilmaz Mar 09 '17 at 07:27
  • 1
    If you have RF=3 and 3 hot nodes, it seems likely that even if your data is evenly distributed, your requests may not be. You can use "nodetool toppartitions" to see the hottest read/written partitions on a given keyspace/table pair. – Jeff Jirsa Mar 09 '17 at 07:33
  • Data can be distributed evenly, but if you requesting same small set of keys (and look like it is very small set of these), your request will go to the same servers. – nevsv Mar 09 '17 at 08:04
  • @JeffJirsa Yes, RF=3. I ran nodetool toppartitions, but that does not make any sense to me. To what should I look at in the output? The same partition in hot node outputs? – Eren Yilmaz Mar 10 '17 at 07:57
  • @nevsv The problem occurs only in load test, and we run a finite set of test users, looping and running similar operations. Does this cause uneven distribution of requests? Does increasing RF help? – Eren Yilmaz Mar 10 '17 at 08:00
  • There's a partition key on the table. Can you verify that your set have multiple values for that partition key? – nevsv Mar 12 '17 at 05:57
  • @nevsv although our application does not control the keys directly, I checked from all tables, and it appears all have unique values for the partition key. I am not 100% sure anyways, because the keys are binary, I can not read them directly. – Eren Yilmaz Mar 29 '17 at 11:38
  • The keys always be unique, you should check that your requests are not going using same key. Can you explain what exactly you are running in your artificial stress? – nevsv Mar 29 '17 at 11:56
  • @nevsv Our application uses Apache Usergrid as BaaS, and therefore we have no control the keys, Usergrid manages that. In the stress test, we login some set of users, post some activities on Usergrid and read the feed. That's all. – Eren Yilmaz Mar 29 '17 at 12:44
  • Some users - what are the number of users? 1,3,10? Please make it at least 100 and see if the load stays on same servers. And also, please, make sure that these are different users, and not the same user in all calls. – nevsv Mar 29 '17 at 12:49
  • Number of users is between 500 and 10,000. They are different users. In the scenario, a user logs in, posts 3 activities (1 sec delay between each call), and gets the feed. – Eren Yilmaz Mar 29 '17 at 13:03
  • So it looks like all of them saved on same partition somehow. Are in the connection string you use list of all 13 servers? Also, can you desc on several tables that used in the flow and add them to the question? – nevsv Mar 29 '17 at 13:10
  • @nevsv Connection string has all of the server addresses. I can not get data from the tables, the data is binary and does not make any sense as text. If the problem is that all data is stored on these 3 servers, do I have any chance to redistribute the data? Will adding and/or removing nodes do any help? – Eren Yilmaz Apr 04 '17 at 12:41
  • After decomissioning problematic nodes, the problem is redistributed to other nodes. We still have 3 nodes peaking on load. – Eren Yilmaz Apr 13 '17 at 09:27
  • 1
    Apparently, the problem is caused by counter tables. We get the high cpu load only for counter related requests. Now we are looking for cassandra counter performance issues. – Eren Yilmaz Apr 14 '17 at 09:05