2

I'm using cassandra 2.1.2 and datastax cassandra-driver-core-2.1.2. Here is a strange problem: when a keyspace is created ( or table created, deleted), some of my clients received duplicated events, about 200+ times. my cluster and my clients are in different places(not in one lan). This cause a lot of problems, Once client received such a event, it should refresh schema, and fetch all schema infos from system.keyspaces and so on. in the end , it also refreshNodeListAndTokenMap. All of these operations may cause some data transfer, and 200+ events in one second is horrible. So any body knows why & how to prevent?

thanks for reading this.

高健峰
  • 21
  • 2

1 Answers1

0

When you mention "some of my clients receive duplicated events", I'm assuming you have multiple Cluster instances, is that correct? If you only have one Cluster object and you are getting multiple events I'm wondering if that is a bug. The java-driver will only subscribe 1 connection (named the 'Control Connection') to schema, topology and node status changes per Cluster instance. That connection is established to one of your contact points initially (and if connection is lost it'll choose another node in the cluster).

Without understanding more about your configuration, I would consider the following:

  1. Follow the 4 simple rules, namely only creating 1 cluster instance per application (JVM).
  2. If you want to prevent 1 node from being responsible for sending events to your clients, randomize your contact points so the same one isn't always primarily chosen for the Control Connection. (Note: There is a ticket for this so the java-driver can do that for you, JAVA-618)
Andy Tolbert
  • 11,418
  • 1
  • 30
  • 45
  • thanks, Andy. Yeah, i've got 2 instances per client, one for normal use, and one for admin use. Is this the reason? But 200+ events is too much. – 高健峰 Mar 18 '15 at 02:59
  • I'm from the same team of @高健峰. We observe that Cassandra client keeps getting the following error: error on cassandra.node/58.211.x.x:9042 connection, trying next host, and keeps trying to refresh node list and token map, and refresh schema. This process keeps on and on and consumes much bandwidth – Jim Huang Mar 18 '15 at 03:18
  • Hi @JimHuang, would you mind providing me the exception that accompanies that message? I'm guessing that error likely means the Driver is having trouble establishing a Control Connection (failing to reconnect). – Andy Tolbert Mar 18 '15 at 04:04
  • com.datastax.driver.core.ConnectionException: [/58.211.191.85:9342] Operation timed out. at com.datastax.driver.core.DefaultResultSetFuture.onTimeout(DefaultResultSetFuture.java:140) at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:855) at org.jboss.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:546) – Jim Huang Mar 18 '15 at 06:29
  • Hi, @AndyTolbert, recently I got the same problem once. 4 node cassandra cluster, only 1 Cluster instance for the java client. But when a table was created, the client received 236 duplicate CREATE TABLE EVENT in 1 second. Thanks for reading this and expecting your answer. – 高健峰 Jun 03 '15 at 07:53