0

I have a 16 node cassandra cluster and I am inserting 50.000 rows, pretty much in parallel, from an external tool(which is installed in every node) in every cassandra node with Simba Cassandra JDBC Driver. While the insertion takes place, sometimes/rarely, I get the following error on (mostly/usually) two of the nodes:

Execute failed: [Simba]CassandraJDBCDriver Error setting/closing connection: All host(s) tried for query failed (tried: localhost/127.0.0.1:9042 (com.simba.cassandra.shaded.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ONE (1 responses were required but only 0 replica responded))). java.sql.SQLException: [Simba]CassandraJDBCDriver Error setting/closing connection: All host(s) tried for query failed (tried: localhost/127.0.0.1:9042 (com.simba.cassandra.shaded.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ONE (1 responses were required but only 0 replica responded))). Caused by: com.simba.cassandra.shaded.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: localhost/127.0.0.1:9042 (com.simba.cassandra.shaded.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ONE (1 responses were required but only 0 replica responded)))

The weird thing is that it is a readtimeout exception, while I am just trying to insert. I have not changed any read_time_out or other parameters in the .yaml file, so they are the default. This means that if I try to count(*) on something from cqlsh I also get a readtimeout exception.

ReadTimeout: Error from server: code=1200 [Coordinator node timed out waiting for replica nodes' responses] message="Operation timed out - received only 0 responses." info={'received_responses': 0, 'required_responses': 1, 'consistency': 'ONE'}

I do not know if these two are related though! Any ideas on what might be going on and how to avoid the first error "All host(s) tried for query failed"??

Des0lat0r
  • 482
  • 3
  • 18
  • 1
    why are you using JDBC instead of using drivers directly? – Alex Ott May 19 '20 at 09:46
  • I am not sure I completely understand your question, but I have linked my external tool with cassandra with simbacassandraJDBCDriver(the .jar file). No specific reason on why I used that. What should/could I use? Do you think the problem is with the JDBC Driver? – Des0lat0r May 19 '20 at 10:18
  • 1
    Can you paste a query that is executed by your tool? Most probably it's trying to work like traditional RDBMS (like `count(*)`) and this is anti-pattern for Cassandra. If the tool is doing something like reporting, etc. - then you can try to use Spark for that. For data insertion - the problem could be if batches are used, etc. But it's hard to say without any additional information about your tool, what queries are executed, etc. – Alex Ott May 19 '20 at 10:57
  • The external tool is Knime and the insert command it uses is mostly like visual programming(see picture https://ibb.co/pJfpytc ). Knime has nodes that allow it to do stuff and two of the nodes is DB connector and DB Insert(whose configuration window you can see in the image). The truth is I am using Spark which activates knime in every node. Knime processes some files and stores the results in cassandra with Simba JDBC driver. The weird thing is that it does not always fail and it does not fail on the same files. So, I am just thinking that maybe it is a network issue? – Des0lat0r May 19 '20 at 11:24
  • Also, in every node batches of 1000 rows are used to insert these 50.000 rows! But I dont think that a batch of 1000 rows is big. – Des0lat0r May 19 '20 at 12:27
  • 1
    Batches in Cassandra are not of insert optimization - they are putting much more load to the nodes than non-batches. Try to decrease batch size to 1, and see how it will go. Although it may not help if the application is doing synchronous calls – Alex Ott May 19 '20 at 14:19
  • 1
    ok..so I decreased it down to 1(and 2) and takes way to long. if I change it to 5000 the batch insert is done in 16 seconds. Maybe with big batches there is a big load on the coordinator/node but I do not know how to fix that. – Des0lat0r May 20 '20 at 07:28
  • I maybe would try with 1000, or something like... just for reference how batches work in Cassandra - see image at the end of https://gist.github.com/alexott/2d5bf95a791450d5dc3e5f5ee11fc94c – Alex Ott May 20 '20 at 08:25
  • Alex brings up some good points. To me, it doesn't sound like Cassandra is a good fit for what you're trying to do. BATCH in Cassandra was not designed to be used with thousands (or even hundreds) of rows. It sounds to me like the data flow needs to be re-architected, as it was designed with RDBMS data store behaviors in mind. – Aaron May 20 '20 at 14:36
  • Also, drop JDBC and use the DataStax driver. You'll be much happier. – Aaron May 20 '20 at 14:37

0 Answers0