2

I need to transfer data from one cluster to an other.

The table structure is the same on both clusters, what I need to do is select data from Table A, Clustering Key A1 on Cluster 1 and copy it to Table B, Clustering Key A1 on Cluster 2.

There is a high number of entries for that clustering key, I suppose > 50.000.000
I do not want and I cannot copy the whole table, because data between clusters in this table is different.

One option would be to write a script and loop through the data, writing to cluster 2. This would work but sounds inefficient and needs to address problems like "what to do if this script crashes in the middle of operation?"

What is the best approach for that?

Alex Tbk
  • 2,042
  • 2
  • 20
  • 38

2 Answers2

2

Based on what I have experienced, Spark is provides the best mechanism to do such activities. You can do it using RDDs and DataFrame APIs both. Below is the code snippet from the reference links:

import com.datastax.spark.connector._
import com.datastax.spark.connector.cql._

import org.apache.spark.SparkContext

sqlContext.setConf("ClusterOne/spark.cassandra.connection.host", "127.0.0.1")
sqlContext.setConf("ClusterTwo/spark.cassandra.connection.host", "127.0.0.2")

//Read from ClusterOne
val dfFromClusterOne = sqlContext
  .read
  .format("org.apache.spark.sql.cassandra")
  .options(Map( 
    "cluster" -> "ClusterOne",
    "keyspace" -> "ks",
    "table" -> "A"
    ))
  .load
  .filter($"id" === 'A1')

//Write to ClusterTwo
dfFromClusterOne
  .write
  .format("org.apache.spark.sql.cassandra")
  .options(Map( 
    "cluster" -> "ClusterTwo",
    "keyspace" -> "ks",
    "table" -> "B"
    ))
  .save
}

Reference links:

Kumar Rohit
  • 507
  • 1
  • 5
  • 13
1

For bulk data copy, you should think about sstableloader. This is a good tool to copy the data from one cluster and load into another cluster. please refer below documentation. https://cassandra.apache.org/doc/latest/tools/sstable/sstableloader.html?highlight=sstableloader

LetsNoSQL
  • 1,478
  • 1
  • 11
  • 23
  • I only need data for a specific key, so this is not helpful as stated in the initial question – Alex Tbk Jan 17 '20 at 05:51
  • You can create sstable(s) with only a subset of data, see https://github.com/PatrickCallaghan/datastax-bulkloader-writer-example. Streaming is more efficient than inserts. – Serban Teodorescu Jan 21 '20 at 11:18