0

I want to read 5+ million events from an Azure SQL DB table and perform a BULK INSERT to Cassandra. The table has 2 columns. I see the SQL component available for reading from Azure SQL DB. https://camel.apache.org/components/3.7.x/sql-component.html

Question: Consuming from Azure SQL DB

  1. Is there a better way to read all the rows and store in a map considering 5M records ?
  2. Is there a possibility to read messages in batches ?

There is a cql component available for Cassandra https://camel.apache.org/components/3.7.x/cql-component.html

Question: Producing to Cassandra

  1. Can we INSERT in batches ?

Can I use camel for this use case ?

srikant_mantha
  • 109
  • 1
  • 13

1 Answers1

1

For each table in your Azure database, we recommend that you export the data into a CSV file. There are lots of tools and methods available that would allow you to do that. For example, have a look at Import and export data from Azure SQL Database.

Once you have exported your data to CSV, you can use the DataStax Bulk Loader tool (DSBulk) to bulk load it to a Cassandra table.

Here are some references with examples to help you get started quickly:

DSBulk is open-source so it's free to use. Cheers!

Erick Ramirez
  • 13,964
  • 1
  • 18
  • 23
  • Thanks @Erick Ramirez, This is really helpful to get started. I was thinking of writing a java application, but this task sounds more easy and repeatable. – srikant_mantha Aug 17 '21 at 13:03
  • 1
    Yep, exactly the reason we decided to write an open-source tool for it so users don't have to reinvent the wheel. Cheers! – Erick Ramirez Aug 17 '21 at 13:10