4

I have a running project with using of akka-persistence-jdbc plugin and postgresql as a backend.

Now I want to migrate to akka-persistence-cassandra. But how can I convert the existing events (more than 4GB size in postgres) to cassandra?

Should I write a manual migration program? Reading from postgres and writing to right format in cassandra?

Mark Rotteveel
  • 100,966
  • 191
  • 140
  • 197
hich9n
  • 1,578
  • 2
  • 15
  • 32

1 Answers1

4

This is a classic migration problem. There are multiple solutions for this.

  1. Spark SQL and Spark Cassandra Connector: Spark JDBC (called as Spark Dataframe, Spark SQL) API allows you to read from any JDBC source. You can read it in chunks by segmenting it otherwise you will go out of memory. Segmentation also makes the migration parallel. Then write the data into Cassandra by Cassandra Spark Connector. This is by far the simplest and efficient way I used in my tasks.

  2. Java Agents: Java Agent can be written based on plain JDBC or other libraries and then write to Cassandra with Datastax driver. Spark program runs on multi machine - multi threaded way and recovers if something goes wrong automatically. But if you write an agent like this manually, then your agent only runs on single machine and multi threading also need to be coded.

  3. Kafka Connectors: Kafka is a messaging broker. It can be used indirectly to migrate. Kafka has connector which can read and write to different databases. You can use JDBC connector to read from PostGres and Cassandra connector to write to Cassandra. It's not that easy to setup but it has the advantage of "no coding involved".

  4. ETL Systems: Some ETL Systems have support for Cassandra but I haven't personally tried anything.

I saw some advantages in using Spark Cassandra and Spark SQL for migration, some of them are:

  1. Code was concise. It was hardly 40 lines
  2. Multi machine (Again multi threaded on each machine)
  3. Job progress and statistics in Spark Master UI
  4. Fault tolerance- if a spark node is down or thread/worker failed there then job is automatically started on other node - good for very long running jobs

If you don't know Spark then writing agent is okay for 4GB data.

Sreekar
  • 995
  • 1
  • 10
  • 22