4

We are proposing Cassandra to be implemented as a database backend for a large archiving solution (a large number of writes compared to reads). We are looking for inputs on Cassandra's replication and deployment strategy to fit our use case.

The choice of Cassandra was based on following factors:

  • Supports Large throughput for ‘write’ operations - thousands of simultaneous writes per second
  • Suitability for Engineering data (mainly Time series data)
  • High availability to support continuous telescope operations
  • Tools support e.g. Analytics, Reporting

Data Estimates

  • 250 TB of growth per year (50 years of System lifetime)

Use Case

We have two data centers - Operations DC and Analytics DC (to isolate the read and write workloads). At the end of this post is the diagram depicting the proposed architecture. Due to storage constraints, we can't store data generated over life-time on Operations DC. Hence, we are planning to move the data from Operations DC to Analytics DC as per defined policy (let's say after 1 week).

Questions

  1. Is it possible to have one-way replication in Cassandra between Datacenters? Data from Operations DC moved into Analytics DC. But data stored after processing in Analytics DC shall not be replicated into Operations DC.
  2. Does Cassandra provide control on what gets replicated? We don't want both the DCs to be in synch. We want to configure what gets replicated (moved actually) into the Analytics DC. Is it possible inherently with Cassandra? If I want to specify that only data of the last one week should be replicated from Operations Data Centre to Analytics Data Center.

  3. We are planning to use Cassandra's inbuilt feature of time-to-live to delete the data (only from operations DC). The data deleted from Operations DC should not get deleted from the Analytics DC. How to prevent replication of deleted data?

  4. I have read that a single Cassandra node can handle up to 2-3 TB of data. Any documented reference of any larger Cassandra implementations will help.

  5. How many Cassandra nodes shall be deployed to handle such growth? And what shall be a recommended deployment strategy?

  6. Performance considerations: Although the storage at Operations DC will be limited (3-7 days data, about 5-10 TB), the data storage at Analytics DC is cumulative and continues to grow with time. Will the database growth at analytics DC affect the replication and degrade the performance of Operations DC.

The purpose here is to know if Cassandra's inbuilt feature can be used to support the above requirements. I'm aware of the most obvious solution. Not to have replication between the two DC. Dump the data of last one week from Operations DC and move it to Analytics DC.

Proposed architecture diagram

Jyotin Ranpura
  • 71
  • 5
  • 1
    Hiring a consultant who has some experience in storing 10+ PB in Cassandra might be a better plan than soliciting random help on an internet Q&A site. – Aaron Jun 06 '18 at 03:24
  • I understand your suggestion, Aaron. Already highlighted it to the Management and they will work on it. For now, any help from the experts will be appreciated. – Jyotin Ranpura Jun 13 '18 at 14:53

2 Answers2

2

I think that in your case is just makes sense to "separate" DCs - keyspaces in one DC aren't replicated into another DC, for example - just create keyspaces with necessary corresponding replication settings.

Or you can replicate "transactional" load into both DCs, and have a job that will periodically copy data from "transactional" keyspace into "analytics" keyspace, and then remove data from "transactional" keyspace to free the space.

But it's not really possible to have something like as you describe, until you use something like DSE's Advanced Replication (but it's not about DCs, but more about separate clusters).

Alex Ott
  • 80,552
  • 8
  • 87
  • 132
  • Thank you Alex. Read about the DSE's Advanced Replication and seems to be a good fit. I would like to clarify my understanding: Deploying separate clusters - One cluster (Edge cluster) responsible for the Writes and another cluster responsible for the Reads (Central Hub). - Can we have control over this replication? By control, I mean scheduling this replication at the said time. - If we purge the data from the Edge cluster, will it be also deleted from the Central Hub? - Does Edge cluster have the full power of Analytics on all the data from Central Hub? – Jyotin Ranpura Jun 13 '18 at 14:51
  • These are interesting questions, but to answer them there is a need for more information, and involvement of AdvRep specialists. It could be better if you contact DataStax to organize more in-depth discussion of these requirements... – Alex Ott Jun 15 '18 at 11:05
2

  1. No

  2. Yes, replication is configured per keyspace.

  3. This won’t work out of the box, but it can be made to work. I can think of two relatively easy options. The easiest is to batch write to both keyspaces/DCs, one with TTLs and One without . You could potentially also make a keyspace per month/year, start with it replicating to multiple DCs, and remove the “normal” DC when appropriate.

  4. Cassandra cluster - data density (data size per node) - looking for feedback and advises

  5. Cassandra does ok up to about 800-1000 instances in a cluster, but it’s often advisable to shard smaller than that for your own ease of operation

  6. DCs can be asymmetrical.

Jeff Jirsa
  • 4,391
  • 11
  • 24