0

I have a use case where requires data backup across multiple data centers and needs strong consistency. The ideal view is each segment is replicated to three clusters located at three different data centers. pulsar supports using multiple clusters as a large bookie pool but I didn't find how to configure the replicas in different clusters. Anyone has similar use case before? i think it should be not hard to do considering pulsar separate broker and storage + replicas in different clusters

dashenswen
  • 540
  • 2
  • 4
  • 21

1 Answers1

1

It's possible to enable a region aware placement policy of bookies (parameter bookkeeperClientRegionawarePolicyEnabled). You'll also need to configure the bookie region with the admin command set-bookie-rack This is not much documented in Pulsar/BookKeeper docs. See this blog post for more details : https://techblog.cdiscount.com/ensure-cross-datacenter-guaranteed-message-delivery-and-resilience-with-apache-pulsar/

Beware that due to the cross-region latency between the brokers and the bookies, the throughput will drop but that can't really be helped if you need strong consistency even in the case of a region failure.

Christophe Bornet
  • 1,062
  • 8
  • 8
  • why it can not help when there is a region failure? assume i have 3 regions and one region failed, the rest two should still be able to serve the traffic to my understanding – dashenswen Dec 08 '21 at 01:32
  • yes the traffic will still be served. My remark only concerns the throughput which will be affected by the cross-region network latency compared to a local setup. – Christophe Bornet Dec 09 '21 at 11:24