0

I am new to druid. In our application we use druid for timeseries data and this can go pretty large(10-20TBs). Druid provide you facility of deep storage. But if this deep storage crashes/or not reachable then it will result in data loss and which in turn affect the analytics the application is running. I am thinking of taking an incremental backup druid segment data to some secure location like ftp server. So if deep storage is unavailable, then they can restore the data from this ftp server.

Is there any tool/utility available in druid to incrementally backup/restore druid segment?

Rahul Vedpathak
  • 1,346
  • 3
  • 16
  • 30

1 Answers1

1

In general it's important to take regular snapshots of the metadata storage as this is the "index" of what's in the Deep Storage. Maybe one snapshot per day, and store them for however long you like. It's good to store them for at least a couple of weeks, in case you need to roll back for some reason.

You also need to back up new segments in deep storage when they appear. It isn't important to take consistent snapshots, just to get every file eventually.

Also see https://groups.google.com/g/druid-user/c/itfKT5vaDl8

One other note as you mentioned data loss: Deep Storage is not queried directly - queries execute on the local segment cache in, for example, the Historical process. The Deep Storage is written to at ingestion time, so you might "lose" data that can't be ingested once it's available again, but you will continue to get analytics capability as the already-loaded data is on the historicals... Just a thought haha !

I hope that helps....?!?!

Peter Marshall
  • 91
  • 5
  • But taking backup of whole deep storage is resource consuming tasks if u have a huge cluster and data can be in TBs. It's better to backup data from deep storage incrementally instead of taking full backup. Is there existing utility that can help to achieve that? – Rahul Vedpathak Sep 25 '20 at 12:31
  • I'm afraid I'm not an S3 expert: maybe something like this is what you need? https://stackoverflow.com/questions/21479110/s3-incremental-backups – Peter Marshall Sep 28 '20 at 11:45