0

I have a repository with snapshots in a AWS S3 bucket created from a Crate Cluster. Now I need to migrate the data to a new Crate Cluster but I cannot restore the snapshots from the previous Crate Cluster. Is that even possible? To restore snapshots from another repository? Appreciate any insights regarding this subject

metase
  • 1,169
  • 2
  • 16
  • 29
ALeX
  • 13
  • 1
  • 3
  • why can't you restore? what have you tried to do? – metase Feb 04 '20 at 15:31
  • I've created a repository from my original Crate cluster in a S3 AWS bucket and did a sanpshot of the data. In my new Crate cluster, to import the snapshot I tried to create a repository pointing to the same S3 AWS bucket, but I get an error saying that the repository already exists. How can I import a snapshot from another Crate cluster? – ALeX Feb 05 '20 at 19:59
  • Could you share the commands you used on a new cluster? Did you copy anything from the old cluster? If the repo already exists you can just restore the snapshot. – metase Feb 05 '20 at 20:36
  • on your new cluster is there anything in "select * from sys.repositories" – metase Feb 05 '20 at 22:43
  • I've used on the first repository: "CREATE REPOSITORY $REPOSITORY TYPE s3 WITH (access_key='$AWS_ACCESS_KEY_ID', secret_key='$AWS_SECRET_ACCESS_KEY', bucket='$BUCKET', base_path='$SNAPSHOT', compress=true);" and then I generated the snapshot "CREATE SNAPSHOT repo.snaphot ALL WITH (wait_for_completion=true);" after completion, in the new cluster I tried: – ALeX Feb 06 '20 at 21:30
  • "RESTORE SNAPSHOT repo.snapshot TABLE sitelog WITH (wait_for_completion=true);" but since I can't point the new crate cluster to the old repository, the snapshot can't be used to restore tables. There aren't other repositories in the new cluster because I want to use the old one and when I try to created a new one pointing to the old one in the S3 bucket it gives me an error saying it already exists. – ALeX Feb 06 '20 at 21:39
  • so if I'm reading this correctly you haven't created a new repository on your new cluster? If no then you have to create it there as well and then you can restore snapshot onto your new cluster – metase Feb 06 '20 at 23:17
  • That's the problem, if I create a new repository with another name, the snapshot isn't there because it's in the old repo. If I try to create the a repository with the same name, pointing to the old repository, I get an error saying that the repository already exists. Have you ever done it? Restored a snapshot from a different repo? – ALeX Feb 08 '20 at 01:14
  • Yes I have, but never with "base_path" I have whole bucket dedicated to snapshots. Does '$SNAPSHOT' change each time? My best guess now would be it's something to do with that. On you new cluster what do you get when you run select * from sys.repositories? Drop the repository on new cluster and create it again – metase Feb 09 '20 at 16:23
  • $SNAPSHOT could change but I'm maintaining the same in order to debug. On my new cluster I get only the repositories I created to test this issue, and not the one in the s3 bucket. I'll do as you suggest, thank you @metase – ALeX Feb 09 '20 at 20:44
  • @metase How did you created a repo in the new cluster pointing to the bucket with the snapshots from the old cluster? In the documentation it says it's not possible to crate a new repo with same same of the old one. https://crate.io/docs/crate/reference/en/latest/admin/snapshots.html An if so then I can't use that repo to import the snapshot – ALeX Feb 17 '20 at 21:08
  • where does it say that? I moved a cluster over the weekend to azure in exactly this way – metase Feb 17 '20 at 21:10
  • That means you can't have the same repo multiple times on one cluster. When you create a new cluster you can create a repo with the same name. Just to confirm you are creating a blank cluster? (no data copied from old cluster?) – metase Feb 17 '20 at 21:28

1 Answers1

0

To restore snapshot to another cluster use CREATE REPOSITORY command that will register a new repository in the cluster. Use existing repository name if you are moving to another cluster and restore snapshot with RESTORE SNAPSHOT command e.g.

assume you already have a repository called "s3backup" and snapshot called "s3backup.firsts3_snapshot"

On you new cluster

CREATE REPOSITORY s3backup TYPE s3 with (max_restore_bytes_per_sec= '500mb', max_snapshot_bytes_per_sec='500mb', access_key= '***', secret_key='***',bucket='***' );

THEN 

RESTORE SNAPSHOT s3backup.firsts3_snapshot TABLE *your_table_name*;
metase
  • 1,169
  • 2
  • 16
  • 29