1

Currently have velero up and running and it's working great. The only issue I have is that the snap shots of the volumes are being created in the same region as the originals which kinda defeats the purpose of disaster recovery. This flag

--snapshot-location-config

doesn't have arg for region. I know there is a config for the default snap shot location

volumesnapshotlocations.velero.io "default"

Does anyone know how to modify the default so I can get my snap shots into new regions?

Hizzy
  • 741
  • 7
  • 27

2 Answers2

2

Snapshots creation from the main region into a different region is not supported.
Azure zone-redundant snapshots and images for managed disks have a decent 99.9999999999% (12 9's) durability. The availability zones in a region are usually physically separated and even if an outage affects one AZ, you can still access your data from a redundant AZ.

However, if you fear calamities that can affect several square kilometers(multiple zones in a region), you can manually move the snapshots in a different region or even automate the process. Here is a guide to do it.

Neo Anderson
  • 5,957
  • 2
  • 12
  • 29
  • Unfortunaly I just cam across that documentation. Velero should consider adding a function to copy and move that data into a new region kinda the whole point of DR – Hizzy Aug 19 '20 at 15:00
  • 1
    Depends on the magnitude of the disaster :) and how valuable the data is. If a meteor would wipe an entire continent, I guess we'd have bigger issues than the data durability. It's all about statistics: if 12 9's isn't enough, then you can take additional safety measures for the data. I guess they aren't offering this out of the box because it would come with an additional cost. Moving TB of data over the ocean is not as cheap as moving data on the intra-availability-zones dedicated channels. You know...at the other extreme, the GITHUB ARCTIC CODE VAULT approach is an alternative :) – Neo Anderson Aug 19 '20 at 15:08
  • So do your snap shots have a TTL all i see is the backups being removed my snap shots are building up – Hizzy Aug 19 '20 at 16:48
  • I never use snapshots as backups. The only reason I'd use a snapshot is to **temporarily** freeze the disk and capture that state if I need it. I don't know what Azure is using under the hood to create snapshots, but I know that in VMware it is a bad practice to stack too many incremental snapshots: hence each snapshot is a delta on top of the previous one, the performance is affected. VMware also has a limit of 32 snapshots on one chain. – Neo Anderson Aug 19 '20 at 16:58
  • interesting yeah i need to freeze it and hold it in time to generate a back from it. So I can then re attach a restored managed disk to the cluster – Hizzy Aug 19 '20 at 16:59
  • What would you use as a back up ASR doesn't support AKS clusters – Hizzy Aug 19 '20 at 17:01
  • Dont care much about backing up the cluster I have teraform and helm that can auto deploy everything in like 15 minutes – Hizzy Aug 19 '20 at 17:02
  • That's the point. You want to backup the persistence data volumes. Personally only have prod Kubernetes clusters in GKE. Unfortunately I don't have the expertise to give you an answer for Azure, but they have good documentation [here](https://docs.microsoft.com/en-us/azure/aks/operator-best-practices-storage) and [here](https://docs.microsoft.com/en-us/azure/aks/operator-best-practices-multi-region). Each backup scenario might have specific requirements: for example, if you want to have resiliency for a DB, you might want to look into that DB vendor-specific replication and HADR solutions – Neo Anderson Aug 19 '20 at 17:07
  • 1
    Yeah i do want to move to the vendor spefic solutions for this but that unfortuantly I can just recommend but some one else makes that call ha. Yeah i have been over that documentation a lot it says to use ASR but that really doesn't work and I asked them for a demo but they haven't put one up so Velero it is. – Hizzy Aug 19 '20 at 17:46
0

--snapshot-location-config doesn't have arg for region

--snapshot-location-config doesn't create the storage, you must do so yourself. You can specify a different region, a different Azure subscription, or even a different provider, like AWS.

For Azure, follow the instructions here to create your storage container.

If your provider supports a region config (Azure does not - see Volume Snapshot Location Config doc and Backup Storage Location Config doc), it is configurable using the --config, e.g. --config region=us-west-2. Check your provider plugin to see whether different regions are supported, what the key name is, and what possible values are supported.

Refer to the Velero locations documentation for examples of using multiple snapshot and backup locations.

Update:

Although velero snapshot-location create allows you to specify a --provider, the Limitations/Caveats section of the Location documentation specifically states that only a single set of credentials is supported, and furthermore that Azure specifically does not allow creation of snapshots in a different region:

Velero only supports a single set of credentials for VolumeSnapshotLocations. Velero will always use the credentials provided at install time (stored in the cloud-credentials secret) for volume snapshots.

Volume snapshots are still limited by where your provider allows you to create snapshots. For example, AWS and Azure do not allow you to create a volume snapshot in a different region than where the volume is. If you try to take a Velero backup using a volume snapshot location with a different region than where your cluster’s volumes are, the backup will fail.

I personally find this confusing -- how could one use a different provider without specifying credentials? Regardless, it seems as if storage of a snapshots in a different region in Azure is not possible.

Codebling
  • 10,764
  • 2
  • 38
  • 66