1

I am trying to deploy Weaviate on Azure Kubernetes Service. During the helm deployment run into a problem where I get the following error message:

Multi-Attach error for volume "pvc-69db6155-4f28-11ea-b829-b2b3d6f12b6f" Volume is already exclusively attached to one node and can't be attached to another
Unable to mount volumes for pod "esvector-master-0_weaviate(20dafc44-4f58-11ea-b829-b2b3d6f12b6f)": timeout expired waiting for volumes to attach or mount for pod "weaviate"/"esvector-master-0". list of unmounted volumes=[esvector-master]. list of unattached volumes=[esvector-master default-token-ckf7v]

The only thing I changed in values.yaml is the Storage Class Name:

pvc:
  size: 2Gi
  storageClassName: default

I made this change as Azure does not have an NFS class installed. Instead I used the default kubernetes class which is levering Azure Managed disks.

Does anyone have an idea on how to solve this issue? Thanks!

Jeroen
  • 66
  • 2
  • Thanks for raising this. We'll look into if whether we can reproduce this on AKS and get back to you. First assumption would be that it is AKS specific, as we don't see this error on GKE or EKS. – etiennedi Feb 17 '20 at 09:23
  • I got things working on AKS. There are some timing issues when deploying the helm chart. First of all I had to set the --timeout flag in helm to 900 (15 minutes). Doing all the volume claims takes a lot of time. After the volume claims were done, I had to delete the etcd pods so k8s can recreate them with the PVC in place. As the last step, after the etcd pods were up and running, I had to delete the weaviate pod and let kubernetes recreate it. – Jeroen Feb 20 '20 at 13:47
  • Did you need to make any changes to the charts @Jeroen? If so, would you mind issuing a PR? – Bob van Luijt Feb 23 '20 at 20:18
  • 1
    Hi @BobvanLuijt I haven't changed the charts. The only thing I changed was the config class used in the values.config. As Azure doesn't have a NFS provider, I changed it to: `storageClassName: default` This will allow K8S to use managed disks in Azure as storage provider. – Jeroen Feb 29 '20 at 18:49
  • This will either have no effect (if `etcd.disasterRecovery.enabled` is set to `false` which is the default) or will break etcd disaster recovery unless your `default` storage class supports `ReadWriteMany` access. Please see my full answer for an explanation of why and what to do to address this. Thanks. – etiennedi Mar 03 '20 at 11:08

1 Answers1

1

We've updated our docs as they weren't complete around the topic of etcd disaster recovery in the helm chart. With the updated docs in mind, let me try to explain what's going on here:

No nfs volumes required by default

By default Weaviate uses Persistent Volumes for its backing databases. The storage classes for those use the defaults, i.e. not nfs. Therefore when using the default values.yaml no nfs support is required on the cluster.

etcd Disaster Recovery

At the time of writing this answer, one of the storage backends for Weaviate is etcd. We use the bitnami etcd chart which is referenced in the Weaviate Chart as a subchart. Etcd does not survive a failure of a quorum of nodes (Source). Especially in a small deployment (e.g. 3 or fewer etcd pods), regular Kubernetes maintenance can easily lead to a disastrous etcd failure. To combat this, the above mentioned chart from Bitnami contains a disaster recovery mode.

Note that etcd.disasterRecovery.enabled defaults to false, but we recommend setting it to true in production.

Deploy an nfs provisioner, if etcd disaster recovery is required.

The etcd disaster recovery feature which is part of the bitnami etcd helm chart requires ReadWriteMany access for the snapshot volumes. The recommendation is to use an nfs provisioner as outlined in the Weaviate Helm Docs.

Why is the nfs-provisioner not part of the Weaviate chart?

It might seem counter-intuitive that the disaster recovery is a crucial part for a stable production setup, yet the provisioner is not included in the weaviate chart as a sub-chart. This has multiple reasons:

  • A mix of concerns: The Weaviate chart installs Weaviate with the goal to isolate all effects to a single namespace. The nfs-provisioner makes cluster wide-changes that might not be entirely obvious
  • Multi-tenancy: We can make no assumption that your Kubernetes cluster runs only a single Weaviate instance or even only Weaviate instances. It might be a big shared cluster with multiple tenants. In this case bundling the provisioner would lead to the installation of multiple provisioners, when the cluster can and should have only a single one
  • Different Lifecycles lead to circular dependencies: If the provisioner were bundled up with Weaviate it would become impossible to delete the Weaviate chart. This is because deleting the Weaviate chart, also deletes the etcd subchart. The latter removes the nfs volumes used for snapshotting. However, if the bundler was part of the chart it would already have been deleting, rendering the cluster unable to delete nfs volumes.

tl;dr: Deploy the provisioner once in a different namespace, deploy as many Weaviate instances as you like in separate namespaces. This avoids lifecycle differences, issues with multi-tenancy and circular dependencies.

etiennedi
  • 346
  • 1
  • 7