Strimzi cant resize PV

Question

I followed Strimzi blog to resize PV.

I use Openshift v3.11 deployed on Azure VMs with PV as Azure managed disk

My Kafka cluster storage config

..
    config:
      offsets.topic.replication.factor: 2
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
      log.message.format.version: "2.6"
    storage:
      type: persistent-claim
      size: 256Gi
      deleteClaim: false
...

I directly edit pvc and changed the resource request to 257Gi. I wait for few mins , checked the status of PVC like blow

 oc get pvc data-0-xx-dev-kafka-0 -o yaml
apiVersion: v1
kind: PersistentVolumeClaim

...

spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 257Gi
  storageClassName: generic
  volumeName: pvc-xx-2849-xx-913f-xx
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 256Gi
  conditions:
  - lastProbeTime: null
    lastTransitionTime: 2021-10-08T15:54:06Z
    status: "True"
    type: Resizing
  phase: Bound

In the description of pvc, I see below

Warning  VolumeResizeFailed     3s (x2 over 1m)  volume_expand                Error expanding volume "kafka/data-0-sirius-dev-kafka-0" of plugin kubernetes.io/azure-disk : compute.DisksClient#CreateOrUpdate: 
Failure sending request: StatusCode=409 -- Original Error: failed request: autorest/azure: 
Service returned an error. Status=<nil> Code="OperationNotAllowed" Message="Cannot resize disk kubernetes-dynamic-pvc-xx-2849-xx-913f-xx while it is 
attached to running VM /subscriptions/xxxx/resourceGroups/xxx-dev-openshift/providers/Microsoft.Compute/virtualMachines/ocp-node-dev-1. Resizing a disk of an Azure Virtual Machine requires the virtual machine to be deallocated. 
Please stop your VM and retry the operation."

I have also tried, redeployed kafka with jbod with single disk, resize and did rolling update. Same result like above

• openshift v3.11.0+cbab8ee-94(K8s v1.11.0+d4cacc0)
• Kafka Version: 2.6.0
• Operators : 0.20.1

Note that resize PV does support in my cluster(Previously I resized a PV of an app successfully by scaling down replicas to zero)

UPDATE

$ oc describe storageclass generic
Name:                  generic
IsDefaultClass:        Yes
Annotations:           storageclass.beta.kubernetes.io/is-default-class=true
Provisioner:           kubernetes.io/azure-disk
Parameters:            kind=managed,location=${location},storageaccounttype=Premium_LRS
AllowVolumeExpansion:  True
MountOptions:
  discard
ReclaimPolicy:      Delete
VolumeBindingMode:  Immediate
Events:             <none>

I tried to scale down reaplica to 0 with oc scale replicas=0 sts/XX-dev, but cluster-operator is not allowing since because of replication factor.

Answer 1

Normally Azure Disk is supporting resizing of Persistent Volumes, but currently, there are some issues in terms of compliancy and implementation check here

So before going any other place; just check one more time following points

• check the Storage Class definition with "storageClassName: generic" if allowVolumeExpansion is set to true. (may be you tried resizing with different storage class before) You can only expand a PVC if its storage class's allowVolumeExpansion field is set to true.
• it is looking like that you need to de-allocate everything which is pointing to that volume first(check warning at https://learn.microsoft.com/en-us/azure/aks/azure-disk-csi) then resize and lastly scale Pods back.
• try to increment with multiplies check the sizes from here(https://learn.microsoft.com/en-us/azure/virtual-machines/disks-types)
• maybe hitting volume limits per VM, default is 16 for Azure, for the time being, check how many volumes you have now in that node

The other issue is that if underlying capacity is not enough for resizing you have to deallocate and relocate VM in the new resource group. Yes cloud is not unlimited