Encountered error applying application application: (kubeflow.error): Code 500 with message: Apply.Run

Question

I'm trying to deplo kubeflow on AKS using this YAML file https://raw.githubusercontent.com/kubeflow/manifests/v1.2branch/kfdef/kfctl_k8s_istio.v1.2.0.yaml. But I'm having the error below I could not know how to fix it . I followed exactly what was mentioned in https://www.kubeflow.org/docs/distributions/azure/deploy/install-kubeflow/ This is the configuration that I'm following :

##### create a local dir
mkdir ~/test-mvp
cd ~/test-mvp

##### login and set subscription
az login
az account set --subscription 63h2dd53-fdf6-4c11-8edf-593ed487c85d

##### create a resource group
az group create -n test-mvp -l westus


##### create a AKS cluster (takes several minutes)
az aks create -g test-mvp -n test-mvp -s Standard_D4s_v3 -c 2 -l westus --generate-ssh-keys

##### create user credentials (this places entries in ~/.kube/config)
az aks get-credentials -n test-mvp -g test-mvp

##### get kfctl and unzip/untar
wget https://github.com/kubeflow/kfctl/releases/download/v1.2.0/kfctl_v1.2.0-0-0gbc038f9_linux.tar.gz

gunzip kfctl_v1.2.0-0-gbc038f9_linux.tar.gz 
tar xvf kfctl_v1.2.0-0-gbc038f9_linux.tar
rm -f kfctl_v1.2.0-0-gbc038f9_linux.tar

##### setup kubeflow
export PATH=$PATH:"/home/abderrahim/test-mvp"
export KF_NAME=test-mvp
export BASE_DIR=/home/abderrahim/test-mvp
export KF_DIR=${BASE_DIR}/${KF_NAME}
export CONFIG_URI="https://raw.githubusercontent.com/kubeflow/manifests/v1.2-branch/kfdef/kfctl_k8s_istio.v1.2.0.yaml"
mkdir -p ${KF_DIR}
cd ${KF_DIR}

##### deploy kubeflow
kfctl apply -V -f ${CONFIG_URI}

Thiss is the error that i'm having right now


    abderrahim@Azure:~/mykf$ kfctl apply -V -f ${CONFIG_URI}
    INFO[0000] Downloading https://raw.githubusercontent.com/kubeflow/manifests/v1.2-branch/kfdef/kfctl_k8s_istio.v1.2.0.yaml to /tmp/025648159/tmp.yaml  filename="utils/k8utils.go:178"
    INFO[0000] Downloading https://raw.githubusercontent.com/kubeflow/manifests/v1.2-branch/kfdef/kfctl_k8s_istio.v1.2.0.yaml to /tmp/414875378/tmp_app.yaml  filename="loaders/loaders.go:71"
    INFO[0000] App directory /home/abderrahim/mykf already exists  filename="coordinator/coordinator.go:270"
    INFO[0000] Writing KfDef to kfctl_k8s_istio.v1.2.0.yaml  filename="coordinator/coordinator.go:273"
    INFO[0000] No name specified in KfDef.Metadata.Name; defaulting to mykf based on location of config file: /home/abderrahim/mykf/kfctl_k8s_istio.v1.2.0.yaml.  filename="coordinator/coordinator.go:202"
    INFO[0000] 
    ****************************************************************
    Notice anonymous usage reporting enabled using spartakus
    To disable it
    If you have already deployed it run the following commands:
      cd $(pwd)
      kubectl -n ${K8S_NAMESPACE} delete deploy -l app=spartakus
    
    For more info: https://www.kubeflow.org/docs/other-guides/usage-reporting/
    ****************************************************************
      filename="coordinator/coordinator.go:120"
    INFO[0000] Deleting cachedir /home/abderrahim/mykf/.cache/manifests because Status.ReposCache is out of date  filename="kfconfig/types.go:477"
    INFO[0000] Fetching https://github.com/kubeflow/manifests/archive/v1.2.0.tar.gz to /home/abderrahim/mykf/.cache/manifests  filename="kfconfig/types.go:498"
    INFO[0001] Updating localPath to /home/abderrahim/mykf/.cache/manifests/manifests-1.2.0  filename="kfconfig/types.go:569"
    INFO[0001] Fetch succeeded; LocalPath /home/abderrahim/mykf/.cache/manifests/manifests-1.2.0  filename="kfconfig/types.go:590"
    INFO[0001] Processing application: namespaces            filename="kustomize/kustomize.go:569"
    INFO[0001] Creating folder /home/abderrahim/mykf/kustomize/namespaces  filename="kustomize/kustomize.go:667"
    INFO[0001] Processing application: application           filename="kustomize/kustomize.go:569"
    INFO[0001] Creating folder /home/abderrahim/mykf/kustomize/application  filename="kustomize/kustomize.go:667"
    INFO[0001] Processing application: istio-stack           filename="kustomize/kustomize.go:569"
    INFO[0001] Creating folder /home/abderrahim/mykf/kustomize/istio-stack  filename="kustomize/kustomize.go:667"
    INFO[0001] Processing application: cluster-local-gateway  filename="kustomize/kustomize.go:569"
    INFO[0001] Creating folder /home/abderrahim/mykf/kustomize/cluster-local-gateway  filename="kustomize/kustomize.go:667"
    INFO[0001] Processing application: istio                 filename="kustomize/kustomize.go:569"
    INFO[0001] Creating folder /home/abderrahim/mykf/kustomize/istio  filename="kustomize/kustomize.go:667"
    INFO[0001] Processing application: cert-manager-crds     filename="kustomize/kustomize.go:569"
    INFO[0001] Creating folder /home/abderrahim/mykf/kustomize/cert-manager-crds  filename="kustomize/kustomize.go:667"
    INFO[0001] Processing application: cert-manager-kube-system-resources  filename="kustomize/kustomize.go:569"
    INFO[0001] Creating folder /home/abderrahim/mykf/kustomize/cert-manager-kube-system-resources  filename="kustomize/kustomize.go:667"
    INFO[0001] Processing application: cert-manager          filename="kustomize/kustomize.go:569"
    INFO[0001] Creating folder /home/abderrahim/mykf/kustomize/cert-manager  filename="kustomize/kustomize.go:667"
    INFO[0001] Processing application: add-anonymous-user-filter  filename="kustomize/kustomize.go:569"
    INFO[0001] Creating folder /home/abderrahim/mykf/kustomize/add-anonymous-user-filter  filename="kustomize/kustomize.go:667"
    INFO[0001] Processing application: metacontroller        filename="kustomize/kustomize.go:569"
    INFO[0001] Creating folder /home/abderrahim/mykf/kustomize/metacontroller  filename="kustomize/kustomize.go:667"
    INFO[0001] Processing application: bootstrap             filename="kustomize/kustomize.go:569"
    INFO[0001] Creating folder /home/abderrahim/mykf/kustomize/bootstrap  filename="kustomize/kustomize.go:667"
    INFO[0001] Processing application: spark-operator        filename="kustomize/kustomize.go:569"
    INFO[0001] Creating folder /home/abderrahim/mykf/kustomize/spark-operator  filename="kustomize/kustomize.go:667"
    INFO[0001] Processing application: kubeflow-apps         filename="kustomize/kustomize.go:569"
    INFO[0001] Creating folder /home/abderrahim/mykf/kustomize/kubeflow-apps  filename="kustomize/kustomize.go:667"
    INFO[0001] Processing application: knative               filename="kustomize/kustomize.go:569"
    INFO[0001] Creating folder /home/abderrahim/mykf/kustomize/knative  filename="kustomize/kustomize.go:667"
    INFO[0001] Processing application: kfserving             filename="kustomize/kustomize.go:569"
    INFO[0001] Creating folder /home/abderrahim/mykf/kustomize/kfserving  filename="kustomize/kustomize.go:667"
    INFO[0001] Processing application: spartakus             filename="kustomize/kustomize.go:569"
    INFO[0001] Creating folder /home/abderrahim/mykf/kustomize/spartakus  filename="kustomize/kustomize.go:667"
    INFO[0001] /home/abderrahim/mykf/.cache/manifests exists; not resyncing   filename="kfconfig/types.go:473"
    INFO[0001] namespace: kubeflow                           filename="utils/k8utils.go:433"
    INFO[0002] Log cluster name into KfDef: test-mvp         filename="kustomize/kustomize.go:253"
    INFO[0002] Deploying application namespaces              filename="kustomize/kustomize.go:266"
    namespace/cert-manager unchanged
    namespace/kubeflow unchanged
    INFO[0003] Successfully applied application namespaces   filename="kustomize/kustomize.go:291"
    INFO[0003] Deploying application application             filename="kustomize/kustomize.go:266"
    serviceaccount/application-controller-service-account unchanged
    clusterrole.rbac.authorization.k8s.io/application-controller-cluster-role unchanged
    clusterrolebinding.rbac.authorization.k8s.io/application-controller-cluster-role-binding unchanged
    service/application-controller-service unchanged
    statefulset.apps/application-controller-stateful-set configured
    WARN[0006] Encountered error applying application application:  (kubeflow.error): Code 500 with message: Apply.Run : [unable to recognize "/tmp/kout005506118": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1", unable to recognize "/tmp/kout005506118": no matches for kind "Application" in version "app.k8s.io/v1beta1"]  filename="kustomize/kustomize.go:284"
    WARN[0006] Will retry in 2 seconds.                      filename="kustomize/kustomize.go:285"
    serviceaccount/application-controller-service-account unchanged
    clusterrole.rbac.authorization.k8s.io/application-controller-cluster-role unchanged
    clusterrolebinding.rbac.authorization.k8s.io/application-controller-cluster-role-binding unchanged
    service/application-controller-service unchanged
    statefulset.apps/application-controller-stateful-set configured

Do you find out how to solve this problem? I face the same thing/ — viet quan dang, Jul 07 '22 at 09:00

score 0 · Answer 1 · answered Jul 15 '22 at 08:47

Seems K8s removed beta CustomResourceDefinition API in 1.22: https://kubernetes.io/blog/2021/07/14/upcoming-changes-in-kubernetes-1-22/#api-changes

And default AKS are using k8s with 1.22.11 which caused the problem.

A work around for this problem is to create an AKS with 1.21.* version of k8s.

Eg:

az aks create -g <RESOURCE_GROUP_NAME> -n <NAME> -s <AGENT_SIZE> -c <AGENT_COUNT> -l <LOCATION> --kubernetes-version 1.21.14 --generate-ssh-keys

Encountered error applying application application: (kubeflow.error): Code 500 with message: Apply.Run

1 Answers1