2

I recently installed FluxCD 1.19.0 on an Azure AKS k8s cluster using fluxctl install. We use a private git (self hosted bitbucket) which Flux is able to reach and check out.

Now Flux is not applying anything with the error message:

ts=2020-06-10T09:07:42.7589883Z caller=loop.go:133 component=sync-loop event=refreshed url=ssh://git@bitbucket.some-private-server.com:7999/infra/k8s-gitops.git branch=master HEAD=7bb83d1753a814c510b1583da6867408a5f7e21b
ts=2020-06-10T09:09:00.631764Z caller=sync.go:73 component=daemon info="trying to sync git changes to the cluster" old=7bb83d1753a814c510b1583da6867408a5f7e21b new=7bb83d1753a814c510b1583da6867408a5f7e21b
ts=2020-06-10T09:09:01.6130559Z caller=sync.go:539 method=Sync cmd=apply args= count=3
ts=2020-06-10T09:09:20.2097034Z caller=sync.go:605 method=Sync cmd="kubectl apply -f -" took=18.5965923s err="running kubectl: error: unable to recognize \"STDIN\": an error on the server (\"\") has prevented the request from succeeding" output=
ts=2020-06-10T09:09:38.7432182Z caller=sync.go:605 method=Sync cmd="kubectl apply -f -" took=18.5334244s err="running kubectl: error: unable to recognize \"STDIN\": an error on the server (\"\") has prevented the request from succeeding" output=
ts=2020-06-10T09:09:57.277918Z caller=sync.go:605 method=Sync cmd="kubectl apply -f -" took=18.5346491s err="running kubectl: error: unable to recognize \"STDIN\": an error on the server (\"\") has prevented the request from succeeding" output=
ts=2020-06-10T09:09:57.2779965Z caller=sync.go:167 component=daemon err="<cluster>:namespace/dev: running kubectl: error: unable to recognize \"STDIN\": an error on the server (\"\") has prevented the request from succeeding; <cluster>:namespace/prod: running kubectl: error: unable to recognize \"STDIN\": an error on the server (\"\") has prevented the request from succeeding; dev:service/hello-world: running kubectl: error: unable to recognize \"STDIN\": an error on the server (\"\") has prevented the request from succeeding"
ts=2020-06-10T09:09:57.2879489Z caller=images.go:17 component=sync-loop msg="polling for new images for automated workloads"
ts=2020-06-10T09:09:57.3002208Z caller=images.go:27 component=sync-loop msg="no automated workloads"

From what I understand, Flux passes the resource definitions to kubectl, which then applies them?

The way I interpret the error would mean that kubectl isn't passed anything to. However I opened a shell in the container and made sure Flux was in fact checking something out - which it did.

I tried raising the verbosity to 9, but it didn't return anything that I deemed relevant (detailed outputs of the http requests and responses against the Kubernetes API).

So what is happening here?

Michael Niemand
  • 1,578
  • 3
  • 23
  • 39
  • 1
    does flux have permissions to actually apply changes to the cluster? – 4c74356b41 Jun 11 '20 at 11:40
  • thought about that, too. The version I installed, creates a Clusterrole with * for verbs and \*.\* for resources which is then applied to the according service account. I'm not an expert, but it looks sufficient? – Michael Niemand Jun 11 '20 at 11:57
  • 1
    are you sure there is something to apply in the git repo? – 4c74356b41 Jun 11 '20 at 13:37
  • yes, absolutely. I copied a yaml from another project that I had deployed already (it's a deployment and a service with a hello-world docker container). It's supposed to be deployed in a namespace that is also configured under the allowed namespaces flag – Michael Niemand Jun 11 '20 at 15:37
  • 1
    are they .yaml (not .yml)? – 4c74356b41 Jun 11 '20 at 15:46
  • yes, hello-world.yaml in a folder that has been passed as --git-path to fluxd. I just noticed that a simple kubectl version in the flux container also throws said error. I'm trying to gain access to the kube-apiserver logs right now (which I don't have currently). Thanks for your continued effort, I highly appreciate it!! – Michael Niemand Jun 11 '20 at 16:07
  • 1
    can you test with a kubectl container if kubectl works? just give it the same service account – 4c74356b41 Jun 11 '20 at 16:14
  • @4c74356b41 Deployed a job, same service account, same namespace, with the bitnami/kubectl container, running kubectl version which worked fine for client AND server versions ... – Michael Niemand Jun 12 '20 at 08:32
  • now that is interesting: kubectl version with the bitnami:latest returned the versions properly, however bitnami/kubectl:1.15.3 (the fluxcd has 1.15.9) returns the same error as the flux container ... – Michael Niemand Jun 12 '20 at 08:38
  • whats your aks version? try updating that?* – 4c74356b41 Jun 12 '20 at 08:45
  • 1.16.9 it's the latest non-preview version on Azure. – Michael Niemand Jun 12 '20 at 08:55

1 Answers1

2

The problem was with the version of kubectl used in the 1.19 flux release, so I fixed it by using a prerelease: https://hub.docker.com/r/fluxcd/flux-prerelease/tags

Michael Niemand
  • 1,578
  • 3
  • 23
  • 39