0

k8s controller manager cannot start (solved by changing AWS EC2 tag)

I have similar issue as this post: kube-controller-manager don't start when using “cloud-provider=aws” with kubeadm, but that does not provide any solution to solve the issue. Below is the log stats from kubernetes controller manager.

I0705 17:59:05.941049       1 controllermanager.go:116] Version: v1.10.3
W0705 17:59:05.941983       1 authentication.go:55] Authentication is disabled
I0705 17:59:05.942004       1 insecure_serving.go:44] Serving insecurely on 127.0.0.1:10252
I0705 17:59:05.942450       1 leaderelection.go:175] attempting to acquire leader lease  kube-system/kube-controller-manager...
I0705 17:59:22.823671       1 leaderelection.go:184] successfully acquired lease kube-system/kube-controller-manager
I0705 17:59:22.823945       1 event.go:218] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"kube-system", Name:"kube-controller-manager", UID:"ec7f1208-807c-11e8-bc66-0e594759e784", APIVersion:"v1", ResourceVersion:"289", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' k8s_1bbcc7bc-807d-11e8-859d-0e594759e784 became leader
I0705 17:59:22.849017       1 aws.go:1026] Building AWS cloudprovider
I0705 17:59:22.849064       1 aws.go:988] Zone not specified in configuration file; querying AWS metadata service
F0705 17:59:22.851606       1 controllermanager.go:159] error building controller context: cloud provider could not be initialized: could not init cloud provider "aws": error finding instance i-07df027cffda674bc: "error listing AWS instances: \"NoCredentialProviders: no valid providers in chain. Deprecated.\\n\\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors\""

The /etc/kubernetes/aws.conf file.

apiVersion: kubeadm.k8s.io/v1alpha1
kind: MasterConfiguration
cloudProvider: aws
kubernetesVersion: 1.10.3

The /etc/kubernetes/cloud-config.conf file.

[Global]
KubernetesClusterTag=k8s
KubernetesClusterID=k8s

Solution

Change EC2 instance tag, KEY=KubernetesCluster, VALUE=k8s(in my case).

jiashenC
  • 1,812
  • 2
  • 16
  • 31
  • Key should be `KubernetesCluster` and value should be `k8s` per your `cloud-config.conf` file. It should be EC2 role not user for IAM. – Const Jul 05 '18 at 19:33
  • Thanks! Changing the tag solves my issue! – jiashenC Jul 05 '18 at 19:59
  • A new issue is kube-dns is always pending. – jiashenC Jul 05 '18 at 19:59
  • This can be several things, but is your network up and operational (flannel, weave...)? Do you have sec group properly configured? – Const Jul 05 '18 at 21:37
  • I have all ports open for all addresses open attached to the sec group just for easier testing purpose, so I don't think that's the case. I do use weave net, and it crashes immediately after I do init. I am not able to see the log information. Also I tried to use `kubectl describe` which does not provide useful information. – jiashenC Jul 05 '18 at 21:42
  • Did you install it correcty? `sysctl net.bridge.bridge-nf-call-iptables=1` and then `export kubever=$(kubectl version | base64 | tr -d '\n')` followed by `kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$kubever"` – Const Jul 05 '18 at 21:45
  • Yes, exactly the same command. Previously, it worked well without cloud-provider flag. – jiashenC Jul 05 '18 at 21:48
  • is yout cgroup driver properly configured in kubelet? – Const Jul 05 '18 at 21:54
  • Also check here for some pointers and log extraction: https://github.com/weaveworks/weave/issues/3247. You have to bring your weave-net up before you can start dns properly. – Const Jul 05 '18 at 22:00
  • Regarding to cgroups, I have an environmental variable setup, as `Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs"`. I also do `daemon-reload` and `restart` after making the changes. Do I need additional adjustments? – jiashenC Jul 05 '18 at 22:22
  • No, that is sufficient for cgroups. Did you manage to check logs as per hints in url above? – Const Jul 05 '18 at 22:27
  • It works after I install weave net on the server. Previously I just applied the image after the cluster is up. – jiashenC Jul 05 '18 at 22:44
  • Also, @Const. Thank you so much for your patience and help! – jiashenC Jul 05 '18 at 22:48
  • Glad that it worked out! If it is not much of a trouble to ask, upvote original answer, and create answer of your own here so other don't have to shuffle through comments.. Thanks :) – Const Jul 05 '18 at 22:54
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/174524/discussion-between-jiashen-cao-and-const). – jiashenC Jul 06 '18 at 20:08

1 Answers1

1

kube-dns is always pending because of weave net crash

Fix cgroups setting.

  • Add Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs" to /etc/systemd/system/kubelet.service.d/10-kubeadm.conf.
  • Restart kubelet service. 
    sudo systemctl deamon-reload
sudo systemctl restart kubelet

Install weave net and launch before creating kubernetes cluster.

jiashenC
  • 1,812
  • 2
  • 16
  • 31