-1

I am installing tiller onto an EKS cluster from a client machine. The goal is to install an application, JupyterHub, via helm onto the EKS kubernetes cluster.

I installed it on one cluster just fine, following the steps below (our test cluster).

Now, on our prod cluster, it does not work. Details below

connect kubectl to the cluster

aws eks update-kubeconfig --name my_cluster

create service account

kubectl --namespace kube-system create serviceaccount tiller
serviceaccount/tiller created

create cluster rolebinding

kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller
clusterrolebinding.rbac.authorization.k8s.io/tiller created

init tiller

helm init --service-account tiller --wait
# ... wait a bit ...
helm version
Client: &version.Version{SemVer:"v2.16.7", GitCommit:"5f2584fd3d35552c4af26036f0c464191287986b", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.16.7", GitCommit:"5f2584fd3d35552c4af26036f0c464191287986b", GitTreeState:"clean"}

install the application (JupyterHub)

helm upgrade --install jhub jupyterhub/jupyterhub --namespace jhub --version=2.0.1605648253 --values jhub.config.yaml
UPGRADE FAILED
Error: configmaps is forbidden: User "system:serviceaccount:kube-system:tiller" cannot list resource "configmaps" in API group "" in the namespace "kube-system"
Error: UPGRADE FAILED: configmaps is forbidden: User "system:serviceaccount:kube-system:tiller" cannot list resource "configmaps" in API group "" in the namespace "kube-system"

check logs

kubectl get event --namespace kube-system
LAST SEEN   TYPE      REASON        OBJECT                               MESSAGE
10s         Warning   FailedMount   pod/tiller-deploy-59bfffd84b-l7lvq   MountVolume.SetUp failed for volume "tiller-token-8bcns" : secret "tiller-token-XXXXX" not found

At first consideration, it might seem that it is an issue with the application (JupyterHub) but I think not, because there was one additional step we make, which is to run a command that secures our cluster by allowing tiller to listen to commands from localhost only, and that also is not working as expected, and that is not dependent on the jupyterhub application at all, so it leads me to believe this is an issue with kubernetes, helm, or tiller

kubectl patch deployment tiller-deploy --namespace=kube-system --type=json --patch=\'[{"op": "add", "path": "/spec/template/spec/containers/0/command", "value": ["/tiller", "--listen=localhost:44134"]}]'
Tiller does not have permission --> this error message is not the exact wording, I neglected to copy the exact message, but IIRC it was short and simple and said tiller does not have permission (I'm currently in the process of deleting and re-creating the cluster. I will update this when I have the actual message)
TylerH
  • 20,799
  • 66
  • 75
  • 101
James Wierzba
  • 16,176
  • 14
  • 79
  • 120
  • If it's a new cluster, do you have a specific reason to need the old version of Helm? Helm 3 doesn't use Tiller (and not needing to manage it is a huge improvement). – David Maze Dec 08 '20 at 00:26
  • @DavidMaze no specific reason other than it would take some effort to update the version of helm on the standard OS image we use to manage the cluster via kubectl, but it just might be worth it for that reason you mentioned (not needing tiller) – James Wierzba Dec 08 '20 at 01:00

1 Answers1

1

Ok, so destroying the cluster completely and re-creating it seems to have solved the issue. No idea what the cause was, but this kubernetes event message (below) might have been a clue. EBS can be notoriously flaky in my experience, probably due to the fact it is a hardware interface

The kubernetes event message:

kubectl get event --namespace kube-system
LAST SEEN   TYPE      REASON        OBJECT                               MESSAGE
10s         Warning   FailedMount   pod/tiller-deploy-59bfffd84b-l7lvq   MountVolume.SetUp failed for volume "tiller-token-8bcns" : secret "tiller-token-XXXXX" not found
James Wierzba
  • 16,176
  • 14
  • 79
  • 120