Worker nodes not available

Question

I have setup and installed IBM Cloud private CE with two ubuntu images in Virtual Box. I can ssh into both images and from there ssh into the others. The ICp dashboard shows only one active node I was expecting two.

I explicitly ran the command (from a root user on master node):

docker run -e LICENSE=accept --net=host \
  -v "$(pwd)":/installer/cluster \
  ibmcom/cfc-installer install -l \
  192.168.27.101

The result of this command seemed to be a successful addition of the worker node:

PLAY RECAP *********************************************************************
192.168.27.101             : ok=45   changed=11   unreachable=0    failed=0

But still the worker node isn't showing in the dashboard.

What should I be checking to ensure the worker node will work for the master node?

score 3 · Answer 1 · answered Oct 13 '17 at 03:48

If you're using Vagrant to configure IBM Cloud Private, I'd highly recommend trying https://github.com/IBM/deploy-ibm-cloud-private

The project will use a Vagrantfile to configure a master/proxy and then provision 2 workers within the image using LXD. You'll get better density and performance on your laptop with this configuration over running two full Virtual Box images (1 for master/proxy, 1 for the worker).

score 0 · Answer 2 · answered Sep 13 '17 at 18:30

0

You can check on your worker node with following steps:

check cluster nodes status kubectl get nodes to check status of the newly added worker node
if it's NotReady, check kubelet log if there is error message about why kubelet is not running properly:
- ICp 2.1 systemctl status kubelet
- ICp 1.2 docker ps -a|grep kubelet to get kubelet_containerid, docker logs kubelet_containerid

answered Sep 13 '17 at 18:30

Rebecca Guo

1

Thanks @rebecca-guo. I ran the command (ICp 2.1beta) and all seems well. kubelet.service - Kubelet Service Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2017-09-14 18:11:20 UTC; 50min ago Docs: ... Main PID: 1173 (hyperkube) Tasks: 21 Memory: 145.8M CPU: 3min 788ms CGroup: /system.slice/kubelet.service ├─1173 /opt/kubernetes/hyperkube kubelet --feature-gates Accelerators=true --allow-privileged=true --docker-disable-shared-p └─1603 journalctl -k -f – jimc Sep 14 '17 at 19:06
what did you get if you run `docker ps -a` on your worker nodes, can you see any exited containers, if yes, can you enter into the containers see if there is any error messages? – Rebecca Guo Sep 14 '17 at 20:19
Thanks for the tip Rebecca! After playing a little bit I notice this one container (ibmcom/calico-node) keeps exiting. The logs say: time="2017-09-14T21:39:35Z" level=info msg="NODENAME environment not specified - check HOSTNAME" and Calico node 'icpworker1' is already using the IPv4 address 10.0.3.15: – jimc Sep 14 '17 at 22:24
if the `icpworker1` is your new worker node try to add in, can you please check in your /etc/hosts file if there is `127.0.1.1 localhost` in it, please remove it – Rebecca Guo Sep 25 '17 at 20:16
Thanks Rebecca. I think there was a 127.0.0.1 mapping in there. Removed it and all is fine! – jimc Sep 29 '17 at 16:14

score 0 · Answer 3 · edited Jan 19 '18 at 15:28

Run this to get the kubectl working

ln -sf /opt/kubernetes/hyperkube /usr/local/bin/kubectl

run the below command to identified failed pods if any in the setup on the master node.

Run this to get the pods details running in the environment kubectl -n kube-system get pods -o wide

for restarting any failed pods of icp

txt="0/";ns="kube-system";type="pods"; kubectl -n $ns get $type | grep "$txt" | awk '{ print $1 }' | xargs kubectl -n $ns delete $type

now run the kubectl cluster-info

kubectl get nodes

Then ckeck the cluster info command of kubectl

Check kubectl version is giving you https://localhost:8080  or https://masternodeip:8001

kubectl cluster-info

Do you get the output

if no.. then

login to https://masternodeip:8443 using admin login

and then copy the configure clientcli settings by clicking on admin on the panel paste it in ur master node.

and run the kubectl cluster-info

Worker nodes not available

3 Answers3