ICP 2.1 Beta 2 can't create Cloudant DB fast enough

Question

When running the following command from the installation instructions:

sudo docker run --net=host -t -e LICENSE=accept -v $(pwd):/installer/cluster \
            ibmcom/icp-inception:2.1.0-beta-2-ee install

The task to create the Cloudant DB times out:

TASK [master : Ensuring that the Cloudant Database is ready] ***********************************************************************************************************************************************
FAILED - RETRYING: TASK: master : Ensuring that the Cloudant Database is ready (20 retries left).
FAILED - RETRYING: TASK: master : Ensuring that the Cloudant Database is ready (19 retries left).
FAILED - RETRYING: TASK: master : Ensuring that the Cloudant Database is ready (18 retries left).
FAILED - RETRYING: TASK: master : Ensuring that the Cloudant Database is ready (17 retries left).
FAILED - RETRYING: TASK: master : Ensuring that the Cloudant Database is ready (16 retries left).
FAILED - RETRYING: TASK: master : Ensuring that the Cloudant Database is ready (15 retries left).
FAILED - RETRYING: TASK: master : Ensuring that the Cloudant Database is ready (14 retries left).
FAILED - RETRYING: TASK: master : Ensuring that the Cloudant Database is ready (13 retries left).
FAILED - RETRYING: TASK: master : Ensuring that the Cloudant Database is ready (12 retries left).
FAILED - RETRYING: TASK: master : Ensuring that the Cloudant Database is ready (11 retries left).
FAILED - RETRYING: TASK: master : Ensuring that the Cloudant Database is ready (10 retries left).
FAILED - RETRYING: TASK: master : Ensuring that the Cloudant Database is ready (9 retries left).
FAILED - RETRYING: TASK: master : Ensuring that the Cloudant Database is ready (8 retries left).
FAILED - RETRYING: TASK: master : Ensuring that the Cloudant Database is ready (7 retries left).
FAILED - RETRYING: TASK: master : Ensuring that the Cloudant Database is ready (6 retries left).
FAILED - RETRYING: TASK: master : Ensuring that the Cloudant Database is ready (5 retries left).
FAILED - RETRYING: TASK: master : Ensuring that the Cloudant Database is ready (4 retries left).
FAILED - RETRYING: TASK: master : Ensuring that the Cloudant Database is ready (3 retries left).
FAILED - RETRYING: TASK: master : Ensuring that the Cloudant Database is ready (2 retries left).
FAILED - RETRYING: TASK: master : Ensuring that the Cloudant Database is ready (1 retries left).
fatal: [10.20.30.29] => Status code was not [200]: Request failed: <urlopen error [Errno 111] Connection refused>

PLAY RECAP *************************************************************************************************************************************************************************************************
10.20.30.29                : ok=149  changed=57   unreachable=0    failed=1

The container is still using CPU in /usr/bin/python /usr/bin/ansible-playbook -e @cluster/config.yaml playbook/site.yaml so I think it's still installing. How do I increase the number of retries?

score 1 · Accepted Answer · answered Oct 05 '17 at 15:26

1

Run the command

docker info | grep -i cgroup

and you should see

Cgroup Driver: systemd

...so matched that in ICP by adding the line

kubelet_extra_args: ["--cgroup-driver=systemd"]

to ICP's config.yaml

answered Oct 05 '17 at 15:26

Ali Kaba

119
6

score 0 · Answer 2 · answered Oct 05 '17 at 01:57

This usually means cloudant db takes too long to be ready and by that time, 20 times retry limit has been reached. Uninstall ICP 2.1 beta2 and install it usually gets past this issue. In the ICP 2.1 beta3, it will have an increased timeout.

If this does not work, you can check the following:

docker exec -it <cloudant container> bash

now inside the container, to check the database status:

cast cluster status -p $ADMIN_PASSWORD

Also can check:

docker logs <long cloudant container name>

Before installing ICP beta2 again, check whether docker volume ls was cleaned up.

score 0 · Answer 3 · answered Oct 13 '17 at 02:56

In some cases, we find that the size of the ICP datastore, which is based on Cloudant, takes awhile to download. Have you considered pre-pulling the image onto your host?

docker pull ibmcom/icp-datastore:{version}

You would need to do this on each master (but not on the worker nodes).

ICP 2.1 Beta 2 can't create Cloudant DB fast enough

3 Answers3