2

I just deployed openstack using kolla and kolla-ansible stable/pike version. I deployed on one control node and one compute node. It worked but while I try to use the horizon dashboard, the control node is very slow and locks up and the terminal outputs the following:

Message from syslogd@openstackcontroller at Nov  7 21:46:39 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [runc:[2:INIT]:10527]

Message from syslogd@openstackcontroller at Nov  7 21:46:39 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [fluentd:5005]

Message from syslogd@openstackcontroller at Nov  7 21:46:39 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#8 stuck for 23s! [keepalived:10664]

Message from syslogd@openstackcontroller at Nov  7 21:46:39 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#15 stuck for 23s! [java:5604]

Message from syslogd@openstackcontroller at Nov  7 21:46:39 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#5 stuck for 23s! [neutron-openvsw:3101]

Message from syslogd@openstackcontroller at Nov  7 21:46:39 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [cinder-schedule:3193]

Message from syslogd@openstackcontroller at Nov  7 21:46:39 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#7 stuck for 23s! [java:6186]

Message from syslogd@openstackcontroller at Nov  7 21:46:39 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [docker-containe:6601]

Message from syslogd@openstackcontroller at Nov  7 21:46:39 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#12 stuck for 22s! [keepalived:4295]

Message from syslogd@openstackcontroller at Nov  7 21:46:39 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [keepalived:10666]

Message from syslogd@openstackcontroller at Nov  7 21:46:39 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#9 stuck for 22s! [gmain:939]

Message from syslogd@openstackcontroller at Nov  7 21:46:39 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#13 stuck for 22s! [node:3261]

Message from syslogd@openstackcontroller at Nov  7 21:46:39 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [neutron-l3-agen:5071]

Message from syslogd@openstackcontroller at Nov  7 21:46:39 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#14 stuck for 22s! [irqbalance:898]

Whenever I stop all the docker containers that are running on the control node it seems to not have any more CPU lockup errors but as soon as I start up all the docker containers again that are running openstack services on the control node it starts to run very slow again and the lockup errors start again

Joey Corkey
  • 475
  • 1
  • 6
  • 19

1 Answers1

3

Can you use a newer Linux kernel > 4.6.x?

Making the kernel panic on soft lockup and reboot on panic will help recover the system to some extent. They worked for me.

sysctl kernel.softlockup_panic=1  # this will panic on soft lockup
sysctl kernel.panic=60            # reset system on panic after 60 seconds

The above kernel settings worked for a few people per https://github.com/moby/moby/issues/19758#issuecomment-272430028.

Vikram Hosakote
  • 3,528
  • 12
  • 23