Nova service fails after rebooting

Question

Devstack installed on Virtualbox VM with Ubuntu 20.04

Succesfully created Cirros tiny VM. But after rebooting this 3 services are failed with similar errors: devstack@n-cond-cell1.service, devstack@n-sch.service, devstack@n-super-cond.service

stack@controller:~$ systemctl status devstack@n-cond-cell1.service --no-pager -l
● devstack@n-cond-cell1.service - Devstack devstack@n-cond-cell1.service
     Loaded: loaded (/etc/systemd/system/devstack@n-cond-cell1.service; enabled; vendor preset:   enabled)
     Active: failed (Result: exit-code) since Wed 2022-04-20 11:58:03 +04; 1h 19min ago
    Process: 1075 ExecStart=/usr/local/bin/nova-conductor --config-file /etc/nova/nova_cell1.con  f (code=exited, status=1/FAILURE)
   Main PID: 1075 (code=exited, status=1/FAILURE)

Apr 20 11:58:02 controller nova-conductor[1075]: ERROR nova   File "/usr/local/lib/python3.8/dis  t-packages/keystoneauth1/identity/base.py", line 271, in get_endpoint_data
Apr 20 11:58:02 controller nova-conductor[1075]: ERROR nova     service_catalog = self.get_acces  s(session).service_catalog
Apr 20 11:58:02 controller nova-conductor[1075]: ERROR nova   File "/usr/local/lib/python3.8/dis  t-packages/keystoneauth1/identity/base.py", line 134, in get_access
Apr 20 11:58:02 controller nova-conductor[1075]: ERROR nova     self.auth_ref = self.get_auth_re  f(session)
Apr 20 11:58:02 controller nova-conductor[1075]: ERROR nova   File "/usr/local/lib/python3.8/dis  t-packages/keystoneauth1/identity/generic/base.py", line 206, in get_auth_ref
Apr 20 11:58:02 controller nova-conductor[1075]: ERROR nova     self._plugin = self._do_create_p  lugin(session)
Apr 20 11:58:02 controller nova-conductor[1075]: ERROR nova   File "/usr/local/lib/python3.8/dis  t-packages/keystoneauth1/identity/generic/base.py", line 158, in _do_create_plugin
Apr 20 11:58:02 controller nova-conductor[1075]: ERROR nova     raise exceptions.DiscoveryFailur  e(
Apr 20 11:58:02 controller nova-conductor[1075]: ERROR nova keystoneauth1.exceptions.discovery.D  iscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Pl  ease check that your auth_url is correct. Unable to establish connection to http://65.152.254.10  0/identity: HTTPConnectionPool(host='65.152.254.100', port=80): Max retries exceeded with url: /  identity (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f458129d  7f0>: Failed to establish a new connection: [Errno 111] ECONNREFUSED'))
Apr 20 11:58:02 controller nova-conductor[1075]: ERROR nova

How to fix it?

keystone is usually managed by apache, so I would check if apache is running first. — eblock, Apr 21 '22 at 10:26

score 0 · Answer 1 · answered Apr 20 '22 at 12:11

According to this error tip: Unable to establish connection to http://65.152.254.100/identity, make sure that you had source the right rc file such as admin-openrc.sh and check the environment parameter auth_url. Then check the network connection whether success to (host='65.152.254.100', port=80), maybe there is some firewall limit like iptables.

Mostly someone miss the IDENTITY_API_VERSION in the auth_url.

In my openstack development environment which deploy by kolla-ansible although, I guess it's similar with your issue. And you should check whether the endpoints value works or not.

# source /etc/kolla/admin-openrc.sh
# echo $OS_AUTH_URL 
http://10.10.10.100:35357/v3

# openstack endpoint list
+-----+-----------+--------------+----------------+---------+-----------+-----------------------------+
| ID  | Region    | Service Name | Service Type   | Enabled | Interface | URL                         |
+-----+-----------+--------------+----------------+---------+-----------+-----------------------------+
| 1*b | RegionOne | keystone     | identity       | True    | admin     | http://10.10.10.100:35357   |

score 0 · Answer 2 · answered Apr 21 '22 at 04:31

I using default source file. And there is no any firewall or iptables.

stack@controller:~$ source devstack/openrc admin admin
WARNING: setting legacy OS_TENANT_NAME to support cli tools.
stack@controller:~$ echo $OS_AUTH_URL
http://65.152.254.100/identity
stack@controller:~$ openstack endpoint list
/usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
  from cryptography.utils import int_from_bytes
/usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
  from cryptography.utils import int_from_bytes
+----------------------------------+-----------+--------------+----------------+---------+-----------+-------------------------------------------------+
| ID                               | Region    | Service Name | Service Type   | Enabled | Interface | URL                                             |
+----------------------------------+-----------+--------------+----------------+---------+-----------+-------------------------------------------------+
| 0cba67915a71447fa64c14151df088a0 | RegionOne | placement    | placement      | True    | public    | http://65.152.254.100/placement                 |
| 17deafcc0235418181fdf6d235f0c0c5 | RegionOne | glance       | image          | True    | public    | http://65.152.254.100/image                     |
| 444fcc4115c7425fb6467b426160bef9 | RegionOne | nova_legacy  | compute_legacy | True    | public    | http://65.152.254.100/compute/v2/$(project_id)s |
| 524f4b0536b04c388226952bc845a401 | RegionOne | nova         | compute        | True    | public    | http://65.152.254.100/compute/v2.1              |
| 85f6c215d99d49a79c656f8623f7e365 | RegionOne | neutron      | network        | True    | public    | http://65.152.254.100:9696/                     |
| c35b8edd701a44848621e0a57c7bca53 | RegionOne | cinder       | block-storage  | True    | public    | http://65.152.254.100/volume/v3/$(project_id)s  |
| c48c074301734601a3d640b19ad93573 | RegionOne | cinderv3     | volumev3       | True    | public    | http://65.152.254.100/volume/v3/$(project_id)s  |
| ee7c8cd13d7d47428aaf7b79c519042d | RegionOne | keystone     | identity       | True    | public    | http://65.152.254.100/identity                  |
+----------------------------------+-----------+--------------+----------------+---------+-----------+-------------------------------------------------+

what's the result of `nc -vz 65.152.254.100 80` and other port like 9696. — Victor Lee, Apr 21 '22 at 06:01

Nova service fails after rebooting

2 Answers2