1

I've setup OpenStack - my first time. I've followed the installation guide (on Ubuntu 14.04) and I believe I've got everything correct. However, when trying to create an instance I get the error:

Build of instance e948205a-2287-4b0e-9829-f2c6fe3a93eb aborted: Unable to establish connection to http://127.0.0.1:9696/v2.0/extensions.json

I've been debugging this on and off for around a week now, and I've exhausted every solution I've found on Google, and that I can think of.

Neutron is listening on the port:

netstat -ln | grep 9696 tcp 0
0 0.0.0.0:9696 0.0.0.0:*
LISTEN

I can also connect to Neutron from the command line:

root@controller:/var/log# neutron ext-list
+---------------------------+-----------------------------------------------+
| alias                     | name                                          |
+---------------------------+-----------------------------------------------+
| default-subnetpools       | Default Subnetpools                           |
| network-ip-availability   | Network IP Availability                       |
| network_availability_zone | Network Availability Zone                     |
| auto-allocated-topology   | Auto Allocated Topology Services              |
| ext-gw-mode               | Neutron L3 Configurable external gateway mode |
| binding                   | Port Binding  
...

I can authenticate with a token and get a response from Neutron manually with the curl test below, this responds with expected data.

curl -i -H'X-Auth-Token: MY_TOKEN' http://controller:9696/v2.0/extensions.json

My neutron api endpoints:

| 440735e5dd8d468e89de9a8077ea9491 | RegionOne | neutron | network | True | public | http://controller:9696 |

| 5985d944b02b4148a87cafae8df006c8 | RegionOne | neutron | network | True | internal | http://controller:9696 |

I've checked all of the Nova and Neutron logs, and i can't see any errors in there, I've recreated my Neutron endpoints, I've checked and double checked configs. I'm at a loss how to continue debugging or where the solution lies.

Edit with Updated Info:

root@controller:~# openstack catalog list
+----------+----------+--------------------------------------------------------------------------+
| Name     | Type     | Endpoints                                                                |
+----------+----------+--------------------------------------------------------------------------+
| neutron  | network  | RegionOne                                                                |
|          |          |   admin: http://controller:9696                                          |
|          |          | RegionOne                                                                |
|          |          |   internal: http://controller:9696                                       |
|          |          | RegionOne                                                                |
|          |          |   public: http://controller:9696                                         |
|          |          |                                                                          |
| nova     | compute  | RegionOne                                                                |
|          |          |   admin: http://controller:8774/v2.1/225f1282f5034949937f5ef71c503dd0    |
|          |          | RegionOne                                                                |
|          |          |   internal: http://controller:8774/v2.1/225f1282f5034949937f5ef71c503dd0 |
|          |          | RegionOne                                                                |
|          |          |   public: http://controller:8774/v2.1/225f1282f5034949937f5ef71c503dd0   |
|          |          |                                                                          |
| keystone | identity | RegionOne                                                                |
|          |          |   admin: http://controller:35357/v3                                      |
|          |          | RegionOne                                                                |
|          |          |   public: http://controller:5000/v3                                      |
|          |          | RegionOne                                                                |
|          |          |   internal: http://controller:5000/v3                                    |
|          |          |                                                                          |
| glance   | image    | RegionOne                                                                |
|          |          |   admin: http://controller:9292                                          |
|          |          | RegionOne                                                                |
|          |          |   public: http://controller:9292                                         |
|          |          | RegionOne                                                                |
|          |          |   internal: http://controller:9292                                       |
|          |          |                                                                          |
+----------+----------+--------------------------------------------------------------------------+

Neutron networks i've created via Horizon and the cli

root@controller:~# neutron net-list
+--------------------------------------+------------------+--------------------------------------------------+
| id                                   | name             | subnets                                          |
+--------------------------------------+------------------+--------------------------------------------------+
| 8a50aef6-b687-483d-ab49-f43460eebdd6 | My Admin Network | d4f840cc-31b9-4d32-963d-3a2110ae5765 10.0.0.0/24 |
| 8bb223ef-828d-4c9a-aa16-15ac13d244f6 | ext-net          |                                                  |
| 98a2606a-3017-48fb-8b60-e18c4a4f1083 | My Network       | b76352ac-dc61-4d75-ba7a-b5b4d76705d0 10.0.0.0/24 |
+--------------------------------------+------------------+--------------------------------------------------+

My Neutron Ext list is below:

root@controller:~# neutron ext-list
+---------------------------+-----------------------------------------------+
| alias                     | name                                          |
+---------------------------+-----------------------------------------------+
| default-subnetpools       | Default Subnetpools                           |
| network-ip-availability   | Network IP Availability                       |
| network_availability_zone | Network Availability Zone                     |
| auto-allocated-topology   | Auto Allocated Topology Services              |
| ext-gw-mode               | Neutron L3 Configurable external gateway mode |
| binding                   | Port Binding                                  |
| agent                     | agent                                         |
| subnet_allocation         | Subnet Allocation                             |
| l3_agent_scheduler        | L3 Agent Scheduler                            |
| tag                       | Tag support                                   |
| external-net              | Neutron external network                      |
| net-mtu                   | Network MTU                                   |
| availability_zone         | Availability Zone                             |
| quotas                    | Quota management support                      |
| l3-ha                     | HA Router extension                           |
| provider                  | Provider Network                              |
| multi-provider            | Multi Provider Network                        |
| address-scope             | Address scope                                 |
| extraroute                | Neutron Extra Route                           |
| timestamp_core            | Time Stamp Fields addition for core resources |
| router                    | Neutron L3 Router                             |
| extra_dhcp_opt            | Neutron Extra DHCP opts                       |
| dns-integration           | DNS Integration                               |
| security-group            | security-group                                |
| dhcp_agent_scheduler      | DHCP Agent Scheduler                          |
| router_availability_zone  | Router Availability Zone                      |
| rbac-policies             | RBAC Policies                                 |
| standard-attr-description | standard-attr-description                     |
| port-security             | Port Security                                 |
| allowed-address-pairs     | Allowed Address Pairs                         |
| dvr                       | Distributed Virtual Router                    |
+---------------------------+-----------------------------------------------+

With the create command with debug, i now have some further info in the actual error (will be best to copy and paste this out to read it):

{u'message': u'Build of instance 3e2f5678-6a4a-478b-8953-d10db929d367 aborted: Unable to establish connection to http://127.0.0.1:9696/v2.0/extensions.json', u'code': 500, u'details': u'  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1926, in _do_build_and_run_instance\n    filter_properties)\n  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2083, in _build_and_run_instance\n    \'create.error\', fault=e)\n  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__\n    self.force_reraise()\n  File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise\n    six.reraise(self.type_, self.value, self.tb)\n  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2067, in _build_and_run_instance\n    instance=instance)\n  File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__\n    self.gen.throw(type, value, traceback)\n  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2244, in _build_resources\n    reason=six.text_type(exc))\n', u'created': u'2016-04-29T13:56:55Z'} |
epea
  • 406
  • 1
  • 9
  • 19

1 Answers1

1

you can debug this by checking if the neutron service is up and running and registered correctly in your service catalog.

look at your keystone catalog

it will ouput the API endpoints

keystone catalog
WARNING: unsupported identity-api-version 3, falling back to 2.0
Service: compute
+-------------+-------------------------------------------------------------+
|   Property  |                            Value                            |
+-------------+-------------------------------------------------------------+
|   adminURL  | http://172.25.60.2:8774/v2/6cbc4e8a49af4fb1be9a4bf6dda8e373 |
|      id     |               15f3972a815642118f319301c01eac26              |
| internalURL | http://172.25.60.2:8774/v2/6cbc4e8a49af4fb1be9a4bf6dda8e373 |
|  publicURL  |  http://87.x.x.x:8774/v2/6cbc4e8a49af4fb1be9a4bf6dda8e373  |
|    region   |                          RegionOne                          |
+-------------+-------------------------------------------------------------+
Service: network
+-------------+----------------------------------+
|   Property  |              Value               |
+-------------+----------------------------------+
|   adminURL  |     http://172.25.60.2:9696/     |
|      id     | 15c3f13c2bd74cd38bbd48bcafe4ccb0 |
| internalURL |     http://172.25.60.2:9696/     |
|  publicURL  |      http://87.x.x.x:9696/      |
|    region   |            RegionOne             |
+-------------+----------------------------------+
Service: volumev2
+-------------+-------------------------------------------------------------+
|   Property  |                            Value                            |
+-------------+-------------------------------------------------------------+
|   adminURL  | http://172.25.60.2:8776/v2/6cbc4e8a49af4fb1be9a4bf6dda8e373 |
|      id     |               979fd7e2bb7a4fef82b9e975249decf5              |
| internalURL | http://172.25.60.2:8776/v2/6cbc4e8a49af4fb1be9a4bf6dda8e373 |
|  publicURL  |  http://87.x.x.x.:8776/v2/6cbc4e8a49af4fb1be9a4bf6dda8e373  |
|    region   |                          RegionOne                          |
+-------------+-------------------------------------------------------------+

then what is the output of the neutron net-list command

neutron net-list
+--------------------------------------+----------------------+-------------------------------------------------------+
| id                                   | name                 | subnets                                               |
+--------------------------------------+----------------------+-------------------------------------------------------+
| 2f97b5e6-e13b-47af-9cb7-f7e33344a788 | default              | f25c7778-33f9-48a7-9696-a19a01fc3e57 192.168.112.0/24 |
| 34263fbb-f93e-45d3-b65e-ab2afce3c1c9 | ruby-network         | 9360aeb5-2c05-431d-8e79-7052788db198 10.0.14.0/24     |
| 3650dbcb-b6d7-4220-8e6d-fa7c30914c57 | cloudfoundry-network | 24ed907f-476f-4440-97d4-9a1f2b7bd75d 10.0.26.0/24     |

in your neutron.conf are you binding to 127.0.0.1 ? - you should bind to all IP's

# Address to bind the API server to
# bind_host = 0.0.0.0
bind_host = 0.0.0.0

# Port the bind the API server to
# bind_port = 9696
bind_port = 9696

Neutron uses api extensions or 'plugins' and also agents for DHCP, security groups and NAT- can you go on to say what extension you are using?

probably this will be openvswitch as your plugin and network agents by default are usually dnsmaq, and iptables.

you can check the openvswitch services in /etc/ini.d/ and look at the switch database with the following show commands

ovs-vsctl show : Prints a brief overview of the switch database configuration.
ovs-vsctl list-br : Prints a list of configured bridges
ovs-vsctl list-ports <bridge> : Prints a list of ports on a specific bridge.
ovs-vsctl list interface : Prints a list of interfaces.

also, for debugging the extension http error, you can enable debug logging in the neutron conf or pass in the --debug option on the cli e.g. 

nova --debug boot --image imageID --flavor flavorID --nic net-id=nicID
Sum1sAdmin
  • 1,934
  • 1
  • 12
  • 20
  • I've edited my original question with the info you've requested, everything seems ok as far as i can see. The Neutron config is as you've quoted, it is bound to all IPs. Thank you. – epea Apr 27 '16 at 17:23
  • updated the answer, can you look for the extension services - thanks. – Sum1sAdmin Apr 27 '16 at 21:42
  • Thanks for sticking with this. I've added my neutron extensions to the original question, and also with the --debug flag i've gotten a python stacktrace with the create failure. I don't think i'm using openvswitch as i have no init scripts for that and no commands. I followed the guide on http://docs.openstack.org/mitaka/install-guide-ubuntu/neutron-controller-install.html – epea Apr 29 '16 at 14:07
  • when you go into Horizon > System > System Information > Network Agents. is anything marked down? – Sum1sAdmin Apr 29 '16 at 14:36
  • I gave up diagnosing, reimaged and reinstalled OpenStack. I have no idea what the problem was, and I don't think I never will. The only thing i can think is that i didn't follow the part of the guide to setup the provider network, but it may well have been something else. I'lll accept this answer anyway as i'm sure it will be useful diagnostics for other users. – epea Apr 29 '16 at 20:49