1

I am attempting to get a simple 2-node deployment set up via devstack. I have followed both the multi-node lab and the "Using devstack with Neutron" guide. i made the most progress with the latter. however, i still cannot seem to communicate with instances running on my compute-only node. instances that run on the controller/compute node seem fine. i can ping/ssh to them from any machine in my office.

my environment: 2 ubuntu 18.04 bare metal servers, private network with a router handing out DHCP addresses (i have a small range of addresses set aside). i disabled Ubuntu NetworkManager and configured via ifupdown in /etc/network/interfaces:

auto enp0s31f6
iface enp0s31f6 inet static
    address 192.168.7.170
    netmask 255.255.255.0
    gateway 192.168.7.254
    multicast 192.168.7.255
    dns-nameservers 8.8.8.8 8.8.4.4

controller/compute node local.conf is configured according to the guide:

[[local|localrc]]

HOST_IP=192.168.7.170
SERVICE_HOST=192.168.7.170
MYSQL_HOST=192.168.7.170
RABBIT_HOST=192.168.7.170
GLANCE_HOSTPORT=192.168.7.170:9292
DATABASE_PASSWORD=Passw0rd
RABBIT_PASSWORD=Passw0rd
SERVICE_PASSWORD=Passw0rd
ADMIN_PASSWORD=Passw0rd

LOGFILE=/opt/stack/logs/stack.sh.log

## Neutron options
Q_USE_SECGROUP=True
FLOATING_RANGE="192.168.7.0/24"
IPV4_ADDRS_SAFE_TO_USE="10.0.0.0/22"
Q_FLOATING_ALLOCATION_POOL=start=192.168.7.249,end=192.168.7.253
PUBLIC_NETWORK_GATEWAY="192.168.7.254"
PUBLIC_INTERFACE=enp0s31f6

# Open vSwitch provider networking configuration
Q_USE_PROVIDERNET_FOR_PUBLIC=True
Q_ASSIGN_GATEWAY_TO_PUBLIC_BRIDGE=False
OVS_PHYSICAL_BRIDGE=br-ex
PUBLIC_BRIDGE=br-ex
OVS_BRIDGE_MAPPINGS=public:br-ex

the one difference is in Q_ASSIGN_GATEWAY_TO_PUBLIC_BRIDGE. i found if i did not set this, i saw a lot of packet loss on the server. i don't understand why the gateway would be added to the vSwitch as a secondary address.

another oddity that i noticed is that once the OVS bride was set up and my public interface added as a port, the network gateway no longer worked as a DNS server. if i use google's it's fine.

on the compute only node i have local.conf:

[[local|localrc]]

HOST_IP=192.168.7.172
LOGFILE=/opt/stack/logs/stack.sh.log

SERVICE_HOST=192.168.7.170
MYSQL_HOST=$SERVICE_HOST
RABBIT_HOST=$SERVICE_HOST
GLANCE_HOSTPORT=$SERVICE_HOST:9292

ADMIN_PASSWORD=Passw0rd
DATABASE_PASSWORD=Passw0rd
RABBIT_PASSWORD=Passw0rd
SERVICE_PASSWORD=Passw0rd

PUBLIC_INTERFACE=enp0s31f6
ENABLED_SERVICES=n-cpu,rabbit,q-agt,placement-client

i run stack.sh on the controller/compute node, then the compute only node. the installation looks good. i can set up the security group, ssh keypair etc. and launch instances. i allocate floating IPs for each and associate. the addresses come from the pool as expected. i can see the tunnels set up on each node with OVS:

controller$ sudo ovs-vsctl show
1cc8a95d-660d-453f-9772-02393adc2031
    Manager "ptcp:6640:127.0.0.1"
        is_connected: true
    Bridge br-tun
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port br-tun
            Interface br-tun
                type: internal
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "vxlan-c0a807ac"
            Interface "vxlan-c0a807ac"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="192.168.7.170", out_key=flow, remote_ip="192.168.7.172"}
    Bridge br-ex
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port br-ex
            Interface br-ex
                type: internal
        Port phy-br-ex
            Interface phy-br-ex
                type: patch
                options: {peer=int-br-ex}
        Port "enp0s31f6"
            Interface "enp0s31f6"
    Bridge br-int
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port "qg-7db4efa8-8f"
            tag: 2
            Interface "qg-7db4efa8-8f"
                type: internal
        Port "tap88eb8a36-86"
            tag: 1
            Interface "tap88eb8a36-86"
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port int-br-ex
            Interface int-br-ex
                type: patch
                options: {peer=phy-br-ex}
        Port br-int
            Interface br-int
                type: internal
        Port "qr-e0e43871-2d"
            tag: 1
            Interface "qr-e0e43871-2d"
                type: internal
        Port "qvo5a54876d-0c"
            tag: 1
            Interface "qvo5a54876d-0c"
        Port "qr-9452dacf-82"
            tag: 1
            Interface "qr-9452dacf-82"
                type: internal
    ovs_version: "2.8.1"

and on the compute-only node:

compute$ sudo ovs-vsctl show
c817878d-7127-4d17-9a69-4ff296adc157
    Manager "ptcp:6640:127.0.0.1"
        is_connected: true
    Bridge br-int
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port "qvo1b56a018-10"
            tag: 1
            Interface "qvo1b56a018-10"
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port br-int
            Interface br-int
                type: internal
        Port int-br-ex
            Interface int-br-ex
                type: patch
                options: {peer=phy-br-ex}
    Bridge br-ex
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port phy-br-ex
            Interface phy-br-ex
                type: patch
                options: {peer=int-br-ex}
        Port br-ex
            Interface br-ex
                type: internal
    Bridge br-tun
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port "vxlan-c0a807aa"
            Interface "vxlan-c0a807aa"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="192.168.7.172", out_key=flow, remote_ip="192.168.7.170"}
        Port br-tun
            Interface br-tun
                type: internal
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
    ovs_version: "2.8.1"

any ideas on what might be wrong in my set up? any suggestions for what i might try?

jmer
  • 371
  • 2
  • 12

1 Answers1

1

It turned out that the Ubuntu Network Manager was reasserting itself after i had stopped the services. i took a more drastic step of disabling and purging it from my servers: systemctl disable NetworkManager.service and then apt-get purge network-manager

once it was gone for good, things started working as advertised. i started with the local.conf above and was able to spin up instances on both servers, and connect to them and they had no trouble connecting to each other etc. i then added more pieces to my stack (heat, magnum, barbican, lbaasv2) and things continue to be reliable.

the moral of the story is: Ubuntu Network Manager and devstack ovs config do not play well together. to get the latter working, you must remove the former (as near as i can tell).

also, prior to all this trouble with ovs, i had to apply a proposed fix to devstack's lib/etcd3 script on my compute-only node. it's a small, but required, change in the stable/queens branch as of 27-Sept-2018. see https://github.com/openstack-dev/devstack/commit/19279b0f8. without this stack.sh fails on the compute node trying to bind to an address on the controller node.

jmer
  • 371
  • 2
  • 12