neutron-linuxbridge-agent is crashing in Openstack Ussuri

Question

On our new Openstack Ussuri installation neutron-linuxbridge-agent crashes on both compute nodes. The installation followed the instructions for Ubuntu (20.04) at https://docs.openstack.org/install-guide

neutron-linuxbridge-agent.log shows

2021-02-09 20:40:30.921 54590 ERROR neutron.agent.linux.utils [req-c6741275-9bd6-47a3-894c-856cb5ad0e62 - - - - -] Exit code: 4; Stdin: ; Stdout: ; Stderr: ebtables v1.8.4 (nf_tables):  CHAIN_USER_DEL failed (Device or resource busy): chain neutronARP-tap0a9b5e3a-21
[...]
2021-02-09 20:40:30.923 54590 ERROR oslo_service.service neutron_lib.exceptions.ProcessExecutionError: Exit code: 4; Stdin: ; Stdout: ; Stderr: ebtables v1.8.4 (nf_tables):  CHAIN_USER_DEL failed (Device or resource busy): chain neutronARP-tap0a9b5e3a-21
2021-02-09 20:40:30.923 54590 ERROR oslo_service.service
2021-02-09 20:40:30.923 54590 ERROR oslo_service.service
2021-02-09 20:40:30.929 54590 INFO neutron.plugins.ml2.drivers.agent._common_agent [-] Stopping Linux bridge agent agent.

A restart of the agent produces the same result. Looks like a problem with nf_tables. But we did not configure anything with nf_tables manually. Any ideas?

relevant part of logfile: https://pastebin.com/7fSVBqdd

neutron config: https://pastebin.com/Yg0HpwXc

Edit: when I removed all the nf_tables rules mentioned in the log file, I could start the agent:

nft flush chain bridge nat neutronARP-tap0a9b5e3a-21

Of course this is not a solution, but just a quick workaround...

Answer 1

There is a bug in the code, and this patch will fix the problem:

diff --git a/neutron/plugins/ml2/drivers/linuxbridge/agent/arp_protect.py b/neutron/plugins/ml2/drivers/linuxbridge/agent/arp_protect.py
index d65e2bb..6ed3f7e 100644
--- a/neutron/plugins/ml2/drivers/linuxbridge/agent/arp_protect.py
+++ b/neutron/plugins/ml2/drivers/linuxbridge/agent/arp_protect.py
@@ -87,8 +87,7 @@
         ebtables(['-D', chain, '-i', vif, '-j',
                   chain_name(vif), '-p', 'ARP'], table=table)
     for vif in vifs:
-        if chain_exists(chain_name(vif), current_rules):
-            ebtables(['-X', chain_name(vif)], table=table)
+        chain_delete(chain_name(vif), table, current_rules)
     _delete_mac_spoofing_protection(vifs, current_rules, table=table,
                                     chain=chain)
 
@@ -154,6 +153,13 @@
     return False
 
 
+def chain_delete(chain, table, current_rules):
+    # flush and delete chain if exists
+    if chain_exists(chain, current_rules):
+        ebtables(['-F', chain], table=table)
+        ebtables(['-X', chain], table=table)
+
+
 def vif_jump_present(vif, current_rules):
     searches = (('-i %s' % vif), ('-j %s' % chain_name(vif)), ('-p ARP'))
     for line in current_rules:
@@ -212,9 +218,7 @@
         ebtables(['-D', chain, '-i', vif, '-j',
                   _mac_chain_name(vif)], table=table)
     for vif in vifs:
-        chain = _mac_chain_name(vif)
-        if chain_exists(chain, current_rules):
-            ebtables(['-X', chain], table=table)
+        chain_delete(_mac_chain_name(vif), table, current_rules)
 
 
 # Used to scope ebtables commands in testing

Ansible to deploy it:

---
- name: Neutron hotfix patch
  hosts: all
  tasks: 
    - name: copy patch 
      copy: 
        src: 2207b88.diff
        dest: /openstack/venvs/neutron-21.2.0/lib/python3.8/site-packages
    - name: Apply patch
      shell: cd /openstack/venvs/neutron-21.2.0/lib/python3.8/site-packages && git apply 2207b88.diff

    - systemd:
        name: neutron-linuxbridge-agent.service
        state: restarted