VoIP one-way audio, only when call initiated from one side

Question

I am setting up some new switches and VLANs and I am getting trouble with our pre-existing Asterisk VoIP set-up.

Most calls work ok. Some get just one-way audio. I tried to narrow it down to this specific test situation:

Asterisk server is 10.0.10.10.

Call that works ok:

If phone 10.0.2.183 calls phone on 10.0.10.47, audio flows well both ways. This working call is depicted in this Wireshark diagram:

Call that gets only one-way audio:

If instead, I swap the situation and initiate from the other phone, i.e. phone 10.0.10.47 calls phone on 10.0.2.183, audio flows only one-way: 10.0.2.183 can't hear 10.0.10.47. This faulty call is depicted in this Wireshark diagram:

NAT?

I see a lot of stuff online suggesting the typical cause of one-way audio is NAT. I don't think I am using NAT. How can I check? This is all internal inside my LAN.

Firewall?

I have spent a ton of time tinkering with the iptables firewall on 10.0.10.10, opening everything up (even if just provisionally, as a test). I don't see how it could be blocking this particular bit of traffic, while letting so much through. But I humbly submit that iptables is a wild beast that I cannot fully tame. Any ideas here?

Routing?

I also suspect possible routing issues, since I am moving servers around and isolating VLANs from each other (except the VoIP VLAN, which is 10.0.10.*, and should be accessible to all other VLANs). I had to play with the routing table on 10.0.10.10 to get some traffic going the way I want:

:/home1/_locals/operator# ip route
(1) 192.168.1.248/29 dev eth1  proto kernel  scope link  src 192.168.1.250
(2) 10.0.0.0/24 dev eth0  proto kernel  scope link  src 10.0.0.1
(3) 10.0.1.0/24 dev eth3  proto kernel  scope link  src 10.0.1.1
(4) 10.0.10.0/24 dev eth2  proto kernel  scope link  src 10.0.10.10
(5) 10.0.0.0/8 via 10.0.10.1 dev eth2
(6) default via 192.168.1.254 dev eth1

I've numbered the rules so you can comment on them if you want.

Any ideas on how to troubleshoot this? Why is everything going through except some very specific bits of RTP traffic? Thanks in advance for you help.

Additional requested info

DHCP Config in 10.0.10.0/24 network:

This should provide straight-forward routing tables on the phones, only 10.0.10.10 has static IP and the altered routing table provided further above.

Simplified network topology:

In reality this is a lot more funky, I am half-way in a move between two topologies. So I have two 10.0.0.0 subnets, for example. But they are supposedly kept separate. My problem probably arises from this funkiness, but I need to pinpoint exactly which bit of configuration is missing to make it work.

Routing table on 10.0.10.1, which is Ubiquiti UDM

# ip route
10.0.0.0/24 dev br3 proto kernel scope link src 10.0.0.1
10.0.1.0/24 dev br5 proto kernel scope link src 10.0.1.1
10.0.2.0/24 dev br6 proto kernel scope link src 10.0.2.1
10.0.3.0/24 dev br4 proto kernel scope link src 10.0.3.1
10.0.10.0/24 dev br8 proto kernel scope link src 10.0.10.1
10.1.1.0/24 dev br0 proto kernel scope link src 10.1.1.1
10.2.2.0/24 dev br2 proto kernel scope link src 10.2.2.1
192.168.1.0/24 dev eth4 proto kernel scope link src 192.168.1.86

Firewall on the same host:

# iptables-save
# Generated by iptables-save v1.6.1 on Fri May  7 22:04:20 2021
*nat
:PREROUTING ACCEPT [30400:5954281]
:INPUT ACCEPT [9407:875595]
:OUTPUT ACCEPT [31671:2057109]
:POSTROUTING ACCEPT [25363:1582041]
:UBIOS_INPUT_JUMP - [0:0]
:UBIOS_OUTPUT_JUMP - [0:0]
:UBIOS_POSTROUTING_JUMP - [0:0]
:UBIOS_POSTROUTING_USER_HOOK - [0:0]
:UBIOS_PREROUTING_JUMP - [0:0]
-A PREROUTING -j LOG --log-prefix "::PREROUTING:"
-A PREROUTING -j UBIOS_PREROUTING_JUMP
-A INPUT -j LOG --log-prefix "::INPUT:"
-A INPUT -j UBIOS_INPUT_JUMP
-A OUTPUT -j LOG --log-prefix "::OUTPUT:"
-A OUTPUT -j UBIOS_OUTPUT_JUMP
-A POSTROUTING -j LOG --log-prefix "::POSTROUTING:"
-A POSTROUTING -j UBIOS_POSTROUTING_JUMP
-A UBIOS_POSTROUTING_JUMP -j UBIOS_POSTROUTING_USER_HOOK
-A UBIOS_POSTROUTING_USER_HOOK -o eth4 -m comment --comment 00000001095216660481 -j MASQUERADE
COMMIT
# Completed on Fri May  7 22:04:20 2021
# Generated by iptables-save v1.6.1 on Fri May  7 22:04:20 2021
*mangle
:PREROUTING ACCEPT [5740653266:1557524250007]
:INPUT ACCEPT [5729417629:1548285462113]
:FORWARD ACCEPT [11076849:9225341544]
:OUTPUT ACCEPT [5729656413:1548109771107]
:POSTROUTING ACCEPT [5741373562:1557359586630]
:UBIOS_FORWARD_JUMP - [0:0]
:UBIOS_FORWARD_TCPMSS - [0:0]
:UBIOS_FORWARD_USER_HOOK - [0:0]
:UBIOS_INPUT_JUMP - [0:0]
:UBIOS_INPUT_USER_HOOK - [0:0]
:UBIOS_OUTPUT_JUMP - [0:0]
:UBIOS_OUTPUT_USER_HOOK - [0:0]
:UBIOS_POSTROUTING_JUMP - [0:0]
:UBIOS_POSTROUTING_SHAPER - [0:0]
:UBIOS_POSTROUTING_USER_HOOK - [0:0]
:UBIOS_PREROUTING_JUMP - [0:0]
:UBIOS_PREROUTING_USER_HOOK - [0:0]
-A PREROUTING -j UBIOS_PREROUTING_JUMP
-A INPUT -j UBIOS_INPUT_JUMP
-A FORWARD -j UBIOS_FORWARD_JUMP
-A OUTPUT -j UBIOS_OUTPUT_JUMP
-A POSTROUTING -j UBIOS_POSTROUTING_JUMP
-A UBIOS_FORWARD_JUMP -j UBIOS_FORWARD_TCPMSS
-A UBIOS_POSTROUTING_JUMP -j UBIOS_POSTROUTING_SHAPER
COMMIT
# Completed on Fri May  7 22:04:20 2021
# Generated by iptables-save v1.6.1 on Fri May  7 22:04:20 2021
*filter
:INPUT ACCEPT [3169950958:856215960609]
:FORWARD ACCEPT [4175578:2973708433]
:OUTPUT ACCEPT [3170232928:856108555985]
:UBIOS_FORWARD_IN_USER - [0:0]
:UBIOS_FORWARD_JUMP - [0:0]
:UBIOS_FORWARD_OUT_USER - [0:0]
:UBIOS_FORWARD_USER_HOOK - [0:0]
:UBIOS_INPUT_JUMP - [0:0]
:UBIOS_INPUT_USER_HOOK - [0:0]
:UBIOS_IN_GEOIP - [0:0]
:UBIOS_LAN_IN_USER - [0:0]
:UBIOS_LAN_LOCAL_USER - [0:0]
:UBIOS_LAN_OUT_USER - [0:0]
:UBIOS_OUTPUT_JUMP - [0:0]
:UBIOS_OUTPUT_USER_HOOK - [0:0]
:UBIOS_OUT_GEOIP - [0:0]
:UBIOS_WAN_IN_USER - [0:0]
:UBIOS_WAN_LOCAL_USER - [0:0]
:UBIOS_WAN_OUT_USER - [0:0]
-A INPUT -j LOG --log-prefix "::INPUT1:"
-A INPUT -j UBIOS_INPUT_JUMP
-A FORWARD -j LOG --log-prefix "::INPUT2:"
-A FORWARD -j UBIOS_FORWARD_JUMP
-A OUTPUT -j LOG --log-prefix "::INPUT3:"
-A OUTPUT -j UBIOS_OUTPUT_JUMP
-A UBIOS_FORWARD_IN_USER -i eth4 -m comment --comment 00000001095216663481 -j UBIOS_WAN_IN_USER
-A UBIOS_FORWARD_IN_USER -i br0 -m comment --comment 00000001095216663482 -j UBIOS_LAN_IN_USER
-A UBIOS_FORWARD_IN_USER -i br2 -m comment --comment 00000001095216663483 -j UBIOS_LAN_IN_USER
-A UBIOS_FORWARD_IN_USER -i br3 -m comment --comment 00000001095216663484 -j UBIOS_LAN_IN_USER
-A UBIOS_FORWARD_IN_USER -i br4 -m comment --comment 00000001095216663485 -j UBIOS_LAN_IN_USER
-A UBIOS_FORWARD_IN_USER -i br5 -m comment --comment 00000001095216663486 -j UBIOS_LAN_IN_USER
-A UBIOS_FORWARD_IN_USER -i br6 -m comment --comment 00000001095216663487 -j UBIOS_LAN_IN_USER
-A UBIOS_FORWARD_IN_USER -i br8 -m comment --comment 00000001095216663488 -j UBIOS_LAN_IN_USER
-A UBIOS_FORWARD_JUMP -j UBIOS_FORWARD_USER_HOOK
-A UBIOS_FORWARD_OUT_USER -o eth4 -m comment --comment 00000001095216663481 -j UBIOS_WAN_OUT_USER
-A UBIOS_FORWARD_OUT_USER -o br0 -m comment --comment 00000001095216663482 -j UBIOS_LAN_OUT_USER
-A UBIOS_FORWARD_OUT_USER -o br2 -m comment --comment 00000001095216663483 -j UBIOS_LAN_OUT_USER
-A UBIOS_FORWARD_OUT_USER -o br3 -m comment --comment 00000001095216663484 -j UBIOS_LAN_OUT_USER
-A UBIOS_FORWARD_OUT_USER -o br4 -m comment --comment 00000001095216663485 -j UBIOS_LAN_OUT_USER
-A UBIOS_FORWARD_OUT_USER -o br5 -m comment --comment 00000001095216663486 -j UBIOS_LAN_OUT_USER
-A UBIOS_FORWARD_OUT_USER -o br6 -m comment --comment 00000001095216663487 -j UBIOS_LAN_OUT_USER
-A UBIOS_FORWARD_OUT_USER -o br8 -m comment --comment 00000001095216663488 -j UBIOS_LAN_OUT_USER
-A UBIOS_FORWARD_USER_HOOK -m comment --comment 00000001095216663481 -j UBIOS_FORWARD_IN_USER
-A UBIOS_FORWARD_USER_HOOK -m comment --comment 00000001095216663482 -j UBIOS_FORWARD_OUT_USER
-A UBIOS_INPUT_JUMP -j UBIOS_INPUT_USER_HOOK
-A UBIOS_INPUT_USER_HOOK -i eth4 -m comment --comment 00000001095216663481 -j UBIOS_WAN_LOCAL_USER
-A UBIOS_INPUT_USER_HOOK -i br0 -m comment --comment 00000001095216663482 -j UBIOS_LAN_LOCAL_USER
-A UBIOS_INPUT_USER_HOOK -i br2 -m comment --comment 00000001095216663483 -j UBIOS_LAN_LOCAL_USER
-A UBIOS_INPUT_USER_HOOK -i br3 -m comment --comment 00000001095216663484 -j UBIOS_LAN_LOCAL_USER
-A UBIOS_INPUT_USER_HOOK -i br4 -m comment --comment 00000001095216663485 -j UBIOS_LAN_LOCAL_USER
-A UBIOS_INPUT_USER_HOOK -i br5 -m comment --comment 00000001095216663486 -j UBIOS_LAN_LOCAL_USER
-A UBIOS_INPUT_USER_HOOK -i br6 -m comment --comment 00000001095216663487 -j UBIOS_LAN_LOCAL_USER
-A UBIOS_INPUT_USER_HOOK -i br8 -m comment --comment 00000001095216663488 -j UBIOS_LAN_LOCAL_USER
-A UBIOS_LAN_IN_USER -d 10.0.10.10/32 -j LOG
-A UBIOS_LAN_IN_USER -d 10.0.10.10/32 -m comment --comment 00000001095216662480 -j RETURN
-A UBIOS_LAN_IN_USER -s 10.0.10.10/32 -j LOG
-A UBIOS_LAN_IN_USER -s 10.0.10.10/32 -m comment --comment 00000001095216662481 -j RETURN
-A UBIOS_LAN_IN_USER -s 10.0.2.0/24 -m comment --comment 00000001095216666481 -j RETURN
-A UBIOS_LAN_IN_USER -s 10.0.3.0/24 -m comment --comment 00000001095216666482 -j RETURN
-A UBIOS_LAN_IN_USER -s 10.0.1.0/24 -m comment --comment 00000001095216666483 -j RETURN
-A UBIOS_LAN_IN_USER -s 10.2.2.0/24 -m comment --comment 00000001095216666484 -j RETURN
-A UBIOS_LAN_IN_USER -s 10.0.10.0/24 -m comment --comment 00000001095216666485 -j RETURN
-A UBIOS_LAN_IN_USER -s 10.1.1.0/24 -m comment --comment 00000001095216666486 -j RETURN
-A UBIOS_LAN_IN_USER -s 10.0.0.0/24 -m comment --comment 00000001095216666487 -j RETURN
-A UBIOS_LAN_IN_USER -j LOG
-A UBIOS_LAN_IN_USER -m comment --comment 00000001097364144127 -j RETURN
-A UBIOS_LAN_LOCAL_USER -j LOG
-A UBIOS_LAN_LOCAL_USER -m comment --comment 00000001097364144127 -j RETURN
-A UBIOS_LAN_OUT_USER -d 10.0.2.0/24 -m comment --comment 00000001095216666481 -j RETURN
-A UBIOS_LAN_OUT_USER -d 10.0.3.0/24 -m comment --comment 00000001095216666482 -j RETURN
-A UBIOS_LAN_OUT_USER -d 10.0.1.0/24 -m comment --comment 00000001095216666483 -j RETURN
-A UBIOS_LAN_OUT_USER -d 10.2.2.0/24 -m comment --comment 00000001095216666484 -j RETURN
-A UBIOS_LAN_OUT_USER -d 10.0.10.0/24 -m comment --comment 00000001095216666485 -j RETURN
-A UBIOS_LAN_OUT_USER -d 10.1.1.0/24 -m comment --comment 00000001095216666486 -j RETURN
-A UBIOS_LAN_OUT_USER -d 10.0.0.0/24 -m comment --comment 00000001095216666487 -j RETURN
-A UBIOS_LAN_OUT_USER -j LOG
-A UBIOS_LAN_OUT_USER -m comment --comment 00000001097364144127 -j RETURN
-A UBIOS_WAN_IN_USER -m conntrack --ctstate RELATED,ESTABLISHED -m comment --comment 00000001095216663481 -j RETURN
-A UBIOS_WAN_IN_USER -m conntrack --ctstate INVALID -m comment --comment 00000001095216663482 -j DROP
-A UBIOS_WAN_IN_USER -m comment --comment 00000001097364144127 -j DROP
-A UBIOS_WAN_LOCAL_USER -m conntrack --ctstate RELATED,ESTABLISHED -m comment --comment 00000001095216663481 -j RETURN
-A UBIOS_WAN_LOCAL_USER -m conntrack --ctstate INVALID -m comment --comment 00000001095216663482 -j DROP
-A UBIOS_WAN_LOCAL_USER -m comment --comment 00000001097364144127 -j DROP
-A UBIOS_WAN_OUT_USER -m comment --comment 00000001097364144127 -j RETURN
COMMIT

I don't have any port forwarding configured. I am not versed in NAT so if there's any other place I should be checking, please advise in the comments.

Please show output of `ip route` on `10.0.2.183` and `10.0.10.47`. Please also add the firewall / NAT configuration on `10.0.10.1`. Also, a network diagram is needed to understand the network topology. — Tero Kilkanen, May 07 '21 at 14:48
@TeroKilkanen ok, I'll work on that. But 10.0.10.47 is a Grandstream GXP-2000 phone, I am not sure I can get a routing table from there... but it should be straight-forward. And we know it is able to talk to the other devices, just not always... I'll be back with more info soon. Thanks. — pgr, May 07 '21 at 14:52
Then you should add information from DHCP server that is providing addresses for the `10.0.10.0/24` network. — Tero Kilkanen, May 07 '21 at 14:56
I've edited to add _some_ of the information you requested. About this "Please also add the firewall / NAT configuration on 10.0.10.1", do you really mean 10.0.0.1? Not 10.0.10.10? Thanks. — pgr, May 07 '21 at 15:30
`10.0.10.1` is the router for `10.0.10.0/24` network, that is why its configuration is important to be visible. — Tero Kilkanen, May 07 '21 at 15:36
BTW, that complicated DROP rule with all the match-set parts is the rule to isolate VLANs from each other. But I just tried disabling it and it doesn't affect the problem I am describing here. — pgr, May 07 '21 at 15:53
Please add `iptables-save` output for `10.0.10.1`, so that all tables are included. — Tero Kilkanen, May 07 '21 at 19:48
This seems to be a SIP issue. The SIP invite is telling the phone to send RTP traffic to 10.0.0.1 (was the old asterisk server address) instead of 10.0.10.10 (new one). I can't figure out where in asterisk I can change this, but I will keep looking. — pgr, May 10 '21 at 14:31

score 0 · Accepted Answer · answered May 21 '21 at 11:37

Just to leave some information regarding how I eventually solved this...

My reasoning was: if the SIP negotiation was successful, as made obvious by the packet captures, then it really wasn't a routing or firewall problem. I mean, it could still be a firewall problem, if it was letting some things through but not others (like other ports etc). But since I knew my rules were generic enough, if these devices could find routes to each other and talk to each other, this wasn't the cause of my issue.

This was made obvious by starting the call from the other phone instead. The RTP traffic flows, it routes, it passes the firewall.

So I should be focusing on SIP issues, how the choices for RTP traffic destinations are made. Eventually I had a break-through by thinking the packets are getting sent somewhere, so where are they getting sent to?. I went looking (with tcpdump) and I found them reaching the UDM on 10.0.0.1 instead of 10.0.10.10.

Inspecting the contents of those SIP INVITE packets you see in the Wireshark diagrams above, I actually found references to 10.0.0.1. Why? Because my Asterisk server thinks it's both 10.0.0.1 and 10.0.10.10 (two interfaces) but that is only true on the old topology, not on the new. So, despite me having configured both phones to use 10.0.10.10 as their SIP server, and despite the fact that they obey when starting the call from one side, and despite the fact that when starting it from the other side they are able to obey for SIP negotiation, one side still (!!!) ends up sending RTP traffic to the wrong address.

How to fix this properly: fix Asterisk configuration. I don't know how, and since this is a server about to go deprecated, and to which I don't have full access, I decided not to try this route.

How I actually fixed it: during the time when this asterisk server is going to be progressively phased out, I am re-routing UDP traffic on these specific RTP ports from 10.0.0.1 to 10.0.10.10. This was hard to achieve, but it works! Read about it here: Routing traffic for specific port range