vlan
is weird.
To quote the pcap-filter man page:
vlan [vlan_id]
True if the packet is an IEEE 802.1Q VLAN packet. If [vlan_id]
is specified, only true if the packet has the specified vlan_id.
Note that the first vlan keyword encountered in expression
changes the decoding offsets for the remainder of expression on
the assumption that the packet is a VLAN packet. The vlan
[vlan_id] expression may be used more than once, to filter on
VLAN hierarchies. Each use of that expression increments the
filter offsets by 4.
For example:
vlan 100 && vlan 200
filters on VLAN 200 encapsulated within VLAN 100, and
vlan && vlan 300 && ip
filters IPv4 protocols encapsulated in VLAN 300 encapsulated
within any higher order VLAN.
The "Note that the first vlan keyword encountered in expression changes the decoding offsets for the remainder of expression on the assumption that the packet is a VLAN packet." is important here.
This means that the order of vlan
and other filters matters.
So:
if I pass (protocol_filter and vlan_filter), I see no packets
That's because "protocol_filter" is evaluated based on the assumption that the packet is not a VLAN packet; anything that passes that filter is not a VLAN packet, so "vlan_filter" fails.
But if I swap exp as (vlan_filter and protocol_filter), then packet filtering works
That's because "vlan_filter" checks for a VLAN packet and changes the way all subsequent tests are done, so that they assume the packet is a VLAN packet; "protocol_filter" is evaluated based on the assumption that the packet is a VLAN packet.
If I include vlan with "not" protocols, I get packets with protocol filter applied: eg: (((vlan 20)) and (not (udp port 3800)) and ((not (tcp)) and (not (udp port 53)) and (not (icmp)) and (not (ip6)) and (not (udp port 5353))))
That filter will match only VLAN packets and, if the packet is for VLAN 20, it will then do all the other checks under the assumption that the packet is a VLAN packet - for example, if you're capturing packets with Ethernet headers, and you're checking for IPv4 or IPv6, the other checks will look at the Ethernet type at an offset of 16, not 12.
If I dont include vlan with protocols, I get all packets including tcp, icmp, ip6 etc when bpf is: (not (udp port 3800)) and ((not (tcp)) and (not (udp port 53)) and (not (icmp)) and (not (ip6)) and (not (udp port 5353))))
That's a little confusing, and it could be argued that it's a bug in libpcap.
"udp port 3800" first checks "is this IPv4 or IPv6", which on Ethernet is done by checking whether the Ethernet type/length field has the value 0x0800 for IPv6 or 0x86dd for IPv6. If that matches for IPv4, it checks the IPv4 protocol field to see whether it's UDP; if it matches for IPv6, it checks the IPv6 next header field to see whether it's UDP. If it's UDP, it then checks the source and destination port numbers to see if either one is 3800; if so, the packet matches, otherwise it doesn't match.
"not (udp port 3800)" does the same checks, except that if any of the tests don't match, the packet does match, and if any of them match, the packet doesn't match.
A VLAN packet has 0x8100 in the Ethernet type/length field, so the checks for IPv4 and IPv6 don't match. Therefore, the "not" filters do match.
The bug here is that a UDP packet to or from port 3800 is an IPv4 or an IPv6 packet; if it's on a VLAN, it just doesn't happen to, on Ethernet, have 0x0800 or 0x86dd at an offset of 12 from the beginning of the packet, it has 0x8100.
(((not vlan)) and (not (udp port 3800)) and ((not (tcp)) and (not (udp port 53)) and (not (icmp)) and (not (ip6)) and (not (udp port 5353)))) get all packets including ip6, tcp, etc
That's another libpcap bug - just as "vlan", on Ethernet, causes all subsequent tests to look at the Ethernet type field after the VLAN header, so does "not vlan". But that's wrong for "not vlan", as, if it matches, the packet isn't a VLAN packet, so there is no VLAN header!
(((not vlan)) and (not (udp port 3800)) and (((tcp)) or ((udp port 53)) or ((icmp)) or ((ip6)) or ((udp port 5353)))) dont get even single packet
Same problem.
The libpcap/WinPcap bugs might not be easy to fix, but I'll look at them when I have time.