Is .apply_on_packets 's timeout unpredictable?

Question

The following code gives unpredictable results with the following advice in use:

import pyshark
import pandas as pd
import asyncio

def ProcessPackets(packet):
    global packet_list
    packet_version = packet.layers[1].version
    layer_name = packet.layers[2].layer_name
    packet_list.append([packet_version, layer_name, packet.length, packet.sniff_time])

def Capture(timeOrPath):
    global packet_list
    packet_list=[]
    try:
        timeout=int(timeOrPath)
        capture = pyshark.LiveCapture()          
        capture.apply_on_packets(ProcessPackets, timeout=timeout)
    except asyncio.TimeoutError:
        pass
    except ValueError:
        capture = pyshark.FileCapture(timeOrPath)
        capture.load_packets()
        capture.apply_on_packets(ProcessPackets)
    data = pd.DataFrame(packet_list, columns=['vIP', 'protocol', 'length','timestamp']) 
    print(data['timestamp'].iloc[-1]-data['timestamp'].iloc[0])

def main(): 
    Capture(6)

if __name__ == '__main__':
    main()

Sometimes the calculated time exceeds the timeout given. (timestamp is packet.sniff_time)

Answer 1

UPDATED 06-03-2021

---

After doing some research into this capture latency issue, I have determined that the problem likely is linked to pyshark waiting for dumpcap to load. dumpcap is loaded in LiveCapture mode

  def _get_dumpcap_parameters(self):
        # Don't report packet counts.
        params = ["-q"]
        if self._get_tshark_version() < LooseVersion("2.5.0"):
            # Tshark versions older than 2.5 don't support pcapng. This flag forces dumpcap to output pcap.
            params += ["-P"]
        if self.bpf_filter:
            params += ["-f", self.bpf_filter]
        if self.monitor_mode:
            params += ["-I"]
        for interface in self.interfaces:
            params += ["-i", interface]
        # Write to STDOUT
        params += ["-w", "-"]
        return params

    async def _get_tshark_process(self, packet_count=None, stdin=None):
        read, write = os.pipe()

        dumpcap_params = [get_process_path(process_name="dumpcap", tshark_path=self.tshark_path)] + self._get_dumpcap_parameters()

        self._log.debug("Creating Dumpcap subprocess with parameters: %s" % " ".join(dumpcap_params))
        dumpcap_process = await asyncio.create_subprocess_exec(*dumpcap_params, stdout=write,
                                                               stderr=self._stderr_output())
        self._created_new_process(dumpcap_params, dumpcap_process, process_name="Dumpcap")

        tshark = await super(LiveCapture, self)._get_tshark_process(packet_count=packet_count, stdin=read)
        return tshark

The code above launches this on my system:

 /usr/local/bin/dumpcap -q -i en0 -w -

and this:

/usr/local/bin/tshark -l -n -T pdml -r -

I have attempted to pass in some custom parameters to LiveCapture

capture = pyshark.LiveCapture(interface='en0', custom_parameters=["-q", "--no-promiscuous-mode", "-l"])

but there is still around a 1/2 of a second delay.

10.015577793121338
0 days 00:00:09.371264

In the dumpcap documentation there is a -a mode, which allows for a duration timeout, but I cannot pass that parameter into pyshark without causing an error.

Tshark also has a -a mode, but it also causes an error within pyshark

capture = pyshark.LiveCapture(interface='en0', override_prefs={'': '-r'}, custom_parameters={'': '-a duration:20'})

There might be way to modify the timeout parameters within pyshark code base, to allow the -a mode. To do this would require some testing, which I don't have the time to do at the moment.

I opened an issue on this problem with the developers of pyshark.

ORIGINAL POST 06-02-2021

---

I reworked your code to write the extracted items to a pandas dataframe. If this isn't what you wanted please update your questions with your exact requirements.

import pyshark
import asyncio
import pandas as pd

packet_list = []


def process_packets(packet):
    global packet_list
    try:
        packet_version = packet.layers[1].version
        layer_name = packet.layers[2].layer_name
        packet_list.append([packet_version, layer_name, packet.length, str(packet.sniff_time)])
    except AttributeError:
        pass


def capture_packets(timeout):
    capture = pyshark.LiveCapture(interface='en0')
    try:
        capture.apply_on_packets(process_packets, timeout=timeout)
    except asyncio.TimeoutError:
        pass
    finally:
        return packet_list


def main():
    capture_packets(6)
    df = pd.DataFrame(packet_list, columns=['packet version', 'layer type', 'length', 'capture time'])
    print(df)
    # output 
          packet version layer type length                capture time
    0                 4        udp     75  2021-06-02 16:22:36.463805
    1                 4        udp     67  2021-06-02 16:22:36.517076
    2                 4        udp   1388  2021-06-02 16:22:36.706240
    3                 4        udp   1392  2021-06-02 16:22:36.706245
    4                 4        udp   1392  2021-06-02 16:22:36.706246
    truncated...


if __name__ == '__main__':
    main()