I know I'm late to the party, but hopefully this will be useful to future visitors.
rdpcap() dissects the entire pcap file and retains an in-memory representation of each and every packet, which explains why it eats up a lot of memory.
As far as I am aware (I am a novice Scapy user myself), the only two ways you can invoke Scapy's session reassembly are:
- By calling
scapy.plist.PacketList.sessions(). This is what you're currently doing (rdpcap(pcap_file) returns a scapy.plist.PacketList).
- By reading the pcap using
sniff() in offline mode while also providing the function with a session decoder implementation. For example, for TCP reassembly you'd do sniff(offline='stackoverflow.pcap', session=TCPSession). (This was added in Scapy 2.4.3).
Option 1 is obviously a dead end (as it requires that we keep all packets of all sessions in memory at one time), so let's explore option 2...
Let's launch Scapy in interactive mode to access the documentation for sniff():
$ scapy
>>> help(sniff)
Help on function sniff in module scapy.sendrecv:
sniff(*args, **kwargs)
Sniff packets and return a list of packets.
Args:
count: number of packets to capture. 0 means infinity.
store: whether to store sniffed packets or discard them
prn: function to apply to each packet. If something is returned, it
is displayed.
--Ex: prn = lambda x: x.summary()
session: a session = a flow decoder used to handle stream of packets.
e.g: IPSession (to defragment on-the-flow) or NetflowSession
filter: BPF filter to apply.
lfilter: Python function applied to each packet to determine if
further action may be done.
--Ex: lfilter = lambda x: x.haslayer(Padding)
offline: PCAP file (or list of PCAP files) to read packets from,
instead of sniffing them
timeout: stop sniffing after a given time (default: None).
L2socket: use the provided L2socket (default: use conf.L2listen).
opened_socket: provide an object (or a list of objects) ready to use
.recv() on.
stop_filter: Python function applied to each packet to determine if
we have to stop the capture after this packet.
--Ex: stop_filter = lambda x: x.haslayer(TCP)
iface: interface or list of interfaces (default: None for sniffing
on all interfaces).
monitor: use monitor mode. May not be available on all OS
started_callback: called as soon as the sniffer starts sniffing
(default: None).
The iface, offline and opened_socket parameters can be either an
element, a list of elements, or a dict object mapping an element to a
label (see examples below).
Notice the store parameter. We can set this to False to make sniff() operate in a streamed fashion (read a single packet, process it, then release it from memory):
sniff(offline='stackoverflow.pcap', session=TCPSession, store=False)
I just tested this with a 193 MB pcap. For store=True (default value), this eats up about 1.7 GB of memory on my system (macOS), but only approximately 47 MB when store=False.
Processing the reassembled TCP sessions (open question)
So we managed to reduce our memory footprint - great! But how do we process the (supposedly) reassembled TCP sessions? The usage instructions indicates that we should use the prn parameter of sniff() to specify a callback function that will then be invoked with the reassembled TCP session (emphasis mine):
sniff() also provides Sessions, that allows to dissect a flow of
packets seamlessly. For instance, you may want your sniff(prn=...)
function to automatically defragment IP packets, before executing the
prn.
The example is in the context of IP fragmentation, but I'd expect the TCP analog to be to group all packets of a session and then invoke prn once for each session. Unfortunately, that's not how it works: I tried this on my example pcap, and the callback is invoked once for every packet---exactly as indicated in sniff()'s documentation shown above.
The usage instructions linked above also states the following about using session=TCPSession in sniff():
TCPSession -> defragment certain TCP protocols*. Only HTTP 1.0 currently uses this functionality.
With the output of the experiment above in mind, I now interpret this as that whenever Scapy finds an HTTP (1.0) request/response that spans across multiple TCP segments, it'll create a single packet in which the payload is the merged payload of those TCP segments (which in total is the full HTTP request/response). I'd appreciate it if anyone can help clarify the meaning of the above quote on TCPSession---or even better: clarify if TCP reassembly is indeed possible this way and that I'm just misunderstanding the API.
