Loading pcap file in spark using python

Question

I am new to Spark. I have pcap file. How can Spark read that file using python? How can I upload pcap file in Spark using python and how can it process?

conf = SparkConf().setMaster("local").setAppName("SparkStreamingPcap")
sc = SparkContext(conf = conf)
sc.setLogLevel("ERROR")
spark = SparkSession(sc)
FileLog = sc.textFile("pcapFiles/ipv4frags.pcap")
df = FileLog.map(lambda line: line.split("\n"))
print("Helloo")
print (df.count())

1. does the file exist in that location? 2. what is your expected output and what do you get now? any error? — Radan, Sep 13 '18 at 18:28
I want to process pcap file. i want to get timeStamp and Ips. I am not getting any output — Zainub Wahid, Sep 13 '18 at 18:48

score 1 · Answer 1 · answered Sep 26 '18 at 08:54

1

You could also try using dpkt or scapy to parse pcap files in pyspark code.

answered Sep 26 '18 at 08:54

Vijay Anand Pandian

1,027
11
23

True: indeed, you can’t read a pcap as text but as binary with its own format. You’ll have to implement magic handling and lots of stuff if not using an already existing API – Cukic0d Sep 28 '18 at 15:51

Loading pcap file in spark using python

1 Answers1