0

I have received real time packet using jnetpcap. Can anyone please tell me how to extract the packet header and other details from a live packet using spark streaming?

Niels Bech Nielsen
  • 4,777
  • 1
  • 21
  • 44
user3823859
  • 469
  • 1
  • 7
  • 20
  • I have the following line in my code. val lines = KafkaUtils.createStream(ssc, zkQuorum, group, topicpMap).map(_._2) How to deserialize this stream "lines" into original object? Serialisability was implemented in the kafka producer by extending class to serialisable – user3823859 Sep 30 '14 at 16:38

1 Answers1

1

If you have read through spark streaming programming guide, you will have discovered that you can implement a custom Receiver for your stream. However, there are little examples in the actual guide, and you will have to read the examples and javadoc specified on the bottom of the page.

This project is a custom receiver, which basically:

  1. Extends Receiver
  2. Implements onStart and onStop methods
  3. Calls store method with the data to pass through the stream

It should be as easy as that. Which parts of the package to parse and how is probably your own decision. You just feed the package in at source level and write your own transformers etc, according to the programming guide.

Niels Bech Nielsen
  • 4,777
  • 1
  • 21
  • 44
  • I have received packets in receiver using jnetpcap library. But how to extract packet details using RDD operations on each packet? – user3823859 Sep 25 '14 at 09:18
  • That is the generic stream processing thing. Your receiver just stores the raw data if possible, then a transformer receives data in the raw format and transforms it to whatever step you need next. Is this question on how to write the Receiver part or the structure of jnetpcap packages? – Niels Bech Nielsen Sep 25 '14 at 09:46
  • The question is on how to transform the data in raw format ? – user3823859 Sep 29 '14 at 04:20
  • Dont know anything about that, but there seems to be reasonable well written guides and tutorials here : http://jnetpcap.com/book/export/html/ – Niels Bech Nielsen Sep 29 '14 at 08:06
  • Thanks. I just wanted to know how to deserialize streams received in spark streaming receiver. – user3823859 Sep 29 '14 at 09:21