0
import kafka.serializer.Encoder
import kafka.utils.VerifiableProperties
import org.jnetpcap.packet.PcapPacket
class PcapEncoder(verifiableProperties: VerifiableProperties) extends Encoder[PcapPacket]  { override def toBytes(customMessage: PcapPacket): Array[Byte] = customMessage.transferStateAndDataFrom(PcapPacket : Array[Byte]) }

`This is the encoder that I have written to encode packets captured using jnetpcap library to pass it to kafka consumer. But I have errors, is this the way to implement encoder ?

user3823859
  • 469
  • 1
  • 7
  • 20

1 Answers1

1

Assuming that you have Kafka already installed and running, you would establish a connection to Kafka to a particular topic using one of the producer interfaces (perhaps kafka.javaapi.producer.Producer). Then when you get captured packet, you pass it to the API. Kafka has not problems with raw bytes. When you are done, you close the connection via that API.

Jim Hoagland
  • 481
  • 1
  • 4
  • 20
  • Thank you. Is it enough to write a kafka producer that captures packets using jnetpcap library and kafka consumer that receives the packets? Also, I need to do this in Intellij Idea. – user3823859 Aug 24 '14 at 14:01
  • You should be able to do that in Intellij. The setup you describes sounds reasonable but I don't know your requirements to say more – Jim Hoagland Aug 24 '14 at 14:53
  • This is the code that I have written for capturing packets. How to implement this as a kafka producer ? https://github.com/swe0523/PacketCapture-in-Spark/blob/master/src/main/scala/PacketCapture1.scala – user3823859 Aug 25 '14 at 13:39
  • At the start of main() code, you would create a kafka.javaapi.producer.Producer instance, selecting the options you want. – Jim Hoagland Aug 26 '14 at 04:29
  • (more complete) At the start of main() code, you would create a kafka.javaapi.producer.Producer instance, selecting the options you want and it will connect to Kafka. At the end of the main() you would close the Producer. In nextPacket() you would call send on the producer, including the PcapPacket. – Jim Hoagland Aug 26 '14 at 04:37
  • I have pushed the packets to kafka and kafka receiver is able to receive the packets as string only. How to convert the received string format into pcap format so as to utilise the functionalities of jnetpcap library in receiver ? – user3823859 Oct 30 '14 at 08:54