JnetPcap: reading from offline file very slow

Question

I'm building a sort of custom version of wireshark with jnetpcap v1.4r1425. I just want to open offline pcap files and display them in my tableview, which works great except for the speed. The files I open are around 100mb with 700k packages.

public ObservableList<Frame> readOfflineFiles1(int numFrames) {  
    ObservableList<Frame> frameData = FXCollections.observableArrayList();

    if (numFrames == 0){
        numFrames = Pcap.LOOP_INFINITE;
    }
    final StringBuilder errbuf = new StringBuilder();  

    final Pcap pcap = Pcap.openOffline(FileAddress, errbuf);  
    if (pcap == null) {  
        System.err.println(errbuf); // Error is stored in errbuf if any  
        return null;  
    } 


    JPacketHandler<StringBuilder> packetHandler =  new JPacketHandler<StringBuilder>() {
        public void nextPacket(JPacket packet, StringBuilder errbuf) {  


            if (packet.hasHeader(ip)){
                sourceIpRaw = ip.source();
                destinationIpRaw = ip.destination();

                sourceIp = org.jnetpcap.packet.format.FormatUtils.ip(sourceIpRaw);  
                destinationIp = org.jnetpcap.packet.format.FormatUtils.ip(destinationIpRaw);  
            }

            if (packet.hasHeader(tcp)){
                protocol = tcp.getName();
                length = tcp.size();

                int payloadOffset = tcp.getOffset() + tcp.size();  
                int payloadLength = tcp.getPayloadLength();  

                buffer.peer(packet, payloadOffset, payloadLength); // No copies, by native reference  
                info = buffer.toHexdump();
            } else if (packet.hasHeader(udp)){
                protocol = udp.getName();
                length = udp.size();


                int payloadOffset = udp.getOffset() + udp.size();  
                int payloadLength = udp.getPayloadLength();  

                buffer.peer(packet, payloadOffset, payloadLength); // No copies, by native reference  
                info = buffer.toHexdump();
            }

            if (packet.hasHeader(payload)){

                infoRaw = payload.getPayload();
                length = payload.size();

            }


            frameData.add(new Frame(packet.getCaptureHeader().timestampInMillis(), sourceIp, destinationIp, protocol, length, info ));
            //System.out.print(i+"\n");
            //i=i+1;
        }  

    };
    pcap.loop(numFrames, packetHandler , errbuf);  
    pcap.close();


    return frameData;
}

This code is very fast for the first maybe 400k packages, but after that it slows down a lot. It needs around 1 minute for the first 400k packages and around 10 minutes for the rest. What is the issue here?

It's not that the list is getting too timeconsuming to work with is it? the listmethod add is O(1), isnt it?

I asked about this on the official jnetpcap forums too but it's not very active.

edit:

turn out it slows down massively because of the heap usage. Is there a way to reduce this?

Use a profiler (e.g. `jvisualvm`) to see what the hotspot is. — Kayaman, Feb 08 '17 at 09:13

score 2 · Accepted Answer · answered Feb 08 '17 at 09:31

2

As the profiler showed you, you're running low on memory and it starts to slow down.

Either give more memory with -Xmx or don't load all the packets into memory at once.

answered Feb 08 '17 at 09:31

Kayaman

72,141
5
83
121

I've never had to think much about performance issues like this before, so I'm unsure. I still want to have all those packets in my OberservableList in the end, is there no way to do that without increasing heap size? – banzai Feb 08 '17 at 09:37
2

If you want to put a lot of things into a bag, is there a way to do that without getting a big enough bag? No. Unless it's a magic bag. The only alternative solution is to not load them all at once, but if you really *need* them all in the memory at the same time, raise your heap. – Kayaman Feb 08 '17 at 09:38
I see. It just seemed weird to me that a file of 100mb needs more than a gb of memory. Now to find out how much I will need. – banzai Feb 08 '17 at 09:42
1

Well, it'll need a lot less if you keep it in the memory as just bytes, like on disk. However since it's being converted to objects there's obviously overhead. It also makes it a lot easier to work with them. There's nothing weird about it. – Kayaman Feb 08 '17 at 09:44
1

I guess I don't know much about these kind of things yet. Thank you very much. – banzai Feb 08 '17 at 09:47

JnetPcap: reading from offline file very slow

1 Answers1