2

We have a C++ based Multi-threaded application on Windows that captures network packets in real-time using the WinPCAP library and then processes these packets for monitoring the network. This application is intended to run 24x7. Our applicatin easily consumes 7-8 GB of RAM.

Issue that we are observing : Lets say the application is monitoring 100Mbps of network traffic and consumes 60% CPU. We have observed that when the application keeps running for a longer duration like a day or two, the CPU consumption of the application increases to like 70-80%, even though it is still processing 100 Mbps traffic (doing the same amount of work).

We have tried to debug this issue to the thread level using ProcessExplorer and noticed that the packet capturing threads start consuming more CPU over time. This issue is not resolved even after re-starting the application. Only a machine restart solves the problem.

We have observed this issue is easily reproducible on Windows 2012 R2 Server OS during over night runs. In Windows 7, the issue happens but over few days.

Any idea what might be causing this ?

Thanks in Advance

Ramandeep Sandhu
  • 21
  • 3
  • No idea. Check out [xperf](http://msdn.microsoft.com/en-us/library/windows/hardware/hh162945.aspx). [Random thinks it's great](https://randomascii.wordpress.com/category/xperf/). – Martin Ba Oct 22 '14 at 08:52
  • Or [PerfView](http://blogs.msdn.com/b/dotnet/archive/2012/10/09/improving-your-app-s-performance-with-perfview.aspx). – Joey Oct 22 '14 at 09:25
  • Restarting the app doesn't help, so probably not a memory leak. How about system resource leaks: look for handle count growing over time. – Chris O Oct 22 '14 at 12:39
  • Could be a bug in the WinPcap driver, or perhaps even in the network drivers. – Harry Johnston Oct 22 '14 at 22:09
  • Chris I will try and keep a watch on the handle count as well – Ramandeep Sandhu Oct 23 '14 at 09:09

2 Answers2

2

What about memory allocation? Because you are using lots of memory it could be a memory fregmentation problem so if you do several allocation/reallocation of buffers this of course will cause a major cost for the processor to find and allocate space available.

IFeelGood
  • 397
  • 1
  • 12
  • Well we run the application on systems having sufficient RAM something like 16GB or more. We are using memory pools; although there are frequent allocations/de-allocations happening. If this was the case, shouldn't this be solved when we restart the application ? – Ramandeep Sandhu Oct 22 '14 at 09:47
  • The memory fragmentation may happens at a OS level, so I don't think there's something you can do at application level more than what you have already done. Eventually you should allocate one single huge memory segment and have your own memory manager... – IFeelGood Oct 22 '14 at 10:02
  • Thanks IFeelGood.. I will try and explore the Memory Fragmentation part. Is there a way I can defragment the RAM after stopping my application and re-run it just to verify if it was caused due to memory fragmentation. A machine reboot would definitely de-fragment the RAM. I want to try it without rebooting the machine. – Ramandeep Sandhu Oct 22 '14 at 10:12
  • As far as I know that's not possible. Eventually check for "hard page fault" and try to disable paged memory. My machine has 16Gb and I live without it. I would like to give it a try. – IFeelGood Oct 22 '14 at 12:16
  • I checked the "hard faults/sec" and these were 0. I still need to check with disabling the paged memory. I was thinking If there could be a simpler app to test this phenomenon. – Ramandeep Sandhu Oct 27 '14 at 12:12
0

I finally found the reason for the above behavior : it was the winpcap code that was causing it. After replacing that, we did not observe this behavior.

Ramandeep Sandhu
  • 21
  • 3