Progammatic browsing performance issues

Question

I'm using Selenium (in Java) to do programmatic browsing (Firefox) and, to speed up page loading, I'm going through a programmatic Java proxy running in the same application (Browsermob) to prevent the loading of external content (ads, etc) as I only need website's hosted content.

The application is browsing through hundreds of thousands of pages but unfortunately, after a few hours, speed is reducing significantly (from ~5s/page to ~30s/page).

Can it be related to browser cache size? Proxy cache? How to check? Any pointer or hint would be more than welcome!

Many thanks, Tom

please do one think apply the concept of cache deleting may be this helps — Rajnish Kumar, Jun 08 '16 at 11:09
@Tom - Are you running BrowserMob Proxy in the same process as the application? Are you observing any issues with CPU or memory usage over time? — Jason Hoetger, Jun 10 '16 at 22:55

score 2 · Answer 1 · answered Jun 08 '16 at 11:14

Hi I would change your User Agent to something like Chrome to check wether or not Firefox is the problem or if its down to memory management in your app.

Are you closing down the Browser after a number of views? remember the browser will cache and start to take up a lot of memory. It may be better for you to use a text based browser (Lynx) in your app that removes a lot of overheads if your are simply going for text content.

Id kill the process every so often as memory leaks will cause the process to slow over time.

Progammatic browsing performance issues

1 Answers1