0

I am running multiple data validation test on selenium grid (only chrome browsers) on CentOS stack. I notice that initially the tests complete real quick. However, with time, the execution slows down considerably. I am trying to validate data from a csv file with data on a web application. I have around 100K records in the csv file. For each record, below are the list of event:

  • launch remote driver(chrome) instance
  • Open the web application and login
  • search for the keywords in the csv file on the application and validate the results(output in csv VS output on the web application)
  • close the remote driver instance

I have configured 7 nodes using CentOS and each node has 10 browser instances.

Also, I am using ThreadPoolExecutor for submitted each thread. So at any given time, I will have 70 threads running where each thread is a webdriver instance.

I am not sure if this is a code level issue or infrastructure related issue. Can someone point me in the right direction of how I can find the root cause for this slowness and rectify it.

I have tried to monitor system resources for one of the nodes and see that the java process takes around 55% CPU and 10% memory. while each browser takes 10% CPU and 4% memory

BountyHunter
  • 1,413
  • 21
  • 35
  • Can you consider showing us your work and update if the **slowness** is visible slowness or you have some data? Thanks – undetected Selenium Jun 01 '17 at 11:17
  • @DebanjanB - updated the question – BountyHunter Jun 01 '17 at 11:39
  • have you tried restarting the nodes on regular basis? e.g once a day. I run into performance issues aswell after a few days non-stop running.. the daily restart solved them ;) – metar Jun 01 '17 at 11:53
  • @metar Yes, nodes are restarted regularly.(However, not in the middle of execution). So if an execution batch of 100K takes 24-32 hours, nodes will restart only after the batch completes – BountyHunter Jun 01 '17 at 11:56
  • @BountyHunter how long runs your driver (chrome) in average? – metar Jun 01 '17 at 12:09
  • @metar Each browser instance runs for around 45 seconds initially. However, with time this number increases to 2 minutes(sometimes even more) – BountyHunter Jun 01 '17 at 12:12
  • so you start your driver 100k times to run ~1min? hum.. is it possible to process for example 500 records (lines) with the same driver? The process would probably run faster.. I only use firefox driver but the creation of a new instances takes a few seconds and produces a higher load – metar Jun 01 '17 at 12:17
  • Error handling becomes a bit of an issue in this case. With that said, I have tried this approach and tried to split 10 records in each instance. However, the slowness starts to show after processing probably 9-10K records. – BountyHunter Jun 01 '17 at 12:20
  • @BountyHunter, is it possible to switch to another webdriver? FirefoxDriver, HTMLUnitDriver or phantomjs and see their performance – metar Jun 01 '17 at 13:10
  • I was initially working with Firefox. However I switched to chrome because the performance was better – BountyHunter Jun 01 '17 at 13:24

1 Answers1

1

Selenium grid will be slow when time increases as selenium grid is running on jvm and it will occupy more memory. There are many factors will affect the performance of the browser like no of browser in a node, node configuration, grid configuration and you web server performance. For better grid performance, you have to restart grid hub and nodes once in a while.

Murthi
  • 5,299
  • 1
  • 10
  • 15