0

I'm looking to build a test with 4000 browser sessions using a Locust/Selenium approach. I tried using the same approach using JMeter/Selenium using headless browsers and for many reasons I've come to the conclusion that it's not possible. I'm familiar with running Browser-based load testing tools and with TrueClient I can reach around 45 Browsers/users per/Slave/agent (loaded servers, 64RAM and 16CPU's). Silk Performer BDLT suggests around 15 or 20 sessions. I've looked at 'realbrowserlocusts' and it's really peaked my interest because I used locust as part of some Covid-19 application testing I ran back in February. Are there any metrics of how many browsers a slave can handle using selenium and any metrics of how many slaves Locust can handle? Should I be looking at PhantomJS instead? My org is looking to do away with all commercial tools and going to opensource and need to add a browser-based load testing solution. JMeter/Gatling/Locust work great for the protocol level applications Any suggestions would be great.

Michael Johnson
  • 95
  • 1
  • 7

1 Answers1

0

The only real answer is "it depends".

I've been able to run some 30 clients & a total of maybe 2-3 clicks/s on a moderately advanced site (plenty of javascript, but nothing crazy) on 2018 macbook pro (6-core i7, 16GB RAM). The problem is that you can't really know before hand how much resources you will need, and you may start to get intermittent errors as you run out of resources.

When running Selenium, the browser will be the heavy part so expect similar performance with Locust+Selenium as you would get with Jmeter+Selenium.

PhantomJS is in most cases slower than headless Chrome so I wouldnt bother with that.

Cyberwiz
  • 11,027
  • 3
  • 20
  • 40
  • I'm well on my way with this effort. I've run 25 clients in full browser mode with 6 actions each (1 iteration; 25 users, 30 second hatch rate) running on my laptop. Only 3 users failed and the cpu only hit 50%. I still have a lot of work to do on the selenium side with error checking. On the Locust side the newer versions aren't compatible with realbrowserlocusts and I've run into major issues running in headlessmode (the users don't get passed the first click for some reason). – Michael Johnson Jan 05 '21 at 16:37
  • Check out locust-plugins for an alternative implementation (it doesnt track response times, but should work better with later locust versions) https://github.com/SvenskaSpel/locust-plugins/blob/master/examples/webdriver.py – Cyberwiz Jan 06 '21 at 17:43
  • 1
    Since my last post we were able to run a 150 clients, in headless mode without any issues. Server: Xeon(r) CPU E5-2660 v4 @ 2.60. If I slowed the rampup and let the test smooth out I could've ran it at 175 clients. So far tripling the amount of processes of a commercial tool. My only issue is error reporting. client.wait.until( EC.visibility_of_element_located ... doesn't post any errors, but rather restarts the execution over of the user. Any thoughts. – Michael Johnson Jan 15 '21 at 05:53
  • Cool! I think wait.until throws an exception, so you need to catch it, and log a failed sample (you still probably want to do a return to avoid continuing in an unknown state) – Cyberwiz Jan 15 '21 at 12:08
  • Is there a sample script that does the catch? I've tried everything I can think of and it appears the wait.until doesn't throw anything. – Michael Johnson Jan 15 '21 at 16:55
  • I dont have a sample script but have a look at https://github.com/SvenskaSpel/locust-plugins/blob/master/examples/webdriver.py (it doesnt use realbrowserlocusts but it is similar) – Cyberwiz Jan 15 '21 at 18:39
  • I tried that one and it didn't work with realbrowserlocusts unfortunately. see: https://stackoverflow.com/questions/65741178/exception-handling-with-realbrowserlocusts – Michael Johnson Jan 15 '21 at 19:36