We have recently created a job with cucumber selenium grid on aws, which will scale pods on fly up to 100. Cucumber has inbuilt parallel feature for testNg in latest version.
If we have 100 nodes and 150 tests(In one job) then 100 tests in that job are running in parallel and 50 are in queue once its done 50 are running parallel later. This looks good.
However if we have 2 jobs(or multiple 50-100 jobs). One job has 150 tests and other job has 100 tests then at this point its competing resources and seeing browser start up failures.
Implementing apache commons pool(queue) can resolve this? If so how can I queue scenarios/tests/threads(from different runners), is there any way I get requests waiting on selenium hub and assign based on resources?