1

I am using pytest-xdist to parallelize my tests. The machine I am working on has 12 cores. Half of my tests use 2 processes in each test, while the other half of the tests only use one process (normal single processing).

Currently I add the addopts = -n6 to my pytest.init file, since some tests actually require two processes (6 * 2 = 12, or all the available cores on my machine). I was wondering if it's possible to specify the amount of cores each test uses?

For example, all tests that use 2 processes I would mark it that way, and pytest-xdist would ideally know to take up 2 processes from it's pool of workers for those tests. Then I can change my pytest.ini to include addopts = -n12? I could not figure out how to do it from reading the documentation, maybe it's not possible.

Thanks so much!

Justin Furuness
  • 685
  • 8
  • 21
  • 1
    I'm not aware of, nor was able to find any mechanism to inform pytest that fewer processes should be started due to an individual test utilizing more than one core. As long as you are not starting hundreds of processes per core, a few too many processes just adds a little overhead. This overhead is probably smaller than the penalty of not using enough cores on accident in terms of completing all the tasks in a given amount of time. – Aaron Oct 26 '21 at 00:50
  • 1
    Thank you for your response. For my particular use case, setting pytest to use all available cores with -n12 causes a slow down compared to -n6, which is why I wanted to know if it was possible to assign multiple cores to a single test. – Justin Furuness Oct 27 '21 at 09:50
  • Did you figure this out? I would *guess* that your computer has 6 actual cores and 6 more due to simulation with hyperthreading. I think -n6 and -n12 would be about the same in that case, since the number of actual cores is just 6. My question is what improvement do you see if you try -n3, or -n4, for example? – pauljohn32 Sep 14 '22 at 04:43
  • My understanding is that it's not possible to specify A cores for some tests and B cores for others, so no I did not figure this out – Justin Furuness Sep 15 '22 at 20:11

0 Answers0