How to use multi-processing in a Python cocotb test?

Question

I have a working cocotb setup as below. I want to use multi-processing in Python to run iterations in the loop to parallelize the execution.

Current setup:

from cocotb.decorators import coroutine
factory = TestFactory(Run)

@coroutine
def Run(dut,arg1, arg2):
    for i in range(N):
        ret = yield host.run_test(arg1, arg2)

What I tried:

from cocotb.decorators import coroutine
from multiprocessing import Process, Queue, Manager, Value

factory = TestFactory(Run)

@coroutine
def run_thread(arg1, arg2):
    for i in range(N/n_threads):
        ret = yield host.run_test(arg1, arg2)

@coroutine
def Run(dut,arg1, arg2):
    threads = []
    for n in range(0, n_threads):
        threads.append(Process(target=run_thread, args=(arg1, arg2)))
    for n in range(0, n_threads):
        threads[n].start()
    for n in range(0, n_threads):
        threads[n].join()

As shown above, I tried putting all the content of the loop into a function and calling it using multiprocessing.Process. But when I execute it this way, run_test is not getting called properly.

I believe this is because I have a yield command for run_test and run_thread function returns a generator because of that. However, I don't see any way to iterate through the output of run_thread function because I am calling it in a thread.

Is there any way that I can get this working? Any guidance is much appreciated.

PS: I made a slight mistake in the second code in run_thread function. We need to have a for loop there are well. I corrected it now.

I am not familiar with `cocotb`, but I think a better approach would be to use `multiprocessing.Pool` inside `run_thread` itself. That way, you will get the output of all `run_test` iterations in a list, and then you can return a generator based on that to `Run`. — Charchit Agarwal, Aug 11 '22 at 07:34
There was a slight mistake in my second code, I corrected it now. We need to have a for loop in the run_thread function too. Will Pool work in this case? I basically want to divide the number of iteration among the threads and run several for loops in parallel. I believe Pool will start all processes in parallel if used inside run_thread. — sammy17, Aug 11 '22 at 15:27
It's Linux 3.10.0-1160.62.1.el7.x86_64 #1 SMP x86_64 x86_64 x86_64 GNU/Linux — sammy17, Aug 11 '22 at 16:30
Yes, pool should work in this case. Instead of creating those for loops inside `Run`, create a pool and do `pool.starmap(run_thread, args=[(arg1, arg2) for _ in range(n_threads)])`. Not so sure about using generators with multiprocessing though, interested to see how that turns out. — Charchit Agarwal, Aug 13 '22 at 10:21

Bob · Answer 1 · 2023-01-03T17:30:32.413

cocotb operates simulating a single instance of the design, if you are trying to run multiple tests in parallel you have to replicate your design as well. In that case it is better to parallelize in the shell, so that you can start multiple simulator instances.

If you simply want parallelism in the testbench so that each job runs simultaneously with respect to simulation, you should consider concurrent execution.

How to use multi-processing in a Python cocotb test?

1 Answers1