1

I am performing some pandas and NumPy operation, in windows I can see all cores of CPU are running but in Linux environment, everything is running on 1 core.

I want to engage all core so perform the operation fast.

e.g.,

def a():
    #some operations
    .....

I want to execute this function only once but using all cores. Because when it is using only one core it takes around 6 hours in Linux, although it is taking only 2 hours in the window.

I have seen in windows numexpr is setting default cores to 4.

Is there any way I can do the same thing in Linux from code?

Adam
  • 2,820
  • 1
  • 13
  • 33
Naveen Rishishwar
  • 36
  • 5

1 Answers1

0

use concurrent.future module for paralel execution. it will get the work for you:

The concurrent.futures module provides a high-level interface for asynchronously executing callables.

The asynchronous execution can be performed with threads, using ThreadPoolExecutor, or separate processes, using ProcessPoolExecutor. Both implement the same interface, which is defined by the abstract Executor class.

import concurrent.futures
with ThreadPoolExecutor(max_workers=psutil.cpu_count()) as executor:
    future = executor.submit(a(), args)
    print(future.result())

submit(fn, *args, **kwargs) Schedules the callable, fn, to be executed as fn(*args **kwargs) and returns a Future object representing the execution of the callable.

use the psutil.cpu_count() to get the logical cores number for dispatching maximum workers and utilizing cpu cores.

Adam
  • 2,820
  • 1
  • 13
  • 33