Python CPU Count works on one windows server but not on another?

Question

I have written come code that has worked on both Windows XP and Windows Server 2008 64-Bit. However I have just launched an Amazon Windows 64-bit instance and the code fails.

Very simple it looks like this

import multiprocessing

processors = multiprocessing.cpu_count()
print processors

I receive a NotImplementedError which I do not understand and the docs are not that helpful in explaining.

I just do not understand why it would work on one server and not on another with identical installations of Python 2.7

Anyone else run into this problem/error?

For win32 `cpu_count` uses `num = int(os.environ['NUMBER_OF_PROCESSORS'])`. It raises `NotImplementedError` if the environment variable isn't defined. — Eryk Sun, Nov 25 '12 at 08:59

score 4 · Accepted Answer · 2017-01-24T05:11:52.943

4

It might be just the multiprocessing module, try using the psutil module it might work. SO in your case just do:

import psutil
processors = psutil.cpu_count()
print processors
>>> 4

I tried this on the Amazon Windows 64-bit and it works quite well.

edited Jan 24 '17 at 05:11

answered Nov 24 '12 at 19:43

Hi enginefree - that works, another module to install but hopefully it will be more reliable than going via multiprocessing. many thanks – tjmgis Nov 24 '12 at 21:21
Glad it helped, had the same problem with a couple of my servers. – Nov 24 '12 at 22:14
This has been replaced with `psutil.cpu_count()` - https://github.com/giampaolo/psutil/blob/master/HISTORY.rst – Jan 24 '17 at 05:11

Eryk Sun · Answer 2 · 2014-03-17T10:32:25.417

If getting the CPU count is all you need to do with psutil, you could use ctypes instead:

import ctypes
from ctypes import wintypes

class SYSTEM_INFO(ctypes.Structure):
    _fields_ = [
        ('wProcessorArchitecture', wintypes.WORD),
        ('wReserved', wintypes.WORD),
        ('dwPageSize', wintypes.DWORD),
        ('lpMinimumApplicationAddress', wintypes.LPVOID),
        ('lpMaximumApplicationAddress', wintypes.LPVOID),
        ('dwActiveProcessorMask', ctypes.c_size_t),
        ('dwNumberOfProcessors', wintypes.DWORD),
        ('dwProcessorType', wintypes.DWORD),
        ('dwAllocationGranularity', wintypes.DWORD),
        ('wProcessorLevel', wintypes.WORD),
        ('wProcessorRevision', wintypes.WORD),
    ]

GetSystemInfo = ctypes.windll.kernel32.GetSystemInfo
GetSystemInfo.restype = None
GetSystemInfo.argtypes = [ctypes.POINTER(SYSTEM_INFO)]

def cpu_count():
    sysinfo = SYSTEM_INFO()
    GetSystemInfo(sysinfo)
    num = sysinfo.dwNumberOfProcessors
    if num == 0:
        raise NotImplementedError('cannot determine number of cpus')
    return num

Edit:

Here's an alternative to try that might return the same value as the NUMBER_OF_PROCESSORS environment variable. Note that the documentation says to use GetSystemInfo instead, which is what psutil uses. This is also using a native NT API, which is generally discouraged.

import ctypes
from ctypes import wintypes

SystemBasicInformation = 0

class SYSTEM_INFORMATION(ctypes.Structure): pass
PSYSTEM_INFORMATION = ctypes.POINTER(SYSTEM_INFORMATION)

class SYSTEM_BASIC_INFORMATION(SYSTEM_INFORMATION):
    _fields_ = [
        ('Reserved1', wintypes.BYTE * 24),
        ('Reserved2', wintypes.LPVOID * 4),
        ('NumberOfProcessors', ctypes.c_ubyte),
    ]

ntdll = ctypes.windll.ntdll
NtQuerySystemInformation = ntdll.NtQuerySystemInformation
NtQuerySystemInformation.argtypes = [
    wintypes.LONG,       # SystemInformationClass
    PSYSTEM_INFORMATION, # SystemInformation
    wintypes.ULONG,      # SystemInformationLength
    wintypes.PULONG]     # ReturnLength

def cpu_count():
    info = SYSTEM_BASIC_INFORMATION()
    retlen = wintypes.ULONG()
    status = NtQuerySystemInformation(SystemBasicInformation,
                                      info, 
                                      ctypes.sizeof(info), 
                                      retlen)
    num = info.NumberOfProcessors
    if status < 0 or num == 0:
        raise NotImplementedError('cannot determine number of cpus')
    return num

@BurhanKhalid, I'd use it as a fallback if `sys.platform == 'win32'` and `multiprocessing.cpu_count` raises `NotImplementedError`. — Eryk Sun, Nov 25 '12 at 10:03
When I run this on my Windows box I get a CPU count of 9. When I run `multiprocessing.cpu_count()` (or `os.getenv("NUMBER_OF_PROCESSORS")`) I get a CPU count of 8. Can anyone explain this? — Yani, Mar 17 '14 at 06:59
You can also check using WMI: `wmic cpu get NumberOfCores, NumberOfLogicalProcessors`. — Eryk Sun, Mar 17 '14 at 10:23

score 1 · Answer 3 · edited May 23 '17 at 12:33

The failure is due to multiprocessing.cpu_count() relying on the NUMBER_OF_PROCESSORS environment variable on Windows, which can be occasionally be missing. Using wmi works as a substitute. Thanks for suggestion eryksun.

if sys.platform == 'win32':
    import wmi
    c = wmi.WMI(find_classes=False)
    return sum(x.NumberOfLogicalProcessors for x in c.Win32_Processor())
else:
    return multiprocessing.cpu_count()

Python CPU Count works on one windows server but not on another?

3 Answers3

Linked