5

I've a situation like this:

The main process generate some sub-process that they should write the result in a shared object in string and numeric types, for the numeric types there's no problem but with the string the value will be lost.

import multiprocessing as mp
from ctypes import Structure, c_double, c_wchar_p, c_int

# shared obj class
class SharedObj(Structure):
    _fields_ = [('name', c_wchar_p), ('val', c_double) ]

def run_mp( values , lock , s ) :
    for i in range( s , len( values ) , 2 ):
        lock.acquire()
        values[i].name = str( i ) # write the string value in the shared obj
        values[i].val = float( i )
        print( "tmp: %d" % i )
        lock.release()

def main():
    # creating the shared obj and mutex
    values = mp.Array(  SharedObj , [SharedObj() for i in range( 10 )] )
    lock_j = mp.Lock()

    # creating two sub-process form the function run_mp
    p1 = mp.Process( target=run_mp , args=( values , lock_j , 0 ))
    p2 = mp.Process( target=run_mp , args=( values , lock_j , 1 ))

    p1.start()
    p2.start()

    p1.join()
    p2.join()

    for v in  values:
        print()
        print( "res name: %s" % v.name )
        print( "res val: %f " % v.val )


if __name__ == '__main__':
    main()

As a result the field in the shared object containing the c_double is written in the field, but the string generated in the sub-processes rum-mp ( string values[i].name = str( i ) ) will be lost in the main process.

There is a method for saving the strings generated in sub-process?

The output of this code looks like:

Where the resulting string in the main process are completely random.

tmp: 0
tmp: 2
tmp: 3
tmp: 4

res name: ����羍����羍
res val: 0.000000
res name: ����羍����羍
res val: 1.000000
res name:
res val: 2.000000   ....
martineau
  • 119,623
  • 25
  • 170
  • 301
Giggi
  • 681
  • 2
  • 9
  • 17
  • 2
    c_wchar_p - is a pointer type. I bet you a passing pointer from one process to another. And the parent reads data at this address. – SanityIO Sep 09 '12 at 21:08
  • Ok c_wchar_p is a pointer, but how I write my string in the shared object, without losing it? – Giggi Sep 09 '12 at 21:16
  • @Giggi, as I understand, it is not possible to properly use `c_char_p` or `c_wchar_p` with `multiprocessing`. Refer to this warning in the `multiprocessing` module: "Although it is possible to store a pointer in shared memory remember that this will refer to a location in the address space of a specific process. However, the pointer is quite likely to be invalid in the context of a second process and trying to dereference the pointer from the second process may cause a crash." – Asclepius Nov 08 '12 at 04:57

2 Answers2

3

Here is a slightly modified version of your code:

#!/usr/bin/env python

import multiprocessing as mp


def run_mp( values ):
    for c_arr, c_double in values:
        c_arr.value = 'hello foo'
        c_double.value = 3.14

def main():
    lock = mp.Lock()
    child_feed = []
    for i in range(10):
        child_feed.append((
            mp.Array('c', 15, lock = lock),
            mp.Value('d', 1.0/3.0, lock = lock)
        ))

    p1 = mp.Process( target=run_mp , args=(child_feed,))
    p2 = mp.Process( target=run_mp , args=(child_feed,))

    p1.start()
    p2.start()

    p1.join()
    p2.join()

    for c_arr, c_double in child_feed:
        print()
        print( "res name: %s" % c_arr.value )
        print( "res val: %f " % c_double.value )


if __name__ == '__main__':
    main()

Take a look at http://docs.python.org/library/multiprocessing.html There is an example of using Array of chars.

There is also mmap module alowing to share memory http://docs.python.org/library/mmap.html but with this you have to Sync access yourself possibly by semaphores. If you like more simple approach just use pipes.

SanityIO
  • 814
  • 6
  • 15
  • This code works with python 2, but with python 3 I've this error: ** mp.Array('c','hello world', lock = lock), ...... TypeError: one character string expected ** – Giggi Sep 09 '12 at 21:31
  • Traceback (most recent ..): File "ct1.py", line 38, in main() File "ct1.py", line 18, in main mp.Array('c','hello world', lock = lock), File "/usr/lib/python3.2/multiprocessing/__init__.py", line 259, in Array return Array(typecode_or_type, size_or_initializer, **kwds) File "/usr/lib/python3.2/multiprocessing/sharedctypes.py", line 112, in Array obj = RawArray(typecode_or_type, size_or_initializer) File "/usr/lib/python3.2/multiprocessing/sharedctypes.py", line 89, in RawArray result.__init__(*size_or_initializer) TypeError: one character string expected – Giggi Sep 09 '12 at 21:36
  • Not yet: **res name: b''** but now I've no errors. And it lost also the numeric value – Giggi Sep 09 '12 at 21:49
  • There is no point to guess for me why this dont wokr in 3.X blindly. Play around with arguments to mp.Array(ctypes.c_char,15) and docs also says there is .raw attribute on array setting c_arr.raw = 'hello foo' also workd for me. What exactly are you trying to do? A dict in shared memory? If so i think i have one already implemented in c + cython. – SanityIO Sep 09 '12 at 22:03
  • It seems to be a problem with python3, and also the example above dont works [example from py3k docs](http://docs.python.org/py3k/library/multiprocessing.html?highlight=multiprocessing#module-multiprocessing.sharedctypes) – Giggi Sep 10 '12 at 09:40
  • Giggi: This will work in Python3 if you change to `c_arr.value = 'hello foo'.encode('latin1')` in `run_mp()` and the `print( "res name: %s" % c_arr.value.decode('latin1'))` in `main()`. This is because in Python3 strings are `utf-8` Unicode by default. The changes I suggested change the encoding to `latin1` temporarily and convert it back to the `utf-8` default before printing. I chose `latin-1` because all 256 possible byte values will be preserved. Just make sure the size of the `mp.Array()` is enough for the longest string expected. – martineau Aug 11 '17 at 17:39
1

I didn't want to use multiprocessing.Array because of the obvious requirement of needing to specify its size ahead of time. The following works for me instead for having a multiprocessing-compatible unicode object. It is tested with Python 2.6 with multiple processes.

>>> shared_str = multiprocessing.Manager().Value(unicode, 'some initial value')
>>> shared_str.value
'some initial value'
>>> shared_str.value = 'some new value'
>>> shared_str.value
'some new value'

To deal with the author's specific question about sharing a string and a number, a serializable object to store these can perhaps be created and given to Value instead.

You may of course object to needing to use Manager for this purpose. If so, please provide an alternate solution.

Asclepius
  • 57,944
  • 17
  • 167
  • 143