How to copy list of tuples to EXISTING numpy array in shared memory

Question

I have a structured numpy array in shared memory, that's only one "layer" of a higher dimensional array.

And I have a list of tuples whose values I want to copy to this (sub) array.

I've found how to make a new numpy structured array out of a list of tuples. But I can't find out how to convert this list of tuples to an EXISTING numpy (sub) array. The sizes already match, of course.

Of course I can copy elementwise in a Python for-loop, but this seems awfully inefficient. I'd like the looping to be done in the C++ that underlies numpy.

Explanation: The reason my array is in shared memory is that I use this as a common datatructure with a C++ process, guarded by mutex semaphores.

My list of tuples looks like:

[(25141156064, 5.3647, 221.32287846), (25141157138, 5.3647, 73.70348602), (25141155120, 5.3646, 27.77147382), (25141160388, 5.3643, 55.5000024), (25141160943, 5.3636, 166.49511561), (25141154452, 5.3578, 92), (25141154824, 5.3539, 37.22246003), (25141155187, 5.3504, 37.22246003), (25141157611, 5.34, 915), (25141157598, 5.3329, 1047.32982582), (25140831246, 5.3053, 915), (25141165780, 5.2915, 2000), (25141165781, 5.2512, 2000), (25140818946, 5.2483, 915), (25138992274, 5.1688, 458), (25121724934, 5.1542, 458), (25121034787, 4.8993, 3.47518861), (24402133353, 2.35, 341), (24859679064, 0.8, 1931.25), (24046377720, 0.5, 100), (25141166091, 5.3783, -650.51242432), (25141165779, 5.3784, -1794.28608778), (25141157632, 5.3814, -2000), (25141157601, 5.3836, -2000), (25141164181, 5.3846, -499.65636506), (25141164476, 5.4025, -91), (25141157766, 5.4026, -634.80061236), (25141153364, 5.4034, -2000), (25141107806, 5.4035, -1601.88882309), (25141157694, 5.4136, -1047.32982582), (25141148874, 5.4278, -266), (25141078136, 5.4279, -48.4864096), (25141165317, 5.4283, -2000), (25141097109, 5.4284, -914), (25141110492, 5.4344, -774.75614589), (25141110970, 5.4502, -928.32048159), (25141166045, 5.4527, -2000), (25141166041, 5.493, -2000), (25139832350, 5.5, -10.2273)]

My numpy array has elements that are defined as follows:

Id = np.uint64
Price = np.float64
Amount = np.float64

Quotation = np.dtype ([
    ('id', Id),
    ('price', Price),
    ('amount', Amount),
])

self._contents = np.ndarray (
    shape = (
        maxNrOfMarkets,
        maxNrOfItemKindsPerMarket,
        maxNrOfQuotationsPerItemKind
    )

    dtype = Quotation,
    buffer = self.sharedMemory.buf,
    offset = offset
)

What's with the 3.8 tag? Are you actually on 3.8? 3.8 isn't even out yet, so you'd have to be on an alpha build or something. — user2357112, May 13 '19 at 09:25
I am using 3.8, because it is the first version to properly support the combination of shared memory and semaphores. Works like a charm upto now, very happy with it! (import multiprocessing.shared_memory as sm) — Jacques de Hooge, May 13 '19 at 09:26

score 1 · Accepted Answer · answered May 13 '19 at 09:14

1

Same way you'd do it if the array wasn't backed by shared memory. Just make sure you synchronize access properly.

your_array[:] = your_list

answered May 13 '19 at 09:14

user2357112

260,549
28
431
505

score 0 · Answer 2 · answered May 13 '19 at 09:20

Say you have an array of shape (list_length, tuples_length).

Is this what you're looking for?

my_sub_array[:] = my_list_of_tuples

As an example :

my_sub_array = np.zeros((5, 3))
my_list_of_tuples = [(i, i + 1, i + 2) for i in range(5)]

my_sub_array
array([[0., 0., 0.],
       [0., 0., 0.],
       [0., 0., 0.],
       [0., 0., 0.],
       [0., 0., 0.]])

my_sub_array[:] = my_list_of_tuples

my_sub_array
array([[0., 1., 2.],
       [1., 2., 3.],
       [2., 3., 4.],
       [3., 4., 5.],
       [4., 5., 6.]])

How to copy list of tuples to EXISTING numpy array in shared memory

2 Answers2