Questions tagged [multiprocessing-manager]
88 questions
2
votes
2 answers
Python 2.7 multiprocessing - cant pass socket object via Manager dictionary
I have a simple program that searches for open ports in a local network and stores the connected sockets in a dictionary along with their local address. Now, I am using a Manager shared dictionary to store these entries but it only accepts simple…
Alex Kasapis
- 147
- 1
- 12
1
vote
1 answer
How to use multiprocessing Value to pass image base64 string?
on a project of mine I try to run Figure.savefig() on seperate multiprocessing.Process (due memory leak) but I am don't manage to allocate shared variable long enough for passing the image string.
See concept example:
import base64
from…
Nitcha
- 45
- 4
1
vote
1 answer
How to access property attribute from multiprocess Proxy class in Python?
I am trying to get attribute from my Proxy class but I don't quite understand the implementation (as per https://docs.python.org/3.9/library/multiprocessing.html?highlight=multiprocessing#multiprocessing.managers.BaseManager.register)
I understand I…
kristapsm1111
- 11
- 3
1
vote
2 answers
Python Multiprocessing custom manager with associated objects
I'm trying to make a class object usable for multiple processes. Unfortunarely this seems to be more of an issue than I anticipated.
I have the following class object:
class BusObject:
inputs: IOObject
outputs: IOObject
def…
Viktor Katzy
- 162
- 10
1
vote
0 answers
python multiprocessing.managers and Flask - how long is data stored?
I have a Flask app that generates output based on a query, and gives user multiple options to do more with the output. I need to store the data somewhere between requests and the best solution I found was the multiprocessing.managers (as suggested…
Peter Poláček
- 21
- 5
1
vote
1 answer
ValueError: 'Pool not running' when trying to loop the multiprocessing pool
I'm trying to run the same pool three times so that it adds the values in the same list:
import multiprocessing
def foo(name,archive):
print('hello ', name)
archive.append(f"hello {name}")
def main():
max_process =…
Digital Farmer
- 1,705
- 5
- 17
- 67
1
vote
1 answer
Even using Manager.list() the values are not being appended to the list using multiprocessing pool
As can see in the code below, I'm generating a multiprocessing with manager and list comprehension to handle the pool:
import multiprocessing
def foo(name,archive):
print('hello ', name)
archive.append('hello ', name)
def main():
…
Digital Farmer
- 1,705
- 5
- 17
- 67
1
vote
1 answer
Changing values of list in multiprocessing
I am new to python multiprocessing, a background about the below code. I am trying to create three processes, one to add an element to the list, one to modify element in the list, and one to print the list.
The three processes are ideally using the…
hariprasath gopal
- 13
- 2
1
vote
1 answer
Sharing and Updating List between Processes and Threads
Trying to Share AND Update a list between Main Process - Main Thread & Sub Process - Sub Thread. Currently it appears that both the main process and main thread share the update of one copy of the list, and the sub process and sub thread share…
Sanko Hunbucse
- 39
- 7
1
vote
2 answers
Sharing asyncio objects between processes
I am working with both the asyncio and the multiprocessing library to run two processes, each with one server instance listening on different ports for incoming messages.
To identify each client, I want to share a dict between the two processes to…
CRoemheld
- 889
- 7
- 26
1
vote
1 answer
How to use Python3.9.2 multiprocessing.SyncManager among remote processes?
I have modeled the following test program on the relevant Python 3.9.2 documentation regarding the synchronization of data among remote processes. As nearly as I can tell, though, it doesn't actually work, so I assume there's something I don't know.…
Steve Newcomb
- 95
- 6
1
vote
1 answer
Multiprocessing proxy: let getters return proxies themselves
I have a complex unpickable object that has properties (defined via getters and setters) that are of complex and unpickable type as well. I want to create a multiprocessing proxy for the object to execute some tasks in parallel.
The problem: While I…
Samufi
- 2,465
- 3
- 19
- 43
1
vote
1 answer
How to access classes stored in a multiprocessing.Manager list?
I can access and change the attributes of classes in a list like this:
class TestClass:
def __init__(self):
self.value = 1
instances = [TestClass()]
instances[0].value = 42
print(instances[0].value) # 42
However, when using the…
Jakob Schödl
- 380
- 2
- 16
1
vote
1 answer
How to share an aiohttp.ClientSession() between processes?
I'm trying to create a shared resource in which I can use an aiohttp.ClientSession() object to fetch certain data. This allows me to use parallelization to speed up the actual calculations.
However, when I try to create a manager in which I use…
Nesterion
- 39
- 1
- 8
1
vote
0 answers
Share a queue between processes and threads in python
I'm writing a program (python 2.7) that uses both multiprocessing and multithreading,the multiprocessing is done using Celery library.
I have a function that have to be parallelized using multithreading,so i've implemented a shared "input" queue…
Dannynis
- 73
- 4