Stack Overflow
Stack Overflow

Questions tagged [pathos]

'pathos' provides a fork of python's 'multiprocessing', where 'pathos.multiprocessing' can send a much broader range of the built-in python types across a parallel 'map' and 'pipe' (similar to python's 'map' and 'apply'). 'pathos' also provides a unified interface for parallelism across processors, threads, sockets (using a fork of 'parallelpython'), and across 'ssh'.

pathos provides a fork of python's multiprocessing, where pathos.multiprocessing can send a much broader range of the built-in python types across a parallel map and pipe (similar to python's map and apply). pathos also provides a unified interface for parallelism across processors, threads, sockets (using a fork of parallelpython), and across ssh.

190 questions
0
votes
1 answer

Python distributed ThreadPool

Do any of the Python ThreadPool packages (multiprocessing, pathos) distribute worker threads across multiple cpus? If not would I need to need to create a ProcessPool, then create a ThreadPool on each ProcessPool worker?
user1040535
  • 201
  • 1
  • 3
  • 14
0
votes
1 answer

Serialization of ctypes.structure in Python

I am trying to work with a Python wrapper using ctypes. Modifying an example given by the manufacturer, I send frames as a ctypes.structure object, which works fine. But this object cannot be serialized, so I cannot setup a queue for a separate…
Cindy Almighty
  • 903
  • 8
  • 20
0
votes
1 answer

Understanding pathos: classes and tkinter

I am having moderate success in using pathos and multiprocessing. However, pickling is an issue - and anything containing tkinter (my current GUI) seems to mean that I have to be extremely careful what class-level variable I use for multiprocessing.…
Tarje Bargheer
  • 175
  • 3
  • 8
0
votes
1 answer

How do I get pathos to spawn processes on my remote server?

I have 2 computers, both of which have the pathos Python module. I have a Pathos multiprocessing pool and have been trying to get pathos to split the number of processes evenly between the two CPUs using the following code: from…
Montana Burr
  • 269
  • 4
  • 21
0
votes
0 answers

Pathos queuing tasks then getting results

I have a computing task which effectively is running the same piece of code against a large number of datasets. I'm wanting to utilise large multicore systems(for instance 72 cores). I'm testing it on a 2 core system output_array = [] pool =…
David Findlay
  • 1,296
  • 1
  • 14
  • 30
0
votes
1 answer

Share a queue object between pool processes

I use the pathos ProcessingPool class to schedule concurrent execution of the run_regex() function across multiple cores. The function takes a regular expression as an argument and evaluates list entries for a match. If a match is found, it puts the…
zan
  • 355
  • 6
  • 16
0
votes
0 answers

Multiprocessing fails for ctypes shared library function inside class

I think my question was not very clear before.. I'm trying to create a class module that includes a function does a mutiprocessing of a ctpyes function. I'm re-posting a working small code. All I want to do is to remove the code below and call the…
윤제균
  • 149
  • 1
  • 10
0
votes
2 answers

Python pathos error "ERROR:root:"

I am using Pathos for parallelism: https://pypi.org/project/pathos/ All good except when instantiating the Pool I get some annoying error messages. from pathos.multiprocessing import ProcessingPool as Pool p = Pool().map #<--This line spits out…
user3761555
0
votes
1 answer

Serializing objects for multiprocessing is slow -- is there a way to serialize just once?

I am trying to parallelize a function that takes in an object in Python: In using Pathos, the map function automatically dills the object before distributing it among the processors. However, it takes ~1 min to dill the object each time, and I need…
user154510
  • 119
  • 2
0
votes
0 answers

Pathos ThreadPool() return unordered sequence

im predicting some data to my machine learning. Then, im using pathos ThreadPool to predict multiple data at the same time. Example data 1 data 2 data 3 data 4 And when i use ThreadPool().map(predict,data) it return wrong sequence of json: data…
Fregy
  • 111
  • 1
  • 7
0
votes
1 answer

pathos ImportError: __import__ not found

I'm running the following code on Linux - Ubuntu with Python 3.5 in a virtual environment, and getting the error below. It works fine for threading and for the list comprehension part, but I'm having problems getting the multiprocessing to…
T. Wilson
  • 33
  • 1
  • 4
0
votes
2 answers

Pathos multiprocessing in class produces garbled standard output

I'm trying to use multiprocessing in a class I have written to speed up calculations. I'm using pathos.multiprocessing and dill, and using map on a ProcessingPool. I've tested the functionality of multiprocessing in a console and it performed as…
S. Laroche
  • 1
0
votes
1 answer

Pathos pool prints mysterious error

I'm running from pathos.multiprocessing import ProcessingPool pool = ProcessingPool(4) pool.map(...) and the following gets printed in run time: not in depth 2 not in depth 3 not in depth 1 not in depth 2 not in depth 3 not in depth 1 not in depth…
FooBar
  • 15,724
  • 19
  • 82
  • 171
0
votes
0 answers

Python Multiprocessing Not Speeding Up

I have a code where a shared resource is changed by a function call. So far, for each input vector (the input matrix of dimension rxc), I'm running it serially. I want to change a shared resource (say R) with each function call. I've so far tried…
damith219
  • 137
  • 11
0
votes
2 answers

how to handover *args to pathos' amap

I have the following problem: I would like to use a amap from pathos.multiprocessing. import pathos as pt class Foo: def __init__(self): pass def f(self, a, b, c): return a + b + c def g(self): pool =…
math
  • 1,868
  • 4
  • 26
  • 60
1 2 3
…
12 13