Stack Overflow
Stack Overflow

Questions tagged [pathos]

'pathos' provides a fork of python's 'multiprocessing', where 'pathos.multiprocessing' can send a much broader range of the built-in python types across a parallel 'map' and 'pipe' (similar to python's 'map' and 'apply'). 'pathos' also provides a unified interface for parallelism across processors, threads, sockets (using a fork of 'parallelpython'), and across 'ssh'.

pathos provides a fork of python's multiprocessing, where pathos.multiprocessing can send a much broader range of the built-in python types across a parallel map and pipe (similar to python's map and apply). pathos also provides a unified interface for parallelism across processors, threads, sockets (using a fork of parallelpython), and across ssh.

190 questions
0
votes
1 answer

Pathos Multiprocessing Pickling Issue

I am developing an automation framework for some testing we are doing on my team. We have come across this Pickling error and after doing some research, I came across a number of of solutions and reviewed the accompanying github page: Python…
dnwobu
  • 1
  • 2
0
votes
1 answer

failing to pickle objects and import modules with python pathos multiprocessing

Below are some toy examples to reproduce the issues I am having with pathos.multiprocessing on Python 3.5. One issue is that the parallel process fails to recognize what Test is, even when Test is not used within test(). I've seen some posts address…
tmakino
  • 618
  • 1
  • 5
  • 20
0
votes
1 answer

How can I make sure that two function calls are always made consecutively with multiple threads in Python?

I'm using Pathos like this: from pathos.multiprocessing import ProcessingPool as Pool def foo(bar): fn1(bar) fn2(bar) Pool().map(foo, data) I want fn1 and fn2 to be executed as one atomic operation such that no threads can produce function…
Lenar Hoyt
  • 5,971
  • 6
  • 49
  • 59
0
votes
1 answer

Pathos Processes Not Dying with Main

I have a main script that passes a function and a list of strings to a (Pathos) ProcessPool.map function. The problem I am having is that when I run the process from command line and try to Ctrl + C out of the main thread I am unable to end the…
ford prefect
  • 7,096
  • 11
  • 56
  • 83
0
votes
1 answer

equivalent functionality to multiprocessing.Pool's join in pathos

I'm using multiprocessing.Process to prevent some function from running for too long. I'd like to use pathos instead of python's multiprocessing (due to it's ability to handle things that the standard multiprocessing package can't), but it seems…
user1999728
  • 913
  • 1
  • 8
  • 25
0
votes
0 answers

cPickle error with pathos multiprocessing?

I'm trying to use multiprocessing to speed up pandas excel reading. However when I use multiprocessing I'm getting the error cPickle.PicklingError: Can't pickle : attribute lookup __builtin__.function failed when I try to run the following: import…
boson
  • 886
  • 1
  • 12
  • 25
0
votes
1 answer

cPickle error while using pathos.multiprocessing?

I'm trying to use multiprocessing to speed up pandas excel reading. However when I use multiprocessing I'm getting the error cPickle.PicklingError: Can't pickle : attribute lookup __builtin__.function failed when I try to run the following: import…
boson
  • 886
  • 1
  • 12
  • 25
0
votes
0 answers

pathos.multiprocessing throws pickle error when running in Linux platform

I am trying to distribute jobs across several nodes in a cluster using a module dispy in python. I am doing two level of parallel execution. First level in jobs submit, second level on each node. Originally I was using multiprocessing module…
candycode
  • 21
  • 3
0
votes
2 answers

When using the multiprocessing library, how do I bind resources to specific processes?

Say I have 50 processes, and I'm using them to operate on (say) 20000 different input values. (I'm using the pathos library, which I think operates similarly to the multiprocessing library in Python.) thread_pool =…
Jessica
  • 2,335
  • 2
  • 23
  • 36
0
votes
0 answers

Memory efficient SMP in python

I have a computationally heavy project that appears to be a good candidate for parallelization. The code uses a large number of GIS contours and takes up roughly 1.5 GB of memory as a single process. There are two levels of logic that can be…
Mike Bannister
  • 129
  • 2
  • 8
0
votes
2 answers

Python Watchdog spawn multiprocessing

I am writing files (at a really fast pace) to a specific directory. I would like to monitor the directory for new files and then spawn a process that runs an external script. Right now, I am getting a pickling error (even though I am using Pathos)…
gleb1783
  • 451
  • 1
  • 6
  • 18
0
votes
1 answer

Does pathos only work utilizing a pool? I'm looking to get around a pickle issue with multiprocessing.queues

I'm trying to create a series of worker threads that have specific jobs. The current is to draw from a queue, do some processing, and put the output in a second queue. When I try to use the multiprocessing (multiprocess? what's the difference?)…
Ryan
  • 80
  • 10
0
votes
1 answer

Pythons multiprocess module (with dill) gives an unhelpful AssertionError

I have installed dill/pathos and its dependencies (with some difficulty) and I'm trying to perform a function over several processes. The class/attribute Model(self.xml,self.exp_data,i).SSR is custom made and depends on loads of other custom…
CiaranWelsh
  • 7,014
  • 10
  • 53
  • 106
0
votes
3 answers

Python multiprocessing: checking if the memory is shared or being replicated

I have to query a large a number of vectors from a sklearn KDTree, which is path of searcher class. I am trying to query them in parallel, using python multiprocessing, however the parallel code takes almost the same (or more) time as the single…
kampta
  • 4,748
  • 5
  • 31
  • 51
0
votes
1 answer

returning a two dimensional array by multiprocessing

In the following code which is an example of my main code, I have tried to use pathos.multiprocessing to increase the speed of iteration of a loop. The output of each iteration which has implemented with multiprocessing is a 2-D array. I used…
Dalek
  • 4,168
  • 11
  • 48
  • 100
1 2 3
…
12
13