Stack Overflow
Stack Overflow

Questions tagged [pathos]

'pathos' provides a fork of python's 'multiprocessing', where 'pathos.multiprocessing' can send a much broader range of the built-in python types across a parallel 'map' and 'pipe' (similar to python's 'map' and 'apply'). 'pathos' also provides a unified interface for parallelism across processors, threads, sockets (using a fork of 'parallelpython'), and across 'ssh'.

pathos provides a fork of python's multiprocessing, where pathos.multiprocessing can send a much broader range of the built-in python types across a parallel map and pipe (similar to python's map and apply). pathos also provides a unified interface for parallelism across processors, threads, sockets (using a fork of parallelpython), and across ssh.

190 questions
4
votes
1 answer

How to run nested, hierarchical pathos multiprocessing maps?

Having build a significant part of my code on dill serialization/pickling, I'm also trying to use pathos multiprocessing to parallelize my calculations. Pathos it is a natural extension of dill. When trying to run nested from pathos.multiprocessing…
Mark Horvath
  • 1,136
  • 1
  • 9
  • 24
4
votes
1 answer

How to pass keywords list to pathos.multiprocessing?

I am using pathos.multiprocessing to parallelize a program that requires using instance methods. Here is a minimum working example: import time import numpy as np from pathos.multiprocessing import Pool, ProcessingPool, ThreadingPool class…
astabada
  • 1,029
  • 4
  • 13
  • 26
4
votes
1 answer

Distributed multiprocessing pool in Python

I have an existing bit of Python code that runs in parallel across the cores in my machine. The job it completing is basically open an input file, read the contents, perform some fairly heavy maths, write the results to an output a file, take the…
Mark
  • 1,277
  • 3
  • 13
  • 27
4
votes
1 answer

Does the dill python module handle importing modules when sys.path differs?

I'm evaluating dill and I want to know if this scenario is handled. I have a case where I successfully import a module in a python process. Can I use dill to serialize and then load that module in a different process that has a different sys.path…
Brent V
  • 43
  • 1
  • 3
4
votes
2 answers

Python Multiprocessing: AttributeError: 'Test' object has no attribute 'get_type'

short short version: I am having trouble parallelizing code which uses instance methods. Longer version: This python code produces the error: Error Traceback (most recent call last): File…
Gil Zellner
  • 889
  • 1
  • 9
  • 20
4
votes
1 answer

Python: multiprocessing, pathos and what not

I have to apologise in advance 'cause this question is quite general and may be not clear enough. The question is: how would you run in parallel a Python function that itself uses a pool of processes for some subtasks and does lots of heavy I/O…
oopcode
  • 1,912
  • 16
  • 26
3
votes
2 answers

Python multiprocessing with large objects: prevent copying/serialization of object

I have implemented multiprocessing for some problem with larger objects like the following: import time import pathos.multiprocessing as mp from functools import partial from random import randrange class RandomNumber(): def __init__(self,…
Cord Kaldemeyer
  • 6,405
  • 8
  • 51
  • 81
3
votes
1 answer

More parallel processes than available processors in pathos

I used to be able to run 100 parallel process this way: from multiprocessing import Process def run_in_parallel(some_list): proc = [] for list_element in some_list: time.sleep(20) p = Process(target=main,…
user165494
  • 49
  • 2
3
votes
1 answer

PyTesseract call working very slow when used along with multiprocessing

I've a function that takes in a list of images and produces the output, in a list, after applying OCR to the image. I have an another function that controls the input to this function, by using multiprocessing. So, when I have a single list (i.e. no…
Mooncrater
  • 4,146
  • 4
  • 33
  • 62
3
votes
1 answer

TypeError: can't pickle CompiledFFI objects

I am trying to get output from Telnet and SSH hosts for some commands and store them in a shelf. Since there are many commands, I am using multiprocessing. I have the following important methods: connectToHost: Making a connection (SSH/ Telnet)…
Shubham Shrivastava
  • 83
  • 1
  • 12
3
votes
1 answer

Pathos.multiprocessing's Pool appears to be nonlocal?

A code of mine does from pathos.multiprocessing import ProcessingPool def myFunc(something): thispool = ProcessingPool(nodes=Result.cores) listOfResults = thispool.map(something) return listOfResults for i in range(1000): …
FooBar
  • 15,724
  • 19
  • 82
  • 171
3
votes
1 answer

Python multiprocessing & pathos: import error

I am writing some python code using multiprocessing & pathos. I have written a small test program to get used to using the mutiprocessing, that runs fine on my local machine, but it refuses to run on a different cluster. I am getting the following…
abinitio
  • 609
  • 6
  • 20
3
votes
1 answer

Pathos multiprocessing can't call any package and function in the class

I want to do multiprocessing in the class. It seems like only pathos.multiprocessing is able to help me. However, when I implement it, it can't load the packages I use in the main function. from pathos.multiprocessing import ProcessingPool; import…
Runpeng Chen
  • 105
  • 8
3
votes
2 answers

parallel list comprehension using Pool map

I have a list comprehension: thingie=[f(a,x,c) for x in some_list] which I am parallelising as follows: from multiprocessing import Pool pool=Pool(processes=4) thingie=pool.map(lambda x: f(a,x,c), some_list) but I get the following…
laila
  • 1,009
  • 3
  • 15
  • 27
3
votes
2 answers

Python: (Pathos) Multiprocessing vs. class methods

I am trying to parallelize a code using class methods via multiprocessing. The basic structure is the following: # from multiprocessing import Pool from pathos.multiprocessing import ProcessingPool as Pool class myclass(object): def…
user4319496
  • 236
  • 3
  • 12
1
2
3
…
12 13