Stack Overflow
Stack Overflow

Questions tagged [pathos]

'pathos' provides a fork of python's 'multiprocessing', where 'pathos.multiprocessing' can send a much broader range of the built-in python types across a parallel 'map' and 'pipe' (similar to python's 'map' and 'apply'). 'pathos' also provides a unified interface for parallelism across processors, threads, sockets (using a fork of 'parallelpython'), and across 'ssh'.

pathos provides a fork of python's multiprocessing, where pathos.multiprocessing can send a much broader range of the built-in python types across a parallel map and pipe (similar to python's map and apply). pathos also provides a unified interface for parallelism across processors, threads, sockets (using a fork of parallelpython), and across ssh.

190 questions
2
votes
1 answer

cProfiler working weirdly with multiprocessing

I got an error for this code: from pathos.multiprocessing import ProcessingPool def diePlz(im): print('Whoopdepoop!') def caller(): im = 1 pool = ProcessingPool() pool.map(diePlz,[im,im,im,im]) if __name__=='__main__': …
Mooncrater
  • 4,146
  • 4
  • 33
  • 62
2
votes
0 answers

Python: pickling issues with pathos and selenium

I'm sorry if the answer to my question is obvious, since I'm a complete newbie to multiprocessing. I'm trying to write a multiprocessed web-scraping script with pathos. I chose pathos since, as far as I understand, unlike Python's traditional…
ntonk
  • 95
  • 8
2
votes
0 answers

Multiprocessing with Keras - Tensor is not an element of this graph

I try to use the predict-function and I am stumbling over some Problems. My keras model gets compiled in a different script (nnGenerator) inside the same package. The function which is compiling the model then returns the model to main script, where…
Lennart S.
  • 91
  • 6
2
votes
0 answers

Unable to obtain results from multiprocessing with pooling

Firstly apologies for any mistakes I make. I am new to StackOverflow and I'm still getting used to it. The Background: I try to train multiple keras models via multiprocessing. Those Keras models are stored in three lists (currently 50 models per…
Lennart S.
  • 91
  • 6
2
votes
0 answers

Running out of RAM python - where does it go

Here is my approximate workflow import multiprocessing as mp import pickle import numpy as np import psutil test = [np.random.choice(range(1, 1000), 1000000) for el in range(1,1000)] step_size = 10**4 for i in range(0,len(test), step_size): p =…
user1700890
  • 7,144
  • 18
  • 87
  • 183
2
votes
0 answers

python pathos multiprocessing

I am trying to use the pathos multiprocessing for a simple function for which I pass an array, it splits into parallel processes and then joins at the end. Here is the code I ported over from using the multiprocessing function, and then instead of…
Trexion Kameha
  • 3,362
  • 10
  • 34
  • 60
2
votes
0 answers

Pathos multiprocessing pool.map not honoring recurse=True

When I try to use pool.map on a lambdified function from sympy.lambdify, dill throws me an error: In [1]: import sympy as sy In [2]: from sympy.abc import x In [3]: f = sy.lambdify(x,x+x,'numpy') In [4]: from pathos.multiprocessing import…
Andrea Olivo
  • 21
  • 5
2
votes
1 answer

parallel files download from ftp

When i try to download files from ftp sequentially it works perfectly import ftplib import os import logging class pyFTPDownload(object): def __init__(self, remote_host=None, port=None, …
sdikby
  • 1,383
  • 14
  • 30
2
votes
1 answer

How to change position of progress bar – multiprocessing

First of, I am new to Python. It's irrelevant to the question, but I have to mention it. I am creating an crawler as my first project, to understand how things work in Python, but so far this is my major issue... Understanding "how to get multiple…
dvlden
  • 2,402
  • 8
  • 38
  • 61
2
votes
0 answers

How to use all cpus with pathos using nested pools

Answering this question about nested pools for multiprocessing: https://stackoverflow.com/a/40852258/6522112 The code I proposed was not responding the way I wanted. Which is to say if ThreadingPool(8) and ProcessingPool(3), I expected the number of…
tupui
  • 5,738
  • 3
  • 31
  • 52
2
votes
1 answer

Pathos processing pool recursion limit

I've got a task I'm trying to parallelize with pathos. When I run a non parallel version, it runs just fine. However, running a pathos pool of 1 process causes a RuntimeError: maximum recursion depth exceeded. For example: import…
CoconutBandit
  • 476
  • 1
  • 3
  • 13
2
votes
1 answer

Interaction between pathos.ProcessingPool and pickle

I have a list of calculations I need to run. I'm parallelizing them using from pathos.multiprocessing import ProcessingPool pool = ProcessingPool(nodes=7) values = pool.map(helperFunction, someArgs) helperFunction does create a class called…
FooBar
  • 15,724
  • 19
  • 82
  • 171
2
votes
2 answers

pathos multiprocessing cannot pickle

I am having a similar issue to this person. I am unable to run a simple multiprocessing routine in the pathos module and receive a pickling error. Below is the code and error. from pathos.multiprocessing import ProcessingPool import dill class…
RHam
  • 21
  • 1
  • 2
2
votes
1 answer

Python multiprocessing.Pool ignores class method

I recently wrote a program with a class for my research, and I've attempted to parallelize it. When I've used Python 2.7's multiprocessing.Process with a JoinableQueue and managed data, my program eventually hangs with defunct processes. import…
Tim
  • 107
  • 1
  • 10
2
votes
1 answer

Installing Python package pathos from git

I am trying to install pathos library from git, but getting the following error: >pip install https://github.com/uqfoundation/pathos Exception: Traceback (most recent call last): File "C:\Python27\lib\site-packages\pip\basecommand.py", line 122,…
jazzblue
  • 2,411
  • 4
  • 38
  • 63
1 2 3
…
12 13