Stack Overflow
Stack Overflow

Questions tagged [pathos]

'pathos' provides a fork of python's 'multiprocessing', where 'pathos.multiprocessing' can send a much broader range of the built-in python types across a parallel 'map' and 'pipe' (similar to python's 'map' and 'apply'). 'pathos' also provides a unified interface for parallelism across processors, threads, sockets (using a fork of 'parallelpython'), and across 'ssh'.

pathos provides a fork of python's multiprocessing, where pathos.multiprocessing can send a much broader range of the built-in python types across a parallel map and pipe (similar to python's map and apply). pathos also provides a unified interface for parallelism across processors, threads, sockets (using a fork of parallelpython), and across ssh.

190 questions
2
votes
0 answers

Multiprocessing Large Objects Using Pathos in Python

I am trying to make use of my computer's multiple CPUs. However, the BeautifulSoup object returned by my function as part of an SQLAlchemy object is not picklable with pickle or cPickle so I am using pathos, a fork of the multiprocssing package…
Michael
  • 13,244
  • 23
  • 67
  • 115
1
vote
2 answers

Python Multiprocessing Can't pickle local object

i have read a little about multiprocessing and pickling problems, I have also read that there are some solutions but I don't really know how can they help to mine situation. I am building Test Runner where I use Multiprocessing to call modified Test…
Alraku
  • 45
  • 1
  • 10
1
vote
1 answer

Multiprocessing return objects created by process

I am using multiprocessing lib for accelerating process of class creation. Minimal example of my code: from matplotlib.backends.backend_qt5agg import FigureCanvas class Custom_Class(FigureCanvas) . . . def…
justRandomLearner
  • 166
  • 2
  • 18
1
vote
1 answer

'cannot pickle 'PyCapsule' object' occurring with Mystic diffev2

I am trying to optimize some inputs to a third party software. This software utilizes a series of input files that are created by separate executables and are then merged together into a single input file for the primary executable. The goal is…
Michael Branagan
  • 71
  • 1
  • 6
1
vote
0 answers

Dill sometimes makes attributes of objects disappear

I have a class 1 (Protpkg) with a custom-defined __new__ method that uses an object of a different custom class 2 (State) as argument and saves it as instance attribute. Then, the initialization (__init__) of this class 1 uses a multiprocess Pool to…
Francho Nerín Fonz
  • 311
  • 2
  • 7
1
vote
0 answers

Multiprocessing with PySwig Object

PySwig is a C++ wrapper which creates objects which (for unavoidable reasons) cannot be pickled by Python. I want to access a method on that object to run across a large dataset (around 1M entries) in parallel. I can run a function with multiple…
Tarquinnn
  • 501
  • 3
  • 8
1
vote
1 answer

Pathos "Error has occured during the function import"

When I try to execute this code: import pathos from network import * def simulation(error_rate=1): network = Network(error_rate=1) network.gen_transceiver_pair(1) network.run() mend = network.master_transceivers[0].cycle[0, 2] …
mooder
  • 341
  • 2
  • 8
1
vote
1 answer

parallelizing code with python pathos multiprocessing in docker

I am parallizing code across 30 CPUs and confirmed that outside a container this works fine using the python library 'pathos'. pool = ProcessPool(nodes=30) results = pool.map(func_that_needs_to_run_in_parallel,…
hoola_huupsh
  • 11
  • 4
1
vote
0 answers

Python Pathos Multiprocessing: How to terminate the pool under a certain condition?

I need to run a bunch of parallel processes, but cannot use the standard multiprocessing package since its serialization with pickle does not work for more complex objects. Therefore I'm currently using pathos.multiprocessing which uses dill for the…
Nick
  • 11
  • 2
1
vote
1 answer

Calling multiprocessing pool within a function is very slow

I am trying to use pathos for triggering multiprocessing within a function. I notice, however, an odd behaviour and don't know why: import spacy from pathos.multiprocessing import ProcessPool as Pool nlp = spacy.load("es_core_news_sm") def…
Vladimir Vargas
  • 1,744
  • 4
  • 24
  • 48
1
vote
0 answers

How to debug Python Pathos multiprocessing

I am using pathos.Pool to run multiprocessing. I try to catch multiprocessing errors and specific information of the multiprocessing errors such as the lines of the wrong codes and error types. I want the error information to be specific and…
刘晏希
  • 11
  • 2
1
vote
0 answers

Pickling error when using pathos with XML

I'm trying to read the multistream Wikipedia dump into a database. This is my attempt at loading smaller chunks in parallel. Here's the script: #!/usr/bin/python3 import xml.sax from bz2 import BZ2File import mwparserfromhell import psycopg2 import…
local_oaf
  • 31
  • 1
  • 4
1
vote
1 answer

Using Pygame with parallelism in python

I am trying to train a neural network to play a SMB1 game made using Pygame. To get some speed up, I would like to use parallel processing in order to play multiple copies of the game at once, by different members of the population (and on different…
minerharry
  • 101
  • 8
1
vote
1 answer

python pathos - main process runs very slow and sub-processes run serially

I need to speed up processing time for tasks which involve rather large data sets that are loaded from up to 1.5GB large pickle files with CSV data. I started with python's multiprocessing but for unpickleable class objects I had to switch to…
pisti
  • 71
  • 10
1
vote
1 answer

How to efficiently share dicts and lists between processes using ProcessPool

Let's consider the following example: from pathos.pools import ProcessPool class A: def run(self, arg: int): my_list = list(...) my_dict = dict(...) def __run_parallel(arg: int): local_variable = 42 …
p4dn24x
  • 445
  • 4
  • 14
1 2 3
…
12 13