Stack Overflow
Stack Overflow

Questions tagged [multiprocess]

Concerning running parallel code in separate processes (unlike multithreading) and/or related to package `multiprocess` from PyPI.

462 questions
2
votes
1 answer

Multiple processes sharing a listening socket: When a new process enters why does the old one stop?

These code is the Server part of my proxy program, and its function is creating the socket and fork four process to accept one by one. In my program I use gevent model to dispatch all my function and before I alter it to multiple process, my…
Ding
  • 105
  • 1
  • 1
  • 8
2
votes
0 answers

Monkey patching cloudpickle as multiprocessing pickler

I am trying to use PyDREAM to sample a likelihood that has a number of dynamically constructed elements, i.e., class factories. The class factories are pretty necessary, so making the likelihood function easier to pickle is not really an option.…
BanjoKayaker
  • 21
  • 2
2
votes
1 answer

Running Two asyncio Coroutines, Each in its own Python Process

If we have 2 asyncio coroutines, is it possible to use Python multiproessing to let each of them run in its own process, and allow the coroutines in both processes to be stopped (by calling their stop method) when the user hits Ctrl+C? This will be…
Nyxynyx
  • 61,411
  • 155
  • 482
  • 830
2
votes
0 answers

Tips to Speed up the keras sequence generator?

I have a custom data generator that inherence tf.keras.utils.Sequence class. And I have found that in tensorflow doc, tf.data is more efficient to use and recommend. But re-writing Sequence to tf.data can be costly. There are two option I can…
Mahmudur Rahman
  • 77
  • 6
2
votes
1 answer

WorkManager 2.5.0 mulitprocess - work not running in designated process

I'm trying to use androidx.work:work-multiprocess:2.5.0 to run work in a process specified using Configuration.setDefaultProcessName(), but no matter which process I enqueue the work from, the doWork() method is called in main application…
Mateusz Piotrowski
  • 83
  • 6
2
votes
1 answer

Function not returning in multiprocess, no errors

I am running multiple processes with Pool import spacy import multiprocessing import logging # global variable nlp_bert = spacy.load("en_trf_bertbaseuncased_lg") logging.basicConfig(level=logging.DEBUG) def job_pool(data, job_number, job_to_do,…
forgetso
  • 2,194
  • 14
  • 33
2
votes
3 answers

multiprocessing: how to send data from parent to continuously running child process?

Is it possible to use multiprocessing to send data to a running child process? For example, running a web server in a child process and using the parent process to send data to the server. I found this question about the exact reverse of my…
rustbird
  • 155
  • 1
  • 5
2
votes
1 answer

parallelizing python finite element analysis

I am trying to parallelize the python function below, which is involved in finite element analysis. In particular, I am trying to make the for loop in the function runs in parallel. I have done it in Matlab using parfor, and I am trying to do the…
Abu2Stack
  • 23
  • 4
2
votes
1 answer

Performance gradually decreases while running Python multiprocess

I am running a complicated python method (single run took about 3-5 mins) over multiple cases using multiprocess. I noticed that the running speed is normal when I first kicked off the program: I can get about 20-30 outputs in 5 mins using 30 cores,…
ilovecp3
  • 2,825
  • 5
  • 18
  • 19
2
votes
1 answer

Sharing/using object between processes in python

I am trying to use an object (of class Event) between two processes. The main process creates the object (self.event) and passes it to the new process as args. The new process will call the inject() method of the Event() in foo1() and again the main…
Akal
  • 21
  • 2
2
votes
3 answers

Process large csv file and limit goroutines

I'm trying to find the best efficient way to read a csv file (~1M row). Each row contain a HTTP link to an image which I need to download. This is my current code using worker pools: func worker(queue chan []string, worknumber int, done, ks chan…
Ilan
  • 569
  • 1
  • 8
  • 21
2
votes
2 answers

Python multiprocess library: NameError on using imported packages

The following code works perfectly: from multiprocessing import Pool import time values = list(range(10)) def print_time_and_value(value): print(time.time(), value) if __name__ == '__main__': p = Pool(4) p.map(print_time_and_value,…
Rodrigo Bonadia
  • 125
  • 2
  • 8
2
votes
0 answers

Multiprocessing error for NLP application

I'm working on an NLP project. I have a massive dataset of 180 million words. Before I begin training I want to correct the spelling of words. To do this I use TextBlob's spell correct. Because TextBlob takes a while to process anyways, it would be…
Sam
  • 641
  • 1
  • 7
  • 17
2
votes
0 answers

Overlapping print lines in multiprocessing

I found myself dealing with multiprocessing recently and I have about 10 processes that are printing simultaneously on different files but on the same console(Eclipse console). The amount of prints is about 40 to 50 lines per second for each…
Alexandru Martalogu
  • 273
  • 2
  • 12
2
votes
0 answers

Python how to use multiprocessing pool in tkinter app

How to use multiprocessing pool in tkinter app? How to avoid getting TypeError: can't pickle _tkinter.tkapp objects import tkinter as tk import multiprocessing class UI(): def __init__(self): self.root = tk.Tk() self.input =…
2607
  • 4,037
  • 13
  • 49
  • 64
1 2 3
…
30 31