Stack Overflow
Stack Overflow

Questions tagged [python-multiprocessing]

multiprocessing is a package that supports spawning processes using an API similar to the threading module in python programming language. When asking a question about this topic, please mention the operating system you are running it on, or add it to the tags.

The multiprocessing package offers both local and remote concurrency, effectively side-stepping the Global Interpreter Lock by using Operating System processes instead of threads.

The multiprocessing module allows the programmer to fully leverage multiple processors on a given machine. It runs on both Unix and Windows.

4758 questions
1
vote
0 answers

How to synchronize background tasks across multiple FastAPI processes?

I have a FastAPI application which sends emails when the users register on the website. The app is implemented in such a way that there is a cron task (scheduled every minute) which checks the database and if a flag is set, it will try to send an…
Ranjan Prasad
  • 31
  • 1
1
vote
0 answers

python parallel pandas chunk

I have extremely large file need be to processed. Also perfect time to learn parallelism. My idea (potentially wrong) was to read file in chunks. Not one at a time, but multiple chunks at once (simultaneously). If I allocate 4 cores, than for file…
Zoomman
  • 37
  • 5
1
vote
0 answers

Python multiprocessing.Manager shared data and CLI pipe

I've narrowed a problem I'm having down to this simple example. This is test.py: #!/usr/bin/env python3 import sys import multiprocessing _THREADS = 4 #------------------------------------------------------------------------------ def _process(…
A. Que
  • 131
  • 1
  • 1
  • 6
1
vote
0 answers

Execute function from loop only after last execution has finished

I have a loop (all this is being done in Python 3.10) that is running relatively fast compared to a function that needs to consume data from the loop. I don't want to slow down the data and am trying to find a way to run the function asynchronously…
kmdewey
  • 61
  • 7
1
vote
1 answer

SystemExit deadlock with multiprocessing + asyncio

The following code deadlocks instead of exiting when SystemExit is raised in the asyncio task. import asyncio import multiprocessing as mp def worker_main(pipe): try: print("Worker started") pipe.recv() finally: …
Ian Goldby
  • 5,609
  • 1
  • 45
  • 81
1
vote
0 answers

How to handle receiving real time signals and passing signals into inference engine on Jetson Platforms

I am just looking for some advice on how to handle receiving a large number of signals in real time, storing these on a buffer and then passing the buffer to be processed by a inference engine/model on Jetson(l4t) platforms. Currently I have…
eskay
  • 11
  • 3
1
vote
1 answer

Changing values of list in multiprocessing

I am new to python multiprocessing, a background about the below code. I am trying to create three processes, one to add an element to the list, one to modify element in the list, and one to print the list. The three processes are ideally using the…
hariprasath gopal
  • 13
  • 2
1
vote
2 answers

Sleeping jobs occupying cores when using python multiprocessing pool

I am using multiprocessing.Pool in python to schedule around 2500 jobs. I am submitting the jobs like this: pool = multiprocessing.Pool() for i from 1 to 2500: # pseudocode jobs.append(pool.apply_async(....)) for j in jobs: _ =…
In78
  • 440
  • 4
  • 17
1
vote
0 answers

Error: "There is no current event loop in thread" when used concurrent.futures with HTMLSession

Can you help me find a bug in my code ? I'm trying to speed up web scraping URLs gathered by using googlesearch.search. *Note! This seems similiar issue as it was described in this post: Concurrent.futures + requests_html's render() = "There is no…
Rad226
  • 97
  • 8
1
vote
1 answer

Mutate a shared object in python Multiprocessing

Assuming that there is a dummy.txt file that holds the following information: a_to_b_from_c 20 a_to_b_from_w 30 a_to_b_from_k 20 a_to_b_from_l 10 c_to_f_from_e 30 c_to_f_from_k 20 c_to_f_from_l 10 (Numerical values are only 10,20, and 30) and the…
ex1led
  • 427
  • 5
  • 21
1
vote
1 answer

Using multiprocessing doesn't speed-up code

I was trying to speed up the following code, using multiprocessing . So I used this code, after getting some help here : from multiprocessing import Pool, Manager, cpu_count from functools import partial import re def process_value(data_set_N,…
Anass
  • 396
  • 2
  • 10
1
vote
0 answers

Problem with Python multiprocessing on Windows

I made a program for a chi square test, which works but was too slow, so I wanted to implement multiprossesing, but it didn't work for me (on a Windows machine). My CPU utilization goes to 100% and starts to fluctuate but i don't get further output.…
Movo
  • 11
  • 2
1
vote
1 answer

Can one get all attributes in a Python multiprocessing manager namespace?

I have a multiprocessing namespace initialized as the following from multiprocessing import Manager manager = Manager() manager_namespace = manager.Namespace() manager_namespace.__setattr__('x', 1) manager_namespace.__setattr__('y',…
Lafonz
  • 23
  • 6
1
vote
1 answer

Why can't execute multiprocess using member variables of class?

I tested a code using python 3.6 and python 3.7 by 2 cases. 1 case) using Member variables of class import multiprocessing as mp import sys class Foo: def do_multiprocessing(self): ctx = mp.get_context('spawn') self.process_1 =…
DongHwan Shin
  • 93
  • 8
1
vote
0 answers

Multiprocessing with numpy causes crash

I am trying to parallelize my code across multiple processes. The computation is mostly CPU intensive, so I decided to go for multiprocessing rather than multithreading. I am given a matrix (~300/400 elements) and a vector (~10 elements). First, I…
tigerjack
  • 1,158
  • 3
  • 21
  • 39
1 2 3
…
99
100