Concerning running parallel code in separate processes (unlike multithreading) and/or related to package `multiprocess` from PyPI.
Questions tagged [multiprocess]
462 questions
2
votes
1 answer
boost::asio server multi-process
I would to make a simple multi process (not thread) server. I've seen the iterative example in which it handles one request at a time.
Instead I need to handle more requests(more on less 10) at the same time.
In the classic c and c++ examples, I've…
marco
- 21
- 3
2
votes
0 answers
Something blocks threads while accessing fifo from different processes
I have a Perl script and Python script. Perl script feeds Python script by means of FIFO. Python script executes the Perl script and also runs some threads to process the data output from Perl. I read the FIFO directly into pandas…
nurp
- 1,239
- 2
- 14
- 23
2
votes
1 answer
why has the function for pool.map to be defined before multiprocessing.pool?
When using multiprocessing.pool and pool.map, it seems that the map function has to be defined before creating the pool. Why is this necessary?
Here is the working version:
import multiprocessing
def my_power(x):
return x * x
_pool =…
user7586189
- 1,249
- 8
- 17
2
votes
1 answer
Migrate from Multiprocess to MPI in python
I'm tring to move a code from multiprocces to MPI using python becouse I'm moving my code in an HPC. In this why I wnat to span the procces on several nodes and aovid to use all the ram of a signle node.
Can you help me implementing it? I have tried…
Lorenzo Bottaccioli
- 441
- 1
- 7
- 20
2
votes
1 answer
how to kill the parent process and all its child processes in c?
So the following function should be called to kill the parent process. But it leaves the child processes alive. I know you can pass a argument to the sigkill in shell command to kill all the child process under this process group. But how to do this…
J.R.
- 769
- 2
- 10
- 25
2
votes
0 answers
Memory issue with multiprocessing in Python
I am trying to use my other cores in my python program. And the following is the basic structure/logic of my code:
import multiprocessing as mp
import pandas as pd
import gc
def multiprocess_RUN(param):
result = Analysis_Obj.run(param)
…
Jason Tam
- 55
- 8
2
votes
1 answer
Python/Twisted - Sending to a specific socket object?
I have a "manager" process on a node, and several worker processes. The manager is the actual server who holds all of the connections to the clients. The manager accepts all incoming packets and puts them into a queue, and then the worker processes…
ryeguy
- 65,519
- 58
- 198
- 260
2
votes
1 answer
How to avoid multiple writers in a named pipe?
I am writing a program with a named pipe with multiple readers and multiple writers. The idea is to use that named pipe to create pairs of reader/writer. That is:
A reads the pipe
B writes in the pipe
(vice versa)
Pair A-B created!
In order to…
Javier J. Salmeron Garcia
- 485
- 7
- 18
2
votes
2 answers
multi task at once in python
I'm trying to apply certain function with two adjacent elements in given data set. Please refer to the following example.
# I'll just make a simple function here.
# In my real case, I send request to database
# to get the result with two…
higee
- 402
- 1
- 7
- 16
2
votes
3 answers
python spreading subprocess.call on multiple CPU cores
I have a python code that uses the subprocess package to run in shell:
subprocess.call(mycode.py, shell=inshell)
When I execute the top command I see that I am only using ~30% or less of CPU.
I realize some commands may be using disk and not cpu…
Arya
- 189
- 4
- 17
2
votes
2 answers
How can I send messages (or signals) from a parent process to a child process and viceversa in Perl?
Im writing a program that manage muti-processes. This is what I have done and its working great! but now, I want to send messages from the child processes to the parent process and viceversa (from the parent to the childs), do you know the best way?…
pablomarti
- 2,087
- 2
- 22
- 35
2
votes
0 answers
OSError: [Errno 22] Invalid argument
I am trying to use multiprocess (in order to better pickle objects) and cassandra.concurrent to insert rows. This is my code,
from multiprocess.pool import Pool
from cassandra.cluster import Cluster
from cassandra.concurrent import…
daiyue
- 7,196
- 25
- 82
- 149
2
votes
2 answers
Python: check count of processed elements when using multiprocessing.map
I am using multiprocessing module to do parallel url retrieval. My code is like:
pat = re.compile("(?Phttps?://[^\s]+)")
def resolve_url(text):
missing = 0
bad = 0
url = 'before'
long_url = 'after'
match = pat.search(text)…
gladys0313
- 2,569
- 6
- 27
- 51
2
votes
1 answer
Why node.js can call clustet.fork() inside of the if statement?
For a typical C program, we do something like this to create a new process:
int main(void)
{
pid_t childPID;
childPID = fork();
if(childPID >0){
do something
}
else if(childPID == 0){
do something
}
else {
…
silverwen
- 287
- 3
- 11
2
votes
0 answers
PHP Apache Enable multiprocess/thread
Well, I have a web application with multiple tools.
One such tool sends a simple Ajax request at the same PHP script, which in turn sends an HTTP request via Curl, but the problem is that this request takes a long time.
As this process takes a long…
Olaf Erlandsen
- 5,817
- 9
- 41
- 73