Stack Overflow
Stack Overflow

Questions tagged [multiprocess]

Concerning running parallel code in separate processes (unlike multithreading) and/or related to package `multiprocess` from PyPI.

462 questions
6
votes
5 answers

multiprocess or multithread? - parallelizing a simple computation for millions of iterations and storing the result in a single data structure

I have a dictionary D of {string:list} entries, and I compute a function f( D[s1],D[s2] ) --> float for a pair of strings (s1,s2) in D. Additionally, I have created a custom matrix class LabeledNumericMatrix that allows me to perform assignments…
J.B. Brown
  • 1,753
  • 2
  • 13
  • 14
6
votes
5 answers

Ant simulation: it's better to create a Process/Thread for each Ant or something else?

The simple study is: Ant life simulation I'm creating an OO structure that see a Class for the Anthill, a Class for the Ant and a Class for the whole simulator. Now I'm brainstorming on "how to" make Ants 'live'... I know that there are projects…
MiPnamic
  • 1,257
  • 10
  • 18
6
votes
1 answer

SQLite update hook in a multi process application

I am using sqlite in a multi process scenario. The sqlite library was compiled with the threadsafe serialized mode (-DSQLITE_THREADSAFE=1). I want to be notified on datachanges and found sqlite3_update_hook. Every process would register it's own…
Felix
  • 6,885
  • 1
  • 29
  • 54
6
votes
1 answer

fgets() call with redirection get abnormal data stream

I was about to write a shell with C language. Here is the source code below: #include #include #include #include #include int getcmd(char *buf, int nbuf) { memset(buf, 0, nbuf); fgets(buf,…
sun
  • 95
  • 1
  • 7
6
votes
1 answer

Detecting Application class running on main process on a multiprocess app

I have an Android app that extends the Application class and has many app components (services, activities and broadcast receivers) running along three different process. Each process will instantiate the Application class as soon as it is started.…
Storo
  • 988
  • 1
  • 7
  • 31
6
votes
3 answers

ETW tracking from .net, user mode and driver

We have an application that parts of it are in .net, c++ usermode and C++ drivers. The application is divided into several executables that run on demand and communication with each other using LPC(the processes run in different…
Jack Juiceson
  • 830
  • 4
  • 12
  • 24
6
votes
2 answers

Can't get multiprocessing to run processes concurrently

The code below doesn't seem to run concurrently, and I'm not sure exactly why: def run_normalizers(config, debug, num_threads, name=None): def _run(): print('Started process for normalizer') sqla_engine =…
Brian Dilley
  • 3,888
  • 2
  • 24
  • 24
6
votes
3 answers

Fast, low-memory, constant key-value database supporting concurrent and random access reads

I need an on-disk key-value store, not too big or distributed. The use case is as follows: The full DB will be few Gbs in size Both key and value are of constant size Its a constant data base. Once the entire database is written I don't need to…
san
  • 4,144
  • 6
  • 32
  • 50
5
votes
2 answers

How to count CPU Usage of multiprocess application in Linux

I try to make a program with C/C++, to behave like top command in Linux. I've done some research and already known how to count CPU Usage of a process. We can get the CPU Usage by calculating stime + utime from /proc/[PID]/stat in current time and…
Luck Haryadi
  • 93
  • 9
5
votes
1 answer

Is it possible to know which SciPy / NumPy functions run on multiple cores?

I am trying to figure out explicitly which of the functions in SciPy/NumPy run on multiple processors. I can e.g. read in the SciPy reference manual that SciPy uses this, but I am more interested in exactly which functions do run parallel…
matiasq
  • 537
  • 1
  • 6
  • 12
5
votes
3 answers

Speed per process getting slower with more processes

I am trying to improve the speed of some code with multiprocess. And I noticed the speed does not increase as expected. I know there are overheads for the spawn of child processes and there are overheads for data transfer between the parent process…
Y. Zhang
  • 51
  • 3
5
votes
0 answers

How can I figure out "cannot pickle '_io.BufferedReader' object" error when trying to multiprocess with django

I'm trying to do db connection through multiprocessing for my app. I assume that an error is occurring from where I call my function. I've been struggled with multiprocessing in these few days. I found the codes from…
Ethan Shin
  • 53
  • 2
  • 6
5
votes
1 answer

assert self._state in (CLOSE, TERMINATE) when using python multiprocess

I am currently trying to use python multiprocessing. The library I use is multiprocess (NOT multiprocessing). I have the following code, which creates a number of computing jobs, and runs it through a map operation: pool =…
DaveTheAl
  • 1,995
  • 4
  • 35
  • 65
5
votes
1 answer

Multiprocessing of a function on a pandas dataframe

I have a large pandas dataframe with multiple "records" consisting of 2 or more line items. I'm trying to efficiently perform a CPU intensive calculation on each record using multiprocessing. Here's a simplified example with a function that just…
David
  • 79
  • 1
  • 1
  • 7
5
votes
1 answer

python multiprocessing freezing

I am trying to implement multiprocessing with Python. It works when pooling very quick tasks, however, freezes when pooling longer tasks. See my example below: from multiprocessing import Pool import math import time def iter_count(addition): …
hercules.cosmos
  • 265
  • 1
  • 3
  • 10
1 2
3
…
30 31