Stack Overflow
Stack Overflow

Questions tagged [concurrent-processing]

[CONCURRENT] processing, in sharp contrast to a [PARALLEL] processing, has no specific requirements on when to start, where & in what particular order to execute and when or how to finish a set of tasks, that got a chance to be executed in a free flowing, uncoordinated fashion on a subset of available resources.

In a "just"-[CONCURRENT] process orchestration, the code-units (be it tasks, threads, programmes... ) may but, need not, appear to get executed in some non-[SERIAL], overlapped in time, fashion or simultaneously (just by coincidence), all that by using more than one CPU or processor core and other shared resources to execute a program or multiple computational units.

A "just"-[CONCURRENT] process orchestration does not warrant any mutual coordination, the less a synchronicity a True-[PARALLEL] warrants by definition.

A "just"-[CONCURRENT] process orchestration enables more things to happen con-currently, the process-scheduler and other constraints participate in evaluating, who gets "elected" for such a con-current processing and who not.

68 questions
2
votes
1 answer

How do i pass objects to executor.map along with iterables?

import glob, os import tarfile import concurrent.futures def function(file1,arc): print(file1) arc.add(file1) destination="/home/lol/org" src=["a","b"] for i in src: if not os.path.exists(os.path.join(destination,i)): …
Varun Sudarshan
  • 29
  • 5
2
votes
3 answers

Invalid Cross-Thread Operations from BackgroundWorker2_RunWorkerCompleted in C#

I'm getting an error that does not make sense. Cross-thread operation not valid: Control 'buttonOpenFile' accessed from a thread other than the thread it was created on. In my application, the UI thread fires off backgroundWorker1, which when almost…
Jim Fell
  • 13,750
  • 36
  • 127
  • 202
2
votes
5 answers

Download multiple pages concurrently?

I'd like to write a script in Python that can grab url's from a database, and download web pages concurrently to speed things instead of waiting for each page to download one after the other. According to this thread, Python doesn't allow this…
Gulbahar
  • 5,343
  • 20
  • 70
  • 93
2
votes
5 answers

Preventing spamming of the functionality of a php page

The background: Ok, I run a legacy BBG over at ninjawars.net. There is an "attack" that players can make on other players that is initialized via form post. Essentially, we can simplify the situation to pretend that there's a page, lets call it…
Kzqai
  • 22,588
  • 25
  • 105
  • 137
2
votes
1 answer

Scala actors as single-threaded queues

I'd like to use actors in a program where I'll have some kind of restriction around treating some of the actors as if they were queues. For example, suppose I have some external system to which change events are applied and also some cache of the…
oxbow_lakes
  • 133,303
  • 56
  • 317
  • 449
1
vote
0 answers

Running independent endpoint requests on parallel in Python

I have a functions which is being accessed using REST API methodology by Fast API. Right now, when user request my endpoints, each method is processed synchronously. async def endpoint1(param1, param2): return await some_async_method1() async…
Katie
  • 21
  • 2
1
vote
0 answers

Aggregation tree for parallel file concatenation in Python?

I have a use-case where I need to concat a large number of CSV files into one, maintaining the order of the rows. For example: > cat file1.csv: 1,bla,bla 2,bla,bla > cat file2.csv 2,bla,bla 2,bla,bla 3,bla,bla > cat…
Bar
  • 2,736
  • 3
  • 33
  • 41
1
vote
0 answers

How to improve running time with joblib.Parallel and web requests?

I am using a script to scrape news from many websites using newspaper3k. Instead of running it sequentially I tried to utilize all of my cores by using joblib.Parallel However, it still takes A LOT of time (50 websites take around 20 minutes). I…
Dr. Prof. Patrick
  • 1,280
  • 2
  • 15
  • 27
1
vote
1 answer

Running the same function multiple times concurrently

So what I want to do is run the same function multiple times simultaneously while getting a result as return and storing it in an array or list. It goes like this: def base_func(matrix,arg1,arg2): result = [] for row in…
Subham Dutta
  • 11
  • 1
1
vote
1 answer

Parallelization of Python code on different machines on different networks

I’m looking to use parallelized code across two computers on different networks to execute a batch of tasks, but am not sure how to do so in Python. Suppose I have two computers, Computer A and Computer B on two different networks, and I have a…
user100000000000000
  • 113
  • 4
1
vote
0 answers

Processing 8 python processes using windows PoweShell parallel {}-section

I have a python script that I need to run 8 times, with 8 different .txt-files. I'm trying to do this with a parallel{}-section in PowerShell. workflow my_script{ parallel { python bin/script.py config/file1.txt python bin/script.py…
Leonardo Sanna
  • 57
  • 7
1
vote
1 answer

Python Multiprocessing: efficiently only save the best runs

I read a lot of posts about parallelization using the multiprocessing module but none of them quite answered my question. I have a very long generator giving me parameter values and for each I want to compute some function value. However, I only…
Laaag
  • 13
  • 2
1
vote
1 answer

Python concurrent.futures trying to import functions

So I got 2 .py files and am trying to import the test function from the first to the secon one. But every time I try that I just get a "BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or…
BrauseMann
  • 11
  • 1
1
vote
3 answers

Safe way to terminate a python subprocess?

I'm using python's multiprocessing module to handle a number of functions concurrently. Each spawned-process' function gets some initial input arguments, and a Pipe connection to send its results back. For various reasons, I must use individual…
bjarkemoensted
  • 2,557
  • 3
  • 24
  • 37
1
vote
0 answers

Multithreading in python: how to hold the code inside the for loop and limit 20 threads in one go

I am working on a project for Algo trading using zerodha broker's API. I am trying to do multithreading to save the costly operation of calling the API function for getting historical data for 50 stocks at a time and then apply my strategy on it for…
Technoder
  • 25
  • 3
1
2
3 4 5