Questions tagged [embarrassingly-parallel]

An embarrassingly parallel problem is one for which little or no effort is required to separate the problem into a number of parallel tasks. This is often the case where there exists no dependency (or communication) between those parallel tasks.

An embarrassingly parallel problem is one for which little or no effort is required to separate the problem into a number of parallel tasks. This is often the case where there exists no dependency (or communication) between those parallel tasks.

These problems tend to require little or no communication of results between tasks, and are thus different from distributed computing problems that require communication between tasks, especially communication of intermediate results. They are easy to perform on server farms which do not have any of the special infrastructure used in a true supercomputer cluster.

29 questions
1
vote
1 answer

SLURM Embarrasingly parrallel submission taking too many resources

so I have the following submission script: #!/bin/bash # #SBATCH --job-name=P6 #SBATCH --output=P6.txt #SBATCH --partition=workq #SBATCH --ntasks=512 #SBATCH --time=18:00:00 #SBATCH --mem-per-cpu=2500 #SBATCH --cpus-per-task=1 #SBATCH…
1
vote
1 answer

Disadvantage of GPUs in embarrassingly parallel

What are the disadvantages of using GPU for embarrassingly parallel programs?
S.Dan
  • 1,826
  • 5
  • 28
  • 55
1
vote
1 answer

"Max jobs to run" does not equal the number of jobs specified when using GNU Parallel on remote server?

I am trying to run many small serial jobs with GNU Parallel on a PBS cluster, each compute node has 16 cores, as I intended to use multiple compute nodes therefore I passed the option -S $SERVERNAME to GNUParallel, however what confuses me is that…
0
votes
1 answer

OpenMP with matrices and vectors

What is the best way to utilize OpenMP with a matrix-vector product? Would the for directive suffice (if so, where should I place it? I assume outer loop would be more efficient) or would I need schedule, etc..? Also, how would I take advantage…
user990246
0
votes
1 answer

How to parallel for loop in Sagemaker Processing job

I'm running a python code on Sagemaker Processing job, specifically SKLearnProcessor. The code run a for-loop for 200 times (each iteration is independent), each time takes 20 minutes. for example: script.py for i in list: run_function(i) I'm…
0
votes
1 answer

Embarrassingly parallel workloads with chunking using dask

I am trying to use dask for an embarrassingly parallel workload. Example would be, sum two large arrays element-by-element: x1 = ... x2 = ... def func(a, b): sleep(0.001 + len(a) * 1e-5) return a + b + 1 I found [1] which seemingly answers…
user3384414
  • 63
  • 2
  • 8
0
votes
0 answers

Running for loop in parallel via python

I have a process that loops over a list of IP addresses and returns some information about them. The simple for loop works great, my issue is running this at scale due to Python's Global Interpreter lock (GIL). My goal is to have this function run…
0
votes
1 answer

Python - why does this code take so long?

My concern is the part between the two hash-lines. The following code runs too long for me to wait for its output. When I replace the problematic part by another chunk of code, the programme runs in a few seconds (see the end of this post). my aim…
0
votes
0 answers

How to generate a different set of random numbers in each iteration of a prallelized For loop?

The following problem arised directly due to applying the answer to this question. In the minimal working example (MWE) there's a place in the myscript definition where I generate some random numbers, then perform some operations on them, and fnally…
0
votes
1 answer

How to parallelize this piece of code?

I've been browsing for some time but couldn't find any constructive answer that I could comprehend. How should I paralellize the following code: import random import math import numpy as np import sys import multiprocessing boot = 20#number of…
0
votes
1 answer

Low performance for an Embarrassingly parallel code

I have this very simple parallel code that I am using to learn openmp which is embarrassingly parallel. However, I don't get the superlinear or at least linear performance increase expected. #pragma omp parallel num_threads(cores) { int id =…
0
votes
2 answers

How can I scale an embarrassingly parallel jobs across multiple computers in .Net?

I have an embarrassingly parallel problem I want to scale across multiple computers using .Net. I’m currently using the Task Parallel Library to scale across multiple cores on a single computer but I want to scale further. How can I do this?
0
votes
2 answers

opencl branching vs memory redundancy

I'm processing items in a grid, depending on the item's type a different type of computation/function needs to be performed. But I've read branching is a very bad thing to do between workitems doing the same thing. To circumvent this I could split…
-1
votes
0 answers

PySpark running embarassingly parallel jobs

So I am trying to run 1000+ embarassingly parallel jobs using PySpark on a cluster. I instantiate 5 executors, each having 20 cores, i.e. should be able to execute 100 jobs concurrently, as far as I understand. This is what I have so far, "values"…
1
2