77

I have the following shell script. The issue is that I want to run the transactions parallel/concurrently without waiting for one request to finish to go to the next request. For example if I make 20 requests, I want them to be executed at the same time.

for ((request=1;request<=20;request++))
do
    for ((x=1;x<=20;x++))
    do
        time curl -X POST --header "http://localhost:5000/example"
    done
done

Any guide?

8 Answers8

90

You can use xargs with -P option to run any command in parallel:

seq 1 200 | xargs -n1 -P10  curl "http://localhost:5000/example"

This will run curl command 200 times with max 10 jobs in parallel.

Saeed Mohtasham
  • 1,693
  • 16
  • 27
  • 6
    For me it connects to 0.0.0.1, 0.0.0.2 and so on as well – Madhur Ahuja Jun 01 '20 at 15:53
  • 2
    This works for me but for some reason it doesn't exit out of the command when it's done, and holds all the connections open (using `netstat -an | wc -l` to see open connections) Any reason it does this? Is there way to get it to exit at the end? – thinktt Dec 07 '20 at 22:45
  • 16
    Yes it appends the number at end of the curl command. Simple fix is this `seq 1 200 | xargs -Iname -P10 curl "http://localhost:5000/example"` Using _I_ parameter we specify placeholder for argument, then we omit the argument in the command invocation – dark knight Aug 11 '21 at 06:17
68

Using xargs -P option, you can run any command in parallel:

xargs -I % -P 8 curl -X POST --header "http://localhost:5000/example" \
< <(printf '%s\n' {1..400})

This will run give curl command 400 times with max 8 jobs in parallel.

anubhava
  • 761,203
  • 64
  • 569
  • 643
  • 1
    `-P` runs given number of processes in parallel. `printf '%s\n' {1..400}` will print numbers from 1 to 400 and `curl` will run total 400 times with max 8 jobs in parallel. – anubhava Jul 05 '19 at 13:45
  • Is it possible to print each output in a new line? – anoopelias May 21 '20 at 02:02
  • 3
    See this answer: https://stackoverflow.com/a/61249019/548225 you can run a bash script in `xargs` and can do anything there – anubhava May 21 '20 at 18:11
  • The command is not working. Try to copy it to the command line and execute. It'll return `bash: 0: ambiguous redirect` – Max Feb 13 '22 at 08:49
  • Of course it works without any error on bash 4 or 5 versions. On older bash versions use: `printf '%s\n' {1..400} | xargs -I % -P 8 curl -I -X POST 'http://localhost:5000/example'` – anubhava Feb 13 '22 at 10:18
  • 2
    I thought that reusing same TCP connection will be faster so `curl --parallel` should be faster than `xargs -P n`, But I found `xargs` is faster with `connection: close` for each request! May you describe Why please? Does `curl` see the parallelized jobs and reuse the same connection? – Mamdouh Saeed May 14 '23 at 07:25
28

Update 2020:

Curl can now fetch several websites in parallel:

curl --parallel --parallel-immediate --parallel-max 3 --config websites.txt

websites.txt file:

url = "website1.com"
url = "website2.com"
url = "website3.com"
Sergey Geron
  • 9,098
  • 2
  • 22
  • 29
  • 3
    How does a POST data work in this case? – Josh Dec 16 '21 at 05:45
  • 2
    @Josh You can send multiple POST requests like this: `curl -i --parallel --parallel-immediate --parallel-max 3 -X POST -H 'Content-Type: application/json' -d '{"x":100}' --config websites.txt` – László Kenéz Feb 02 '22 at 16:48
  • what about running x thousands request in parallel ...its curl capable of it ? – Morty May 04 '23 at 13:37
  • How do you send multiple data through same tcp connection and same url? I know about `--next` but I feel stupid repeating same long url, `-H` values with diffrent `-d` values which leads to error `Arguments list too long` – Mamdouh Saeed May 10 '23 at 19:49
  • You can add the `-L` and `--remote-name-all` options if you want curl to grab remote filenames after following locations. – Tom Nguyen Aug 28 '23 at 10:48
12

This is an addition to @saeed's answer.

I faced an issue where it made unnecessary requests to the following hosts

0.0.0.1, 0.0.0.2 .... 0.0.0.N

The reason was the command xargs was passing arguments to the curl command. In order to prevent the passing of arguments, we can specify which character to replace the argument by using the -I flag.

So we will use it as,

 ... xargs -I '$' command ...

Now, xargs will replace the argument wherever the $ literal is found. And if it is not found the argument is not passed. So using this the final command will be.

seq 1 200 | xargs -I $ -n1 -P10  curl "http://localhost:5000/example"

Note: If you are using $ in your command try to replace it with some other character that is not being used.

Subesh Bhandari
  • 1,062
  • 1
  • 10
  • 21
7

Adding to @saeed's answer, I created a generic function that utilises function arguments to fire commands for a total of N times in M jobs at a parallel

function conc(){
    cmd=("${@:3}")
    seq 1 "$1" | xargs -n1 -P"$2" "${cmd[@]}"
}

$ conc N M cmd
$ conc 10 2 curl --location --request GET 'http://google.com/'

This will fire 10 curl commands at a max parallelism of two each.

Adding this function to the bash_profile.rc makes it easier. Gist

isopropylcyanide
  • 423
  • 4
  • 16
  • 1
    Thanks, very convenient. One note: xargs will pass the index read from `seq` to the command. E.g. `conc 2 2 echo test` will print `test 1\ntest 2`. To avoid that, using `-I'$XARGI'` instead of `-n1` works (and you can use $XARGI in your command then if you need an index, or leave it out if you don't). – Dario Seidl Dec 31 '21 at 15:35
2

Add “wait” at the end, and background them.

for ((request=1;request<=20;request++))
do
    for ((x=1;x<=20;x++))
    do
        time curl -X POST --header "http://localhost:5000/example" &
    done
done

wait

They will all output to the same stdout, but you can redirect the result of the time (and stdout and stderr) to a named file:

time curl -X POST --header "http://localhost:5000/example" > output.${x}.${request}.out 2>1 &
Gaétan RYCKEBOER
  • 305
  • 1
  • 8
  • I tried this but it was giving me other issues. This is helpful though. Let do also the redirection of time to a file. –  Sep 22 '17 at 10:31
  • The "output to file" issue is tricky, and may have to be sent to a synchronized buffer (via GNU parallel), named pipes, or flock. – Gaétan RYCKEBOER May 15 '18 at 07:06
2

Wanted to share my example how I utilised parallel xargs with curl.

The pros from using xargs that u can specify how many threads will be used to parallelise curl rather than using curl with "&" that will schedule all let's say 10000 curls simultaneously.

Hope it will be helpful to smdy:

#!/bin/sh

url=/any-url
currentDate=$(date +%Y-%m-%d)
payload='{"field1":"value1", "field2":{},"timestamp":"'$currentDate'"}'
threadCount=10

cat $1 | \
xargs -P $threadCount -I {} curl -sw 'url= %{url_effective}, http_status_code = %{http_code},time_total = %{time_total} seconds \n' -H "Content-Type: application/json" -H "Accept: application/json" -X POST $url --max-time 60 -d $payload

.csv file has 1 value per row that will be inserted in json payload

Dzmitry Hubin
  • 1,091
  • 12
  • 14
0

Based on the solution provided by @isopropylcyanide and the comment by @Dario Seidl, I find this to be the best response as it handles both curl and httpie.

# conc N M cmd - fire (N) commands at a max parallelism of (M) each
function conc(){
    cmd=("${@:3}")
    seq 1 "$1" | xargs -I'$XARGI' -P"$2" "${cmd[@]}"
}

For example:

conc 10 3 curl -L -X POST https://httpbin.org/post -H 'Authorization: Basic dXNlcjpwYXNz' -H 'Content-Type: application/json' -d '{"url":"http://google.com/","foo":"bar"}'

conc 10 3 http --ignore-stdin -F -a user:pass httpbin.org/post url=http://google.com/ foo=bar
h q
  • 1,168
  • 2
  • 10
  • 23