downloading with curl using multiple connections

Question

I'm trying to tranfer a multi-gb file from one server to another; problem is RTT is 150ms+. Ive already tried using aria2 but its limited to 16 connections, lftp doesn't have any protection against stalled transfers.

I'm wondering if its possible to download one file with multiple connections using curl cli.

score 8 · Answer 1 · edited Jun 20 '20 at 09:12

8

It's possible. Fetch the total file size with -I option in curl.

Then you can fork many process in a shell, every curl connection with a different Content-Length header to download different part of the file.

After all the tasks finish, then merge all the download slices to a big file.

I have written a simple script and it's available here mcurl.sh, with -s option you can specify how many tasks you create to download the big file.

edited Jun 20 '20 at 09:12

Community

1
1

answered Nov 25 '16 at 14:42

alijandro

11,627
2
58
74

You meant different `Range:` headers, right? That's what curl uses when you ask it to download just a "range" of the remote resource. – Daniel Stenberg Aug 11 '21 at 14:29
Linked script does not work correctly (tested with some small text file). – Kokos Aug 12 '21 at 10:42

Daniel Stenberg · Answer 2 · 2021-08-11T14:27:37.443

1

No, the curl tool has no such ability built-in. To do it with curl, you need to invoke it multiple times with range downloads.

(oh and btw, a large RTT is very rarely the explanation to why a plain TCP transfer is slow)

edited Aug 11 '21 at 14:27

answered Dec 05 '11 at 11:59

Daniel Stenberg

54,736
17
146
222

downloading with curl using multiple connections

2 Answers2

Linked