run CURL from C++ to asyncronously fetch several URLs

Question

One logical part of my C++ program has to fetch text files calling curl (linux program, not a library) and getting its output via pipe. There may be many urls to download. And download time may be reasonable. The program is meant to be "scalable" and effective.

So the question is what variant is preferable:

Run single instance of curl giving it a list of urls
Create a pool of threads each calling a curl process program with single url and subsequent aggregation of thread outputs.

In other words is curl asynchronous when downloading several urls or I have to implement it manually.

Do you want to use the `curl` program, or the libcurl library ? — , Jul 11 '15 at 16:01
duplicate http://stackoverflow.com/questions/8378899/downloading-with-curl-using-multiple-connections — perencia, Jul 11 '15 at 16:30

score 0 · Answer 1 · answered Jul 12 '15 at 20:45

If you are calling cURL from pipes then the preferred solution would be creating a pool of threads. Since the threads are independent from each other, they fetch multiple URLs at once.

However, another efficient solution would be to use the cURL library instead of pipes. Here is an example from libcurl.

run CURL from C++ to asyncronously fetch several URLs

1 Answers1