Waiting for multiple asynchronous download tasks

Question

I want to download some files, for example 100 files, at the same time. So I decided to add my download threads to a dispatch queue, and GCD will adjust how many threads will run at the same time.

The problem here is: the block in dispatch_async will be completed immediately, because task will run on another thread. So, if urls's length is 100, it will create 100 threads immediately.

var queueDownloadTask = dispatch_queue_create("downloadQueue", nil)

for url in urls {
    dispatch_async(queueDownloadTask) {

        let config = NSURLSessionConfiguration.defaultSessionConfiguration()
        let fileTransferSession = NSURLSession(configuration: config)

        let task = fileTransferSession.downloadTaskWithURL(url, completionHandler: { (responseUrl, response, error) -> Void in
            println("completed")
        })
        task.resume()
    }
}

How can I configure the block in dispatch_async to wait for the download task to complete? I don't want use dispatch_semaphore, because it only allow run one download task at the same time.

Answer 1

In Swift 4,

func executeMultiTask() {
     //1. Create group
     let taskGroup = DispatchGroup()

     //2. Enter group
     taskGroup.enter()
     myTask1.execute(completeHandler: {
         // ...
         //3. Leave group
         taskGroup.leave() //< balance with taskGroup.enter()
     })

     /* Add more tasks ...
     //2. Enter group
     taskGroup.enter()
     myTask2.execute(completeHandler: {
         //3. Leave group
         defer {
            // Use `defer` to make sure, `leave()` calls are balanced with `enter()`.
            taskGroup.leave() 
         }
         // ... more
     })
     */

     //4. Notify when all task completed at main thread queue.
     taskGroup.notify(queue: .main) { 
         // All tasks are done.
         // ...   
     }

}

Answer 2

To expand on Abhinav's answer, you should:

1. Use dispatch_group_create() to create a group.
2. Call dispatch_group_enter(group) before starting each download task.
3. Call dispatch_group_leave(group) inside the task's completion handler.
4. Then call dispatch_group_notify(group, queue, ^{ ... }) to enqueue a block that will be executed when all the tasks are completed.

You can see an example in this post.

(By the way, it's not true that doing 100 dispatch_asyncs in a row will create 100 threads immediately. The system still retains control over how many threads to use to satisfy the queue. However, your code does not wait for any of the tasks to complete before it returns, nor does it attempt to synchronize between multiple tasks completing.)

Answer 3

A working objective-c example for downloading multi files

- (void) downloadFiles: (NSMutableArray *) fileArray : (NSString *)destParentDir{
    dispatch_group_t serviceGroup = dispatch_group_create();

    for (id fileInfo in fileArray) {
        dispatch_group_enter(serviceGroup);

        NSFileManager *fileManager = [NSFileManager defaultManager];
        NSString *fileName = [fileInfo valueForKey:@"name"];

        //Create SubDirs if needed, note that you need to exclude file name for the dirsString :)
        //[fileManager createDirectoryAtPath:dirsString withIntermediateDirectories:true attributes:nil error:NULL];

        //Download file
        NSURL  *url = [NSURL URLWithString:@"YOUR_FILE_URL"];
        NSData *urlData = [NSData dataWithContentsOfURL:url];
        if(urlData)
        {
            NSString  *localPath = [NSString stringWithFormat:@"%@/%@", destParentDir, fileName];
            [urlData writeToFile:localPath atomically:YES];
        }
        dispatch_group_leave(serviceGroup);
    }

    dispatch_group_notify(serviceGroup, dispatch_get_main_queue(),^{
        NSLog(@"Complete files download");
    });
}

Answer 4

You should use dispatch_group_t. Please refer to Apple documentation to solutionize your case.

Answer 5

Create a DispatchGroup. Wrap each download in enter and leave. Provide a callback to notify, which will be called when all pairings have completed.

var dispatchGroup = DispatchGroup()

for url in urls {
    // ... Set up the download
    dispatchGroup.enter()
    let task = fileTransferSession.downloadTaskWithURL(url) { (responseUrl, response, error) -> Void in
        println("Individual download complete.")
        dispatchGroup.leave()
     })
    task.resume()

    dispatchGroup.notify(queue: .main) {
        println("All downloads complete.")
    }
}