downloading data from several pffile's at once asynchronosly

Question

If I have an array of Message objects, each with a PFile containing data, is it possible to download the data for every single message by queuing them up asynchronously like so:

for (int i = 0; i < _downloadedMessages.count; i++) {
    PFObject *tempMessage = (PFObject *)[_downloadedMessages objectAtIndex:i];    
    [[tempMessage objectForKey:@"audio"] getDataInBackgroundWithBlock:^(NSData *data, NSError *error) {
            [self persistNewMessageWithData:data];
    }];
}

This seems to cause my app to hang, even though this should be done in the background...

Using the solution below:

NSMutableArray* Objects = ...

[self forEachPFFileInArray:Objects retrieveDataWithCompletion:^BOOL(NSData* data, NSError*error){
    if (data) {
        PFObject *tempObj = (PFObject *)Object[someIndex...];
        [self persistNewMessageWithData:data andOtherInformationFromObject:tempObj];
        return YES;
    }
    else {
         NSLog(@"Error: %@", error);
         return NO; // stop iteration, optionally continue anyway
    }
} completion:^(id result){
     NSLog(@"Loop finished with result: %@", result);    
}];

What kind of object is the result of `[tempMessage objectForKey:@"audio"]`, a subclass of `NSOperation`? — CouchDeveloper, Mar 08 '14 at 09:47

CouchDeveloper · Accepted Answer · 2014-03-10T08:30:37.583

What you are possibly experiencing is, that for a large numbers of asynchronous requests which run concurrently, the system can choke due to memory pressure and due to network stalls or other accesses of resources that get exhausted (including CPU).

You can verify the occurrence of memory pressure using Instruments with the "Allocations" tool.

Internally (that is, in the Parse library and the system) there might be a variable set which sets the maximum number of network requests which can run concurrently. Nonetheless, in your for loop you enqueue ALL requests.

Depending of what enqueuing a request means in your case, this procedure isn't free at all. It may cost a significant amount of memory. In the worst case, the network request will be enqueued by the system, but the underlying network stack executes only a maximum number of concurrent requests. The other enqueued but pending requests hang there and wait for execution, while their network timeout is already running. This may lead to cancellation of pending events, since their timeout expired.

The simplest Solution

Well, the most obvious approach solving the above issues would be one which simply serializes all tasks. That is, it only starts the next asynchronous task when the previous has been finished (including the code in your completion handler). One can accomplish this using an asynchronous pattern which I name "asynchronous loop":

The "asynchronous loop" is asynchronous, and thus has a completion handler, which gets called when all iterations are finished.

typedef void (^loop_completion_handler_t)(id result);
typedef BOOL (^task_completion_t)(PFObject* object, NSData* data, NSError* error);

- (void) forEachObjectInArray:(NSMutableArray*) array 
   retrieveDataWithCompletion:(task_completion_t)taskCompletionHandler
   completion:(loop_completion_handler_t)completionHandler 
{
    // first, check termination condition:
    if ([array count] == 0) {
        if (completionHandler) {
            completionHandler(@"Finished");
        }
        return;
    }
    // handle current item:
    PFObject* object = array[0];
    [array removeObjectAtIndex:0];
    PFFile* file = [object objectForKey:@"audio"];
    if (file==nil) {
        if (taskCompletionHandler) {
            NSDictionary* userInfo = @{NSLocalizedFailureReasonErrorKey: @"file object is nil"}
            NSError* error = [[NSError alloc] initWithDomain:@"RetrieveObject"
                                                        code:-1 
                                                    userInfo:userInfo]; 
            if (taskCompletionHandler(object, nil, error)) {                    
                // dispatch asynchronously, thus invoking itself is not a recursion
                dispatch_async(dispatch_get_global(0,0), ^{ 
                    [self forEachObjectInArray:array 
                    retrieveDataWithCompletion:taskCompletionHandler
                             completionHandler:completionHandler];
                });
            }
            else {
                if (completionHandler) {
                    completionHandler(@"Interuppted");
                }
            }
        }
    }
    else {
        [file getDataInBackgroundWithBlock:^(NSData *data, NSError *error) {
            BOOL doContinue = YES;
            if (taskCompletionHandler) {
                doContinue = taskCompletionHandler(object, data, error);
            }
            if (doContinue) {
                // invoke itself (note this is not a recursion")
                [self forEachObjectInArray:array 
                retrieveDataWithCompletion:taskCompletionHandler
                         completionHandler:completionHandler];
            }
            else {
                if (completionHandler) {
                    completionHandler(@"Interuppted");
                }
            }
        }];
    }
}

Usage:

// Create a mutable array 
NSMutableArray* objects = [_downloadedMessages mutableCopy];

[self forEachObjectInArray:objects 
retrieveDataWithCompletion:^BOOL(PFObject* object, NSData* data, NSError* error){
    if (error == nil) {
        [self persistNewMessageWithData:data andOtherInformationFromObject:object];
        return YES;
    }
    else {
         NSLog(@"Error %@\nfor PFObject %@ with data: %@", error, object, data);
         return NO; // stop iteration, optionally continue anyway
    }
} completion:^(id result){
     NSLog(@"Loop finished with result: %@", result);    
}];

each index in "files" is actually a PFObject which contains a PFFile as a field. How would I maintain the PFObject in "array" when I call [self forEachPFFileInArray:array ... ] if I remove the object before the index is returned. I've edited my question above to be more clear... — Apollo, Mar 08 '14 at 16:27
and thank you so much for such a complete answer. This is extremely helpful. — Apollo, Mar 08 '14 at 16:28
@Auser In my answer, I just made the assumption, that you have an extra mutable array of PFFile objects. You can easily create this mutable array, taking your original array containing PFObject objects which have this field which is this PFFile object, see updated answer. Other note: self in the sample code above is possibly a View Controller or some "Data Manager" helper class. — CouchDeveloper, Mar 09 '14 at 09:12
sorry, my question is actually this: if my method persistNewMessageWithData: takes a PFObject as a parameter, does it make more sense to call my persistNewMessageWithData: from the method call (where you currently say self persistNewMessageWithData:data in your answer) OR instead put that method call where you say // invoke self. I'm basically wondering where I would still have access to the PFObject and the object associated with a given retrieved data. — Apollo, Mar 09 '14 at 15:27
@Auser You can of course make a mutable copy of your original array, where everything is contained. Then, pass it through the modified version, e.g.: `[self forEachPFObjectInArray:mutableCopyOfMessages retrieveDataWithTaskCompletion:^BOOL(PFObject* object, NSData* data, NSError*error){ /*here, you got the original PFObject object AND the resulting data of the async task*/} completion:onFinalCompletion]`. If you have trouble adjusting the sample for this modification, please ask. — CouchDeveloper, Mar 10 '14 at 07:58

downloading data from several pffile's at once asynchronosly

1 Answers1

The simplest Solution

Usage:

Linked