NSOperation deadlocks and blocks NSOperationQueue

Question

I use a subclass of NSOperation to upload large files to AWS S3 using Amazon's iOS SDK (v1.3.2). This all works fine, but some beta testers experience deadlocks (iOS 5.1.1). The result is that the NSOperationQueue in which the operations are scheduled is blocked as only one operation is allowed to run at one time. The problem is that I cannot reproduce the issue whereas the beta testers experience this problem every single time.

The operation is quite complex due to how the AWS iOS SDK works. However, the problem is not related to the AWS iOS SDK as far as I know based on my testing. The operation's main method is pasted below. The idea of the operation's main method is based on this Stack Overflow question.

- (void)main {
    // Operation Should Terminate
    _operationShouldTerminate = NO;

    // Notify Delegate
    dispatch_async(dispatch_get_main_queue(), ^{
        [self.delegate operation:self isPreparingUploadWithUuid:self.uuid];
    });

    // Increment Network Activity Count
    [self incrementNetworkActivityCount];

    // Verify S3 Credentials
    [self verifyS3Credentials];

    while (!_operationShouldTerminate) {
        if ([self isCancelled]) {
            _operationShouldTerminate = YES;

        } else {
            // Create Run Loop
            [[NSRunLoop currentRunLoop] runMode:NSDefaultRunLoopMode beforeDate:[NSDate distantFuture]];
        }
    }

    // Decrement Network Activity Count
    [self decrementNetworkActivityCount];

    NSLog(@"Operation Will Terminate");
}

The method that finalizes the multipart upload sets the boolean _operationShouldTerminate to YES to terminate the operation. That method looks like this.

- (void)finalizeMultipartUpload {
    // Notify Delegate
    dispatch_async(dispatch_get_main_queue(), ^{
        [self.delegate operation:self didFinishUploadingUploadWithUuid:self.uuid];
    });

    // Operation Should Terminate
    _operationShouldTerminate = YES;

    NSLog(@"Finalize Multipart Upload");
}

The final log statement is printed to the console, but the while loop in the main method does not seem to exit as the final log statement in the operation's main method is not printed to the console. As a result, the operation queue in which the operation is scheduled, is blocked and any scheduled operations are not executed as a result.

The operation's isFinished method simply returns _operationShouldTerminate as seen below.

- (BOOL)isFinished {
    return _operationShouldTerminate;
}

It is odd that the while loop is not exited and it is even more odd that it does not happen on any of my own test devices (iPhone 3GS, iPad 1, and iPad 3). Any help or pointers are much appreciated.

score 2 · Accepted Answer · answered Sep 12 '12 at 10:04

The solution to the problem is both complex and simple as it turns out. What I wrongly assumed was that the methods and delegate callbacks of the operation were executed on the same thread, that is, the thread on which the operation's main method was called. This is not always the case.

Even though this was true in my test and on my devices (iPhone 3GS), which is why I did not experience the problem myself. My beta testers, however, used devices with multicore processors (iPhone 4/4S), which caused some of the code to be executed on a thread different from the thread on which the operation's main method was invoked.

The result of this is that _operationShouldTerminate was modified in the finalizeMultipartUpload method on the wrong thread. This in turn means that the while loop of the main method was not exited properly resulting in the operation deadlocking.

In short, the solution is to update _operationShouldTerminate on the same thread as the main method was invoked on. This will properly exit the while loop and exit the operation.

score 1 · Answer 2 · answered Sep 11 '12 at 18:00

There are a number of problems with your code, and I can offer two solutions:

1) read up on Concurrent NSOperations in Apple's Concurrency Programming Guide. To keep the runLoop "alive" you have to add either a port or schedule a timer. The main loop should contain a autorelease pool as you may not get one (see Memory Management in that same memo). You need to implement KVO to let the operationQueue know when your operation is finished.

2) Or, you can adopt a small amount of field tested hardened code and reuse it. That Xcode project contains three classes of interest to you: a ConcurrentOperation file that does well what you are trying to accomplish above. The Webfetcher.m class shows how to subclass the concurrent operation to perform an asynchronous URL fetch from the web. And the OperationsRunner is a small helper file you can add to any kind of class to manage the operations queue (run, cancel, query, etc). All of the above are less than 100 lines of code, and provide a base for you to get your code working. The OperationsRunner.h file provide a "how to do" too.

Thank you for your answer David. The Concurrency Programming Guide is not new to me and is indeed a useful guide for this type of problem. Keeping the run loop alive does not seem to be the problem in this situation. — Bart Jacobs, Sep 12 '12 at 09:55

NSOperation deadlocks and blocks NSOperationQueue

2 Answers2