Memory issues when encrypting/decrypting a large file with RNCryptor on iOS

Question

I'm trying to use RNCryptor to encrypt and decrypt large files (600+MB) on iOS. On the github I found example code on how to use the library asynchronously on streams. This code is similar to the answer of Rob Napier on a question about this same subject.

However, although I think I implemented the code correctly, the app uses up to 1.5 GB of memory (in the iPad 6.1 simulator). I thought the code was supposed to prevent the app from keeping more than one block of data in-memory? So what is going wrong?

In my controller, I create a 'CryptController' which I message with encrypt/decrypt requests.

  // Controller.m
  NSString *password = @"pw123";
  self.cryptor = [[CryptController alloc] initWithPassword:password];

  //start encrypting file
  [self.cryptor streamEncryptRequest:self.fileName andExtension:@"pdf" withURL:[self samplesURL]];

  //wait for encryption to finish
  NSDate *timeout = [NSDate dateWithTimeIntervalSinceNow:1];
  do {
    [[NSRunLoop currentRunLoop] runMode:NSDefaultRunLoopMode
                             beforeDate:timeout];
  } while (![self.cryptor isFinished]);

In CryptController I have:

- (void)streamEncryptionDidFinish {
  if (self.cryptor.error) {
    NSLog(@"An error occurred. You cannot trust decryptedData at this point");
  }
  else {
    NSLog(@"%@ is complete. Use it as you like", [self.tempURL lastPathComponent]);
  }
  self.cryptor = nil;
  self.isFinished = YES;
}

- (void) streamEncryptRequest:(NSString *)fileName andExtension:(NSString *)ext withURL:(NSURL *)directory {

  //Make sure that this number is larger than the header + 1 block.
  int blockSize = 32 * 1024;

  NSString *encryptedFileName = [NSString stringWithFormat:@"streamEnc_%@", fileName];
  self.tempURL = [[NSFileManager defaultManager] URLForDirectory:NSDocumentDirectory inDomain:NSUserDomainMask appropriateForURL:nil create:NO error:nil];
  self.tempURL = [self.tempURL URLByAppendingPathComponent:encryptedFileName isDirectory:NO];
  self.tempURL = [self.tempURL URLByAppendingPathExtension:@"crypt"];

  NSInputStream *decryptedStream = [NSInputStream inputStreamWithURL:[[directory URLByAppendingPathComponent:fileName isDirectory:NO] URLByAppendingPathExtension:ext]];
  NSOutputStream *cryptedStream = [NSOutputStream outputStreamWithURL:self.tempURL append:NO];

  [cryptedStream open];
  [decryptedStream open];

  __block NSMutableData *data = [NSMutableData dataWithLength:blockSize];
  __block RNEncryptor *encryptor = nil;

  dispatch_block_t readStreamBlock = ^{
    [data setLength:blockSize];
    NSInteger bytesRead = [decryptedStream read:[data mutableBytes] maxLength:blockSize];
    if (bytesRead < 0) {
      //Throw an error
    }
    else if (bytesRead == 0) {
      [encryptor finish];
    }
    else {
      [data setLength:bytesRead];
      [encryptor addData:data];
      //NSLog(@"Sent %ld bytes to encryptor", (unsigned long)bytesRead);
    }
  };

  encryptor = [[RNEncryptor alloc] initWithSettings:kRNCryptorAES256Settings
                                           password:self.password
                                            handler:^(RNCryptor *cryptor, NSData *data) {
                                              //NSLog(@"Encryptor received %ld bytes", (unsigned long)data.length);
                                              [cryptedStream write:data.bytes maxLength:data.length];
                                              if (cryptor.isFinished) {
                                                [decryptedStream close];
                                                //call my delegate that i'm finished with decrypting
                                                [self streamEncryptionDidFinish];
                                              }
                                              else {
                                                readStreamBlock();
                                              }
                                            }];

  // Read the first block to kick things off
  self.isFinished = NO;
  readStreamBlock();
}

When I profile using the Allocation Instrument, the allocation categories I see consistently growing are malloc 32.50 KB, malloc 4.00 KB, NSConcreteData and NSSubrangeData. Especially the malloc 32.50 KB grows big, over 1 GB. The responsible caller is [NSConcreteData initWithBytes:length:copy:freeWhenDone:bytesAreVM:] For NSConcreteData the responsible caller is -[NSData(NSData) copyWithZone:].

When I profile using the Leaks Instrument, no leaks are found.

I'm new to Objective-C, and from what I understood, the new ARC is supposed to handle allocation and deallocation of memory. When googling on anything memory related, all the information I find is assuming you don't use ARC (or it didn't exist at time of writing). I sure am using ARC, since I get compile errors saying so when I try to manually deallocate memory.

If anyone can help me with this, it would be greatly appreciated! If any more information is needed, I'll be happy to provide it :) Also, I'm new to StackOverflow, so if there's anything I've overlooked that I should have done, kindly inform me!

Answer 1

I finally tried the solution given here, which uses semaphores instead of depending on the callback to wait for the stream. This works perfectly :) The memory usage hovers around 1.1 MB according to the Allocations Instrument. It may not look as neat because of the semaphore syntax, but at least it does what I need it to do.

Other suggestions are still welcome of course :)

- (void)encryptWithSemaphore:(NSURL *)url {
  dispatch_semaphore_t semaphore = dispatch_semaphore_create(0);

  __block int total = 0;
  int blockSize = 32 * 1024;

  NSString *encryptedFile = [[url lastPathComponent] stringByDeletingPathExtension];
  NSURL *docsURL = [[NSFileManager defaultManager] URLForDirectory:NSDocumentDirectory inDomain:NSUserDomainMask appropriateForURL:nil create:NO error:nil];
  self.tempURL = [[docsURL URLByAppendingPathComponent:encryptedFile isDirectory:NO] URLByAppendingPathExtension:@"crypt"];

  NSInputStream *inputStream = [NSInputStream inputStreamWithURL:url];
  __block NSOutputStream *outputStream = [NSOutputStream outputStreamWithURL:self.tempURL append:NO];
  __block NSError *encryptionError = nil;

  [inputStream open];
  [outputStream open];

  RNEncryptor *encryptor = [[RNEncryptor alloc] initWithSettings:kRNCryptorAES256Settings
                                                        password:self.password
                                                         handler:^(RNCryptor *cryptor, NSData *data) {
                                                           @autoreleasepool {
                                                             [outputStream write:data.bytes maxLength:data.length];
                                                             dispatch_semaphore_signal(semaphore);

                                                             data = nil;
                                                             if (cryptor.isFinished) {
                                                               [outputStream close];
                                                               encryptionError = cryptor.error;
                                                               // call my delegate that I'm finished with decrypting
                                                             }
                                                           }
                                                         }];
  while (inputStream.hasBytesAvailable) {
    @autoreleasepool {
      uint8_t buf[blockSize];
      NSUInteger bytesRead = [inputStream read:buf maxLength:blockSize];
      if (bytesRead > 0) {
        NSData *data = [NSData dataWithBytes:buf length:bytesRead];

        total = total + bytesRead;
        [encryptor addData:data];

        dispatch_semaphore_wait(semaphore, DISPATCH_TIME_FOREVER);
      }
    }
  }

  [inputStream close];
  [encryptor finish];  
}

Answer 2

Just run:

self.cryptorQueue = dispatch_queue_create([queueName cStringUsingEncoding:NSUTF8StringEncoding], NULL);

dispatch_async(self.cryptorQueue, ^{
        readStreamBlock();
    });

trouble: stack is growing and autorelease pull will not execute release for buffer.

solution: add async in the same queue.. this will let current block to finish execution.

Here is the code:

__block NSMutableData *data = [NSMutableData dataWithLength:blockSize];
__block RNDecryptor *decryptor = nil;


dispatch_block_t readStreamBlock = ^{

    [data setLength:blockSize];

    NSInteger bytesRead = [inputStream read:[data mutableBytes] maxLength:blockSize];
    if (bytesRead < 0) {
        // Throw an error
    }
    else if (bytesRead == 0) {
        [decryptor finish];
    }
    else {

        [data setLength:bytesRead];
        [decryptor addData:data];
    }
};

decryptor = [[RNDecryptor alloc] initWithPassword:@"blah" handler:^(RNCryptor *cryptor, NSData *data) {

        [decryptedStream write:data.bytes maxLength:data.length];
        _percentStatus = (CGFloat)[[decryptedStream propertyForKey:NSStreamFileCurrentOffsetKey] intValue] / (CGFloat)_inputFileSize;
        if (cryptor.isFinished)
        {
            [decryptedStream close];
            [self decryptFinish];
        }
        else
        {
            dispatch_async(cryptor.responseQueue, ^{
                readStreamBlock();
            });
            [self decryptStatusChange];
        }

}];


// Read the first block to kick things off

decryptor.responseQueue = self.cryptorQueue;
[self decryptStart];
dispatch_async(decryptor.cryptorQueue, ^{
    readStreamBlock();
});

Answer 3

I may be wrong, but I think your do...while loop prevents the autorelease pool from draining frequently enough.

Why are you using this loop to wait for the decryptor to finish? You should use the completion block to notify your controller that the drcryptor has finished, instead.

(BTW, welcome to SO, your question is really well asked and that's highly appreciated).

Answer 4

I might again be wrong, but in your readStreamBlock, data should be a parameter to the block, rather than a reference to a __block NSMutableData declared outside of it. As you can see, the RNEncryptor handler provides its own data variable, which is different from the one you've declared yourself.

Ideally, put all your readStreamBlock directly inside the handler, without even declaring it a block.