Uploading Large NSData to the Web

Question

I'm currently working on an application that has to upload large files (mainly movies/videos) to the web. After reading what I can, I went the the approach of converting the movie to NSData and then including that as the NSURLConnection's HTTPBody. However, upon converting the movie (which was originally an ALAsset) into NSData, I receive a memory warning and then a subsequent crash.

I have no idea how I would go about uploading these types of large files, if that data just causes an instant crash. One solution that I was thinking of is writing to the filesystem and then uploading a file directly from there, but I have not been able to find any information on how one would accomplish this.

Here is the relevant code that I use. If there is something that I'm doing wrong right here, I'd love to know.

ALAssetRepresentation *representation = [asset defaultRepresentation];

Byte *buffer = (Byte *)malloc([representation size]);
NSUInteger buffered = [representation getBytes:buffer fromOffset:0.0 length:[representation size] error:nil];

uploadData = [NSData dataWithBytes:buffer length:buffered];

free(buffer);

i am stuck with the same issue. Aaron if you have got success doing this can you share some code as to how to do this. Thanks — Rishi, May 10 '13 at 06:30
The answers below helped me and should contain all you need to get started. — Aaron Wojnowski, May 10 '13 at 21:01

David H · Accepted Answer · 2015-11-26T15:16:42.580

2

Assuming that it makes sense to upload the movie in its native format, you can really make this easier using the BSD (ie Unix) section 3 interface:

given a filePath, open the file and get an int file descriptor (fd)
with fd, get the length of the file
keep track of how much you've loaded so you know where to get more data
use mmap(3) to map in JUST the data you want to upload at any time, and use the void * pointer returned by mmap as the location of the data
when the data has been sent, munmap the old data chunk and mmap a new chunk
after all data is sent, munmap the last chunk, the close(fd).

No temporary memory - no mallocs. I use mmap whenever I have to deal with huge files.

Edit: you can also use NSData dataWithContentsOfFile:options with options set to use mmap. You would then use the byte pointer to read small chunks as you need them.

edited Nov 26 '15 at 15:16

answered Aug 14 '12 at 23:22

David H

40,852
12
92
138

what to do in case we dont have absolute path of the file but the relative path from alassetrepresentation?? – sachy Oct 25 '14 at 14:01
@sachy, if must be able to open() the file using open or fopen or have some means to get the file descriptor. Otherwise you cannot use this technique. – David H Oct 26 '14 at 16:52
@AaronWojnowski Im also facing same issue, Could u plz help me how to proceed with some code or tutorial – siva krishna Nov 26 '15 at 07:31

score 1 · Answer 2 · edited May 23 '17 at 11:59

In case anyone got here and couldn't solve your problems, I figured out a way to do this. You have to firstly write your ALAssetRepresentation to disk (as described here):

NSUInteger chunkSize = 100 * 1024;
NSString *tempFile = [NSTemporaryDirectory() stringByAppendingPathComponent:@"temp.tmp"];

uint8_t *chunkBuffer = malloc(chunkSize * sizeof(uint8_t));
NSUInteger length = [rep size];

NSFileHandle *fileHandle = [[NSFileHandle fileHandleForWritingAtPath: tempFile] retain];
if(fileHandle == nil) {
    [[NSFileManager defaultManager] createFileAtPath:tempFile contents:nil attributes:nil];
    fileHandle = [[NSFileHandle fileHandleForWritingAtPath:tempFile] retain];
}

NSUInteger offset = 0;
do {
    NSUInteger bytesCopied = [rep getBytes:chunkBuffer fromOffset:offset length:chunkSize error:nil];
    offset += bytesCopied;
    NSData *data = [[NSData alloc] initWithBytes:chunkBuffer length:bytesCopied];
    [fileHandle writeData:data];
    [data release];
} while (offset < length);
[fileHandle closeFile];
[fileHandle release];
free(chunkBuffer);
chunkBuffer = NULL;

Then you have to create an NSData object that can map the disk without using memory resources (kind of like David's answer, but inspired by this answer):

NSError *error;
NSData *fileData = [NSData dataWithContentsOfFile:tempFile options:NSDataReadingMappedIfSafe error:&error];
if (!fileData) {
    NSLog(@"Error %@ %@", error, [error description]);
    NSLog(@"%@", tempFile);
    //do what you need with the error
}

EDIT Although, if you are uploading the file somewhere, you should open a connection and send small buffers of the file, kind of like what I did above. I had to write a C++ class to handle the socket and the connection

score 0 · Answer 3 · answered Aug 14 '12 at 22:34

You probably shouldn't be trying to read the whole asset in one shot:

Byte *buffer = (Byte *)malloc([representation size]);
NSUInteger buffered = [representation getBytes:buffer fromOffset:0.0 length:[representation size] error:nil];

Instead, set up a loop and read from the asset in chunks. I've outlined the basic approach. You'll need to fill in a few gaps, but it should solve the memory issue.

You might also want to consider running this in a thread so you don't lock up the UI.

    NSError error;
    int bufferSize = 1000;
    float offset=0.0;

//TODO: Open Connection

    while (1)
    {
      Byte *buffer = (Byte *)malloc(bufferSize);
       NSUInteger buffered = [representation getBytes:buffer fromOffset:offset length:bufferSize  error:&error];

    //TODO: Write data  
    //TODO: Increment offset, check errors

    free(buffer);

    //if (done){
    //break;
    //}

    }

    //TODO close eonnection

I still seem to be receiving a crash due to memory with this one when appending it to NSMutableData (understandably). I use this exact code, and it works very well, for writing to a file though. — Aaron Wojnowski, Aug 15 '12 at 00:02

Uploading Large NSData to the Web

3 Answers3

Linked