0

I'm developing an upload project in swift. I'm taking very large files (video, picture with size over 500 MB) with imagepickercontroller and dividing this file into chunks which has a size 1 MB. Then I send these chunks to remote server and make them defragment in server and I'm showing this file to user.

I have no problem if the file size is under 300 MB. But after this size, memory goes up too much and app is being crashed. Actually, in every case memory usage are raising but there is no crash.

When I watch progress on console, I see URLSession task begins. But, because of these tasks are waiting response from completion handler, the task queue is growing and memory usage goes up. Is there a way when a task begins, this task's completion handler begins too? I think if I can make task queue free concurrently, my problem solves. I'm waiting your helps. 

let url:URL = URL(string: "\(addressPrefix)UploadFile")!
let session = URLSession.shared
let request = NSMutableURLRequest(url: url)
request.cachePolicy = NSURLRequest.CachePolicy.reloadIgnoringCacheData
request.httpMethod = "POST"

let bodyData = "\(metaDataID)~\(chunkIndex)~\(chunkSize)~\(chunkHash)~\(wholeTicket)~\(fileDataString)"

request.httpBody = bodyData.data(using: String.Encoding(rawValue: String.Encoding.utf8.rawValue));
request.timeoutInterval = .infinity

let task = session.dataTask(with: request as URLRequest, completionHandler: {(data, response, error) in
    guard let _:Data = data, let _:URLResponse = response, error == nil else {
       var attemptCounter = 1
       if attemptCounter <= 3 {
            completion("\(attemptCounter).attempt",chunkSize, error)
            attemptCounter += 1
        }
         return
     }
    let jsonStr = String(data: data!, encoding: String.Encoding(rawValue: String.Encoding.utf8.rawValue))
    completion(jsonStr, chunkSize, error) 
 SingletonConnectionManager.sharedConnectionDataManager.dataTasks["uploadFile"] = nil 
})  
SingletonConnectionManager.sharedConnectionDataManager.dataTasks["uploadFile"] = task 
task.resume()

---I call this URLSession task from this function in a tableview controller

 tmpConnection.uploadFile(chunk, metaDataID!, chunkIndex: chunkIndex, completion: {(result, chunkSize, error) in
   // I want to enter immediately when 'uploadFile' get called })
ensar.koseoglu
  • 5
  • 5
  • Are you scheduling all 500 requests at once? Yikes. You should probably chunk the file up on disk (not in memory), and then keep track of which ones have been sent, and send at most eight or so of them at a time, sending out the next new chunk as soon as one finishes or fails. And if you control the server code, then chunking it on disk would also let you use an upload task instead of having to construct a request body yourself. (Just the chunk ID and other values into the query string, and make the body be a raw blob of binary data.) – dgatwood Feb 06 '19 at 22:38
  • Thanks for reply @dgatwood. I scheduling requests in a loop and send them to data task. As you said, I should track requests by sending to server and get response. But, in this case requests are waiting until all of them has been sent. After that, I get response with completion handler but memory is raising until get response. I can't finish any task until get response from completion handler. Is there any chance that I can get response without waiting completion handler? – ensar.koseoglu Feb 07 '19 at 06:27

1 Answers1

1

The requests aren't really waiting until all of them have been sent. When things are working correctly, each callback happens when the associated request finishes, and it wouldn't make sense for that to happen sooner, because the callback provides the response from the server (which you can't possibly get back until after the request has been fully sent out).

The problem here is that you are completely clogging up the session by starting entirely too many tasks at the same time. There's a known bug in NSURLSession that causes it to start to fall apart when you create a large number of tasks in a single session all at once. When you get too many tasks in a session, IIRC, the session stops calling callbacks entirely, and basically the session becomes unusable. (There's another Stack Overflow question in which this was discussed a couple of years ago, though I can't seem to find it right now.)

And because the tasks never complete, your app ends up leaking all the memory that you're using for the body data, which means your app just allocates more and more memory until it gets evicted.

The only way to fix this problem is to stop adding all of the requests to the session all at once. Start a few tasks at first (for at most eight parts or so), and then wait to send the next part until one of the previous parts finishes or fails. This approach will not only prevent you from bricking the NSURLSession, but also will prevent you from allocating an insane amount of memory to hold all of the request body NSData objects, which are currently all sitting in RAM at once.

I suggest keeping an NSMutableArray of NSNumber object representing each unsent chunk. That way, you know what is still left to send, and you can just loop to 8 and pull off the first 8 numbers, and send the chunks with those numbers. When a request completes successfully, grab the next number out of the array and send the chunk with that number.

Also, you shouldn't stop after a particular number of retries. Instead, when a request fails, check the failure to decide whether to retry (network failure) or give up (server error). Then use reachability to wait until a good time to try again, and try again when it says that the destination host is reachable. Cancel the upload only if the user explicitly asks you to cancel the upload by hitting a cancel button or similar. If the user asks you to cancel the upload, tear down your data structure so you don't start any new requests, then invalidate the URL session.

dgatwood
  • 10,129
  • 1
  • 28
  • 49
  • Actually, I tried to keep waiting requests until task queue number reaches to ten. But, then I couldn't make them continue again. Should I use dispatch queue or operation queue? If I can start the requests which waited, I think problem will solve. I tried DispatchGroups, but probably used wrong. Can someone of these handle this issue? – ensar.koseoglu Feb 07 '19 at 10:21
  • Don't do any of those things. You're making it way too complicated. Just have a method that takes the file URL and returns chunk number *n* (i.e. if you ask for chunk 0, you get the first megabyte of data). Keep an array containing the numbers from 0 to megabytesInFileRoundedUp. Then, write another method that pops a number off of the front of that mutable array, asks for that chunk, and starts the upload. Call that method in a loop 8 times. In the completion handler, call the method that pops the number off the front of the array, asks for the chunk, and starts the upload. – dgatwood Feb 07 '19 at 21:33
  • To simplify the data management, pass the mutable array into all of these methods. When the last attempt to pop things off the empty array returns, the empty array goes away because no block is retaining it anymore. – dgatwood Feb 07 '19 at 21:34
  • This is a clear and good algorithm @dgatwood. I'm trying this. But there are some points I couldn't understand. First of all, should the array I keep numbers contain chunkSize (i.e. 1MB,1MB) or chunkNo (i.e. 0. ,1. , 2. chunk). Secondly, how can I make chunkNo continue after the loop returns 8 times. Is it like a recursive function? Sorry for that stupid questions but the project is really confusing and I'm a bit new in Swift. Additionally, I have to chunk this file into Data and it's a big load on memory. Because, when I send this chunk to server, I must take hash and hex of this all chunks. – ensar.koseoglu Feb 08 '19 at 12:05
  • The latter. An array containing the actual numbers 0, 1, 2, ... n where n is the number of megabytes in the file, rounded up. And don't chunk the file into data objects ahead of time. Instead, use dataWithContentsOfFile:options:error: to memory-map the file so that only the bits that you're actively working with are actively using RAM, then use subdataWithRange: to create a data object that contains only the nth chunk whenever you're ready to send that chunk, compute hashes on it, etc. – dgatwood Feb 09 '19 at 07:16
  • And then at the end, you just loop up to 8 (or some other small number) and call a method that removes the first item from the array, get the subdata for that particular chunk number, and send it. When that completes, do it again. So that method is recursively picking chunk numbers off the front of the array and sending out the corresponding chunk of data until it runs out of numbers, then stops. And you're calling it eight times in parallel. :-) Be sure to dispatch the call onto the main thread each time so that you aren't modifying the mutable array from multiple threads concurrently. – dgatwood Feb 09 '19 at 07:19
  • Hi, @dgatwood. I could apply changes as you said and it's working better than before. There is an error I couldn't understand. "Error Domain=NSPOSIXErrorDomain Code=12 'Cannot allocate memory'". I think it depends this code line -> "let dataPath = try NSData(contentsOf: self.fileSelectedURL!, options: .alwaysMapped).subdata(with: range)". I'm dividing data as you said against to chunk number with range. But, I think there is an issue on map process. – ensar.koseoglu Feb 11 '19 at 09:56
  • I searched a little and I found "This is not about physical memory but address space. iOS puts limits on the address space available to app processes, meaning your can’t map files beyond a certain size" on apple forum. https://forums.developer.apple.com/thread/99852 . Do you think it's because of that? – ensar.koseoglu Feb 11 '19 at 09:58
  • Are you still building 32-bit targets? I just assumed you were building 64-bit only, but if you are still building 32-bit targets, then yes, you can't use mmap usefully, because the address space is way too small. Instead, you would need to explicitly read the correct chunk from the file the old-fashioned way. Unfortunately, AFAIK, Cocoa APIs like NSInputStream don't support seeking, so you're much better off writing a small chunk of code that does fopen, fseek, fread, and fclose on the file. – dgatwood Feb 11 '19 at 22:58
  • Actually, I thought I'm building 64-bit targets. But it's probably not. Instead, I used FileHandle. I tried first InputStream and I couldn't arrange start and finish offset. Then, I tried FileHandle and I made it. You really helped me a lot about this problem and I'm really thank you @dgatwood. – ensar.koseoglu Feb 14 '19 at 06:03