How to write a file downloaded with SwiftNIO AsyncHTTPClient to the filesystem?

Question

I would like to download a large file (hundreds of megabytes) with the AsyncHTTPClient library based on SwiftNIO. I would like this file to be streamed to the filesystem, while consuming as little RAM as possible (ideally it shouldn't keep the whole file in the RAM), and also being able to report the download progress with a simple print output that shows the completion percentage.

As far as I understand, I need to implement an HTTPClientResponseDelegate, but what exact API should I use for file writes? Can file writes be blocking, while still allowing the HTTP client to progress? How would the delegate code look in this scenario?

score 0 · Answer 1 · answered Jun 24 '20 at 16:38

Turns out, HTTPClientResponseDelegate allows returning a future in its functions to allow it to correctly handle backpressure. Combined this approach with NonBlockingFileIO and NIOFileHandle, the delegate that writes the file to disk with progress reporting while it's downloaded looks like this:

import AsyncHTTPClient
import NIO
import NIOHTTP1

final class FileDownloadDelegate: HTTPClientResponseDelegate {
  typealias Response = (totalBytes: Int?, receivedBytes: Int)

  private var totalBytes: Int?
  private var receivedBytes = 0

  private let handle: NIOFileHandle
  private let io: NonBlockingFileIO
  private let reportProgress: (_ totalBytes: Int?, _ receivedBytes: Int) -> ()

  private var writeFuture: EventLoopFuture<()>?

  init(
    path: String,
    reportProgress: @escaping (_ totalBytes: Int?, _ receivedBytes: Int) -> ()
  ) throws {
    handle = try NIOFileHandle(path: path, mode: .write, flags: .allowFileCreation())
    let pool = NIOThreadPool(numberOfThreads: 1)
    pool.start()
    io = NonBlockingFileIO(threadPool: pool)

    self.reportProgress = reportProgress
  }

  func didReceiveHead(
    task: HTTPClient.Task<Response>,
    _ head: HTTPResponseHead
  ) -> EventLoopFuture<()> {
    if let totalBytesString = head.headers.first(name: "Content-Length"),
      let totalBytes = Int(totalBytesString) {
      self.totalBytes = totalBytes
    }

    return task.eventLoop.makeSucceededFuture(())
  }

  func didReceiveBodyPart(
    task: HTTPClient.Task<Response>,
    _ buffer: ByteBuffer
  ) -> EventLoopFuture<()> {
    receivedBytes += buffer.readableBytes
    reportProgress(totalBytes, receivedBytes)

    let writeFuture = io.write(fileHandle: handle, buffer: buffer, eventLoop: task.eventLoop)
    self.writeFuture = writeFuture
    return writeFuture
  }

  func didFinishRequest(task: HTTPClient.Task<Response>) throws -> Response {
    writeFuture?.whenComplete { [weak self] _ in
      try? self?.handle.close()
      self?.writeFuture = nil
    }
    return (totalBytes, receivedBytes)
  }
}

With this code, the process downloading and writing the file does not consume more than 5MB of RAM for a ~600MB downloaded file.

How to write a file downloaded with SwiftNIO AsyncHTTPClient to the filesystem?

1 Answers1