Issues uploading/downloading files in akka-http/akka-streams

Question

I'm trying to use akka-streams and akka-http and the alpakka library to download/upload files to Amazon S3. I am seeing two issues which might be related...

I can only download very small files, the largest one 8kb.
I can't upload larger files. It fails with the message

Error during processing of request: 'Substream Source has not been materialized in 5000 milliseconds'. Completing with 500 Internal Server Error response. To change default exception handling behavior, provide a custom ExceptionHandler. akka.stream.impl.SubscriptionTimeoutException: Substream Source has not been materialized in 5000 milliseconds

Here are my routes

pathEnd {
           post {
             fileUpload("attachment") {
               case (metadata, byteSource) => {
                 val writeResult: Future[MultipartUploadResult] = byteSource.runWith(client.multipartUpload("bucketname", key))
                 onSuccess(writeResult) { result =>
                   complete(result.location.toString())
                 }
               }

             }
           }

         } ~

     path("key" / Segment) {
            (sourceSystem, sourceTable, sourceId) =>
              get {
                val result: Future[ByteString] = 
         client.download("bucketname", key).runWith(Sink.head)
                onSuccess(result) {
                  complete(_)
                }
              }
          }

Trying to download a file of say 100KB will end up fetching a truncated version of the file usually of size around 16-25Kb Any help appreciated

Edit: For the download issue, I took Stefano's suggestion and got

[error]  found   : akka.stream.scaladsl.Source[akka.util.ByteString,akka.NotUsed]
[error]  required: akka.http.scaladsl.marshalling.ToResponseMarshallable

This made it work

complete(HttpEntity(ContentTypes.`application/octet-stream`, client.download("bucketname", key).runWith(Sink.head)))

score 2 · Answer 1 · answered Sep 19 '17 at 20:44

1) On the download issue: by calling

val result: Future[ByteString] = 
         client.download("bucketname", key).runWith(Sink.head)

you are streaming all the data from S3 into memory, and then serve the result.

Akka-Http as streaming support that allows you to stream bytes straight from a source, without buffering them all in memory. More info on this can be found in the docs. Practically, this means the complete directive can take a Source[ByteString, _], as in

...
get {
  complete(client.download("bucketname", key))
}

2) On the upload issue: you can try to tweak Akka HTTP akka.http.server.parsing.max-content-length setting:

# Default maximum content length which should not be exceeded by incoming request entities.
# Can be changed at runtime (to a higher or lower value) via the `HttpEntity::withSizeLimit` method.
# Note that it is not necessarily a problem to set this to a high value as all stream operations
# are always properly backpressured.
# Nevertheless you might want to apply some limit in order to prevent a single client from consuming
# an excessive amount of server resources.
#
# Set to `infinite` to completely disable entity length checks. (Even then you can still apply one
# programmatically via `withSizeLimit`.)
max-content-length = 8m

Resulting code to test this would be something along the lines of:

  withoutSizeLimit {
    fileUpload("attachment") {
      ...
    }
  }

Issues uploading/downloading files in akka-http/akka-streams

1 Answers1