How do I create a stream for actix-web HttpResponse to send a file chunk by chunk?

Question

I want to stream a encrypted file with actix-web in Rust. I have a loop that decrypts the encrypted file chunk by chunk using sodiumoxide. I want to send the chunks to the client.

My loop looks like this:

while stream.is_not_finalized() {
    match in_file.read(&mut buffer) {
        Ok(num_read) if num_read > 0 => {
            let (decrypted, _tag) = stream
                .pull(&buffer[..num_read], None)
                .map_err(|_| error::ErrorInternalServerError("Incorrect password"))
                .unwrap();

            // here I want to send decrypted to HttpResponse
            continue;
        }
        Err(e) => error::ErrorInternalServerError(e),
        _ => error::ErrorInternalServerError("Decryption error"), // reached EOF
    };
}

I found a streaming method, which needs a Stream as a parameter. How can I create a stream where I can add chunk by chunk?

score 2 · Answer 1 · answered Oct 22 '20 at 16:26

Depending on your workflow(How big your data chunks are, how time consuming the decrypting is, etc) you may have different options on how to make the stream. The most legimitate way that comes to my mind is using some kind of thread-pool with a channel to communicate between the thread and your handler. Tokio's mpsc can be an option in that situation and its Receiver already implements Stream and you can feed it from your thread by using it's Sender's try_send from your thread, considering you use an unbounded channel or a bounded channel with enough length, it should work.

Another possible option for cases where your decryption process isn't that time consuming to be considered blocking, or you just want to see how you can implement a Stream for actix, here's an example:

use std::pin::Pin;
use std::task::{Context, Poll};

use actix_web::{get, App, HttpResponse, HttpServer, Responder};
use pin_project::pin_project;
use sodiumoxide::crypto::secretstream::{Pull, Stream};
use tokio::{fs::File, io::AsyncRead};

#[pin_project]
struct Streamer {
    crypto_stream: Stream<Pull>,
    #[pin]
    file: File,
}

impl tokio::stream::Stream for Streamer {
    type Item = Result<actix_web::web::Bytes, actix_web::Error>;

    fn poll_next(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<Self::Item>> {
        let this = self.project();

        let mut buffer = [0; BUFFER_LENGTH];

        if this.crypto_stream.is_not_finalized() {
            match this.file.poll_read(cx, &mut buffer) {
                Poll::Ready(res) => match res {
                    Ok(bytes_read) if bytes_read > 0 => {
                        let value = this.crypto_stream.pull(&buffer, None);
                        match value {
                            Ok((decrypted, _tag)) => Poll::Ready(Some(Ok(decrypted.into()))),
                            Err(_) => Poll::Ready(Some(Err(
                                actix_web::error::ErrorInternalServerError("Incorrect password"),
                            ))),
                        }
                    }
                    Err(err) => {
                        Poll::Ready(Some(Err(actix_web::error::ErrorInternalServerError(err))))
                    }
                    _ => Poll::Ready(Some(Err(actix_web::error::ErrorInternalServerError("Decryption error")))),
                },
                Poll::Pending => Poll::Pending,
            }
        } else {
            // Stream finishes when it returns None
            Poll::Ready(None)
        }
    }
}

and use it from your handler:

let in_file = File::open(FILE_NAME).await?;
let stream = Stream::init_pull(&header, &key)?;

let stream = Streamer {
    crypto_stream: stream,
    file: in_file,
};
HttpResponse::Ok()
//    .content_type("text/text")
    .streaming(stream)

Note that you need pin_project and tokio with ["stream", "fs"] as dependency for it to work.

How do I create a stream for actix-web HttpResponse to send a file chunk by chunk?

1 Answers1