How to create a Stream from reading and transforming a file?

Question

I'm trying to read a file, decrypt it, and return the data. Because the file is potentially very big, I want to do this in a stream.

I cannot find a good pattern to implement the stream. I'm trying to do something like this:

let stream = stream::unfold(decrypted_init_length, |decrypted_length| async move {
    if decrypted_length < start + length {
        let mut encrypted_chunk = vec![0u8; encrypted_block_size];
        match f.read(&mut encrypted_chunk[..]) {
            Ok(size) => {
                if size > 0 {
                    let decrypted = my_decrypt_fn(&encrypted_chunk[..]);
                    let updated_decrypted_length = decrypted_length + decrypted.len();
                    Some((decrypted, updated_decrypted_length))
                } else {
                    None
                }
            }
            Err(e) => {
                println!("Error {}", e);
                None
            }
        }
    } else {
        None
    }
});

The problem is that f.read is not allowed in the above async closure with the following error:

89  | |             match f.read(&mut encrypted_chunk[..]) {
    | |                   -
    | |                   |
    | |                   move occurs because `f` has type `std::fs::File`, which does not implement the `Copy` trait
    | |                   move occurs due to use in generator

I don't want to open f inside the closure itself. Is there any better way to fix this? I am OK with using a different crate or trait, or method (i.e. not stream::unfold).

Are you using an asynchronous file type? (e.g. [async-std](https://docs.rs/async-std/*/async_std/fs/struct.File.html) or [tokio](https://docs.rs/tokio/*/tokio/fs/struct.File.html)). — squiguy, Mar 26 '20 at 22:41
@squiguy No, I am using a regular file type, i.e. `std::fs::File`. — user1783732, Mar 27 '20 at 04:47
It's hard to answer your question because it doesn't include a [MRE]. We can't tell what crates (and their versions), types, traits, fields, etc. are present in the code. It would make it easier for us to help you if you try to reproduce your error on the [Rust Playground](https://play.rust-lang.org) if possible, otherwise in a brand new Cargo project, then [edit] your question to include the additional info. There are [Rust-specific MRE tips](//stackoverflow.com/tags/rust/info) you can use to reduce your original code for posting here. Thanks! — Shepmaster, Mar 27 '20 at 18:50

user1783732 · Answer 1 · 2020-04-03T15:28:15.347

I found a solution: using async-stream crate at here.

One of the reasons stream::unfold did not work for me is that the async move closure does not allow access mut variables outside, for example the f file handle.

Now with async-stream, I changed my code to the following, and it works: (note the yield added by this crate).

use async_stream::try_stream;

<snip>

    try_stream! {
        while decrypted_length < start + length {
            match f.read(&mut encrypted_chunk[..]) {
                Ok(size) => 
                    if size > 0 {
                        println!("read {} bytes", size);
                        let decrypted = my_decrypt_fn(&encrypted_chunk[..size], ..);
                        decrypted_length = decrypted_length + decrypted.len();
                        yield decrypted;
                    } else {
                        break
                    }
                Err(e) => {
                    println!("Error {}", e);
                    break
                }
            }
        }
    }

UPDATE:

I found that async-stream has some limitations that I cannot ignore. I ended up implementing Stream directly and no longer using async-stream. Now my code looks like this:

pub struct DecryptFileStream {
    f: File,
    <other_fields>,
}

impl Stream for DecryptFileStream {
    type Item = io::Result<Vec<u8>>;

    fn poll_next(self: Pin<&mut Self>,
                  _cx: &mut Context<'_>) -> Poll<Option<io::Result<Vec<u8>>>> {
         // read the file `f` of self and business_logic
         // 
         if decrypted.len() > 0 {
             Poll::Ready(Some(Ok(decrypted)))
         } else {
             Poll::Ready(None)
         }
    }
}

//. then use the above stream: 

    let stream = DecryptFileStream::new(...);
    Response::new(Body::wrap_stream(stream))

I think it's from this one https://docs.rs/futures-core/0.3.25/futures_core/stream/trait.Stream.html — Aditya Kresna Permana, Dec 11 '22 at 10:15

score 0 · Answer 2 · answered Mar 27 '20 at 10:08

0

stream::unfold is only for types that implement Stream, which in Rust is used exclusively for asynchronous programming. If you want to do synchronous reading, what you're calling a "stream" is tagged as implementing Read in Rust. Thus you can call Read::read() to read some data off the current position of your File (limited by the length of the buffer you pass in), and decrypt that data.

answered Mar 27 '20 at 10:08

djc

11,603
5
41
54

Although I am using `std::fs::File` to read the chunk, the higher level logic is asynchronous programming. In this case, I am using `hyper` crate and I wanted to create a stream for `hyper` response body. – user1783732 Mar 27 '20 at 16:24
So you want to take an encrypted file and apply [`Body::wrap_stream()`](https://docs.rs/hyper/0.13.4/hyper/body/struct.Body.html#method.wrap_stream) to it? In that case you probably want to use `tokio::fs::File` and its `AsyncRead` impl. It would probably help if you clear up your question as Shepmaster suggested, though. – djc Mar 27 '20 at 20:48
yes I will use `Body::wrap_stream()` once I got a stream out of the file. I found a good crate to do that now. Posted details as separated answer. – user1783732 Mar 27 '20 at 22:19

How to create a Stream from reading and transforming a file?

2 Answers2