tokio-curl: capture output into a local `Vec` - may outlive borrowed value

Question

I do not know Rust well enough to understand lifetimes and closures yet...

Trying to collect the downloaded data into a vector using tokio-curl:

extern crate curl;
extern crate futures;
extern crate tokio_core;
extern crate tokio_curl;

use std::io::{self, Write};
use std::str;

use curl::easy::Easy;
use tokio_core::reactor::Core;
use tokio_curl::Session;

fn main() {
    // Create an event loop that we'll run on, as well as an HTTP `Session`
    // which we'll be routing all requests through.
    let mut lp = Core::new().unwrap();
    let mut out = Vec::new();
    let session = Session::new(lp.handle());

    // Prepare the HTTP request to be sent.
    let mut req = Easy::new();
    req.get(true).unwrap();
    req.url("https://www.rust-lang.org").unwrap();
    req.write_function(|data| {
            out.extend_from_slice(data);
            io::stdout().write_all(data).unwrap();
            Ok(data.len())
        })
        .unwrap();

    // Once we've got our session, issue an HTTP request to download the
    // rust-lang home page
    let request = session.perform(req);

    // Execute the request, and print the response code as well as the error
    // that happened (if any).
    let mut req = lp.run(request).unwrap();
    println!("{:?}", req.response_code());
    println!("out: {}", str::from_utf8(&out).unwrap());
} 

Produces an error:

error[E0373]: closure may outlive the current function, but it borrows `out`, which is owned by the current function
  --> src/main.rs:25:24
   |
25 |     req.write_function(|data| {
   |                        ^^^^^^ may outlive borrowed value `out`
26 |             out.extend_from_slice(data);
   |             --- `out` is borrowed here
   |
help: to force the closure to take ownership of `out` (and any other referenced variables), use the `move` keyword, as shown:
   |     req.write_function(move |data| {

---

Investigating further, I see that Easy::write_function requires the 'static lifetime, but the example of how to collect output from the curl-rust docs uses Transfer::write_function instead:

use curl::easy::Easy;

let mut data = Vec::new();
let mut handle = Easy::new();
handle.url("https://www.rust-lang.org/").unwrap();
{
    let mut transfer = handle.transfer();
    transfer.write_function(|new_data| {
        data.extend_from_slice(new_data);
        Ok(new_data.len())
    }).unwrap();
    transfer.perform().unwrap();
}
println!("{:?}", data);

The Transfer::write_function does not require the 'static lifetime:

impl<'easy, 'data> Transfer<'easy, 'data> {
    /// Same as `Easy::write_function`, just takes a non `'static` lifetime
    /// corresponding to the lifetime of this transfer.
    pub fn write_function<F>(&mut self, f: F) -> Result<(), Error>
        where F: FnMut(&[u8]) -> Result<usize, WriteError> + 'data
    {
...

But I can't use a Transfer instance on tokio-curl's Session::perform because it requires the Easy type:

pub fn perform(&self, handle: Easy) -> Perform {

transfer.easy is a private field that is directly passed to session.perform.

It this an issue with tokio-curl? Maybe it should mark the transfer.easy field as public or implement new function like perform_transfer? Is there another way to collect output using tokio-curl per transfer?

Answer 1

The first thing you have to understand when using the futures library is that you don't have any control over what thread the code is going to run on.

In addition, the documentation for curl's Easy::write_function says:

Note that the lifetime bound on this function is 'static, but that is often too restrictive. To use stack data consider calling the transfer method and then using write_function to configure a callback that can reference stack-local data.

The most straight-forward solution is to use some type of locking primitive to ensure that only one thread at a time may have access to the vector. You also have to share ownership of the vector between the main thread and the closure:

use std::sync::Mutex;
use std::sync::Arc;

let out = Arc::new(Mutex::new(Vec::new()));
let out_closure = out.clone();

// ...

req.write_function(move |data| {
    let mut out = out_closure.lock().expect("Unable to lock output");
    // ...
}).expect("Cannot set writing function");

// ...

let out = out.lock().expect("Unable to lock output");
println!("out: {}", str::from_utf8(&out).expect("Data was not UTF-8"));

Unfortunately, the tokio-curl library does not currently support using the Transfer type that would allow for stack-based data.