0

I am trying to download a file using reqwest. The response status is 200. At the line when I am trying to read the response as bytes response.bytes().await? it waits forever.

However, when I try to make a curl request for the same URL, it passes and I am able to successfully download the file.

I am not sure what is wrong or how I should debug from here. Any suggestion is welcome.

use anyhow::Context;
use reqwest_middleware::ClientBuilder;
use reqwest_retry::{policies::ExponentialBackoff, RetryTransientMiddleware};
use reqwest_tracing::TracingMiddleware;
use std::fs;
use std::io::Cursor;
use std::os::unix::fs::PermissionsExt;
use std::path::PathBuf;
use std::{
    path::Path,
    process::{Command, Output},
};
        
async fn download() -> Result<(), anyhow::Error> {
            let panda_link = format!(
                "https://surya.jfrog.io/artifactory/binaries/panda/v2.34.0/{}/panda",
                ARCH_NAME
            );


            let retry_policy = ExponentialBackoff::builder().build_with_max_retries(3);
            let client = ClientBuilder::new(reqwest::Client::new())
                .with(TracingMiddleware)
                .with(RetryTransientMiddleware::new_with_policy(retry_policy))
                .build();
            println!("the client has been successfully built");

            let response = client
                .get(panda_link)
                .send()
                .await?;

            println!("got the response {}", response.status());
            response
                .error_for_status_ref()
                .context("Failed to download panda")?;
            println!("check if response was error is done");

            let response_bytes = response.bytes().await?;
            println!("reading response bytes");

            let mut content = Cursor::new(response_bytes);
            println!("reading bytes");

            let new_file_path = PathBuf::from("/");
            println!("this is the newfile_path {:?}", new_file_path);

            let mut file = std::fs::File::create(&new_file_path)
                .context(format!("Failed creating file {}", &new_file_path.display()))?;
            fs::set_permissions(&new_file_path, fs::Permissions::from_mode(0o750)).context(
                format!("Failed making {} executable", &new_file_path.display()),
            )?;
            println!("file created");
            std::io::copy(&mut content, &mut file)
                .context(format!("Failed saving file {}", &new_file_path.display()))?;

            Ok(())
}
Herohtar
  • 5,347
  • 4
  • 31
  • 41
Surya
  • 2,429
  • 1
  • 21
  • 42

1 Answers1

0

I think the issue you are having is a combination of multiple things:

  1. Personal internet download speed.
  2. The Website speed itself. (website's traffic load, location of the website, the website's speed to generate download).
  3. Ram speed.
  4. Disk speed.

I did some tests with the code below. (It's fuctional and doesn't "wait forever")

If I pointed the download link to a large file (~100 Mb) on google docs it would download in (give or take) a second, but if I pointed it to a large file (~100 Mb) on a website that wasn't very fast then it would take a few seconds.

    // Download link
    let panda_link = format!(
        "https://surya.jfrog.io/artifactory/binaries/panda/v2.34.0/{}/panda",
        ARCH_NAME
    );

    // File Path
    let new_file_path = PathBuf::from("/");

    // Gets the response from website as bytes (Not sure if its stored in the memory. If it is stored in the memory than it would be impossible to download something that is larger than your total ram size.)
    let content = reqwest::get(panda_link).await?.bytes().await?;
    // Writes the bytes to the disk.
    fs::write(new_file_path, content)?;
Austin Hamner
  • 135
  • 9