0

I am trying to obtain an HTTP response asynchronously, but the speed is slow. I am not sure if the asynchronous approach is successful. Removing json::<serde_json::Value>().await can make it faster, but then won't be able to get the returned data Would be grateful for any assistance.

let package_json_string = read_to_string("package.json")
    .expect("read pacakge.json error");
let mut query_key_list = vec![];
let PackageJson {
    dependencies,
    devDependencies,
}: PackageJson = serde_json::from_str(package_json_string.as_str())
    .expect("parse error");
for key in dependencies.keys() {
    query_key_list.push(key)
}
for key in devDependencies.keys() {
    query_key_list.push(key)
}
let client = Client::new();
let mut task = vec![];
let start = Instant::now();
for key in query_key_list {
    let url = format!("{}{}", "https://registry.npmjs.org/", key);
    let client = client.clone();
    task.push(tokio::spawn(async move {
        let result = client.get(url).send().await;
        let res = result.unwrap().json::<serde_json::Value>().await.unwrap();
    }));
}
join_all(task).await;
let duration = start.elapsed();
println!("Time elapsed in expensive_function() is: {:?}", duration);
Chayim Friedman
  • 47,971
  • 5
  • 48
  • 77
Scc
  • 446
  • 5
  • 4
  • 1
    Obligatory "Are you running in `--release` mode?". Also, are you sure that what's slow is the actual JSON parsing, and not just the receiving, with `reqwest` (you're using reqwest, right?) throwing away the body if you don't call `.json()`? – Caesar Jun 14 '23 at 02:48
  • 2
    On the latter point - I think that means "is there any measurable difference between using `.json()` and `.text()`?" (that is, between receiving and parsing vs just receiving). – Cerberus Jun 14 '23 at 03:46
  • Yes, most of the time is spent on parsing the JSON data. The JSON data returned is several tens of megabytes in size. If we only retrieve `result.unwrap().content_length()`, it will be faster. – Scc Jun 14 '23 at 06:18
  • It may be faster if you parse into a typed struct rather than `serde_json::Value`. – Chayim Friedman Jun 14 '23 at 06:19
  • @Chayim Friedman,I tried, but same result – Scc Jun 14 '23 at 06:28
  • Then maybe try a faster JSON parser, e.g. [simd-json](https://github.com/simd-lite/simd-json). Although I'm surprised the JSON parsing is the bottleneck, unless the data is really big. – Chayim Friedman Jun 14 '23 at 06:31
  • I will check this out,Thankyou. I tried JavaScript version, It's more faster – Scc Jun 14 '23 at 06:33
  • 3
    `content_length` does not need to retrieve the response data, only the headers. Benchmark again `result.unwrap().text()` and see if there is still a difference in speed. If not, the JSON parsing is not to blame. – Chayim Friedman Jun 14 '23 at 06:33
  • @Chayim Friedman,https://gist.github.com/ShiChenCong/f7e0d8759fd5677b7b9c19de75e3462d, seems still slow – Scc Jun 14 '23 at 06:37
  • 1
    Then the problem is not the JSON. – Chayim Friedman Jun 14 '23 at 06:38

0 Answers0