reqwest post request freezes after random amount of time

Question

I started learning rust 2 weeks ago, and has been making this application that watches a log file, and sends a bulk of the information to an elasticsearch DB.

The problem is that after certain amount of time, it freezes (using 100% CPU) and I don't understand why.

I've cut down on a lot of code to try to figure out the issue, but it still keeps freezing on this line according to clion debugger

let _response = reqwest::Client::new()
    .post("http://127.0.0.1/test.php")
    .header("Content-Type", "application/json")
    .body("{\"test\": true}")
    .timeout(Duration::from_secs(30))
    .send() // <-- Exactly here
    .await;

It freezes and doesn't return any error message.

This is the code in context:

use std::{env};
use std::io::{stdout, Write};
use std::path::Path;
use std::time::Duration;
use logwatcher::{LogWatcher, LogWatcherAction};
use serde_json::{json, Value};
use serde_json::Value::Null;
use tokio;

#[tokio::main]
async fn main() {
    let mut log_watcher = LogWatcher::register("/var/log/test.log").unwrap();
    let mut counter = 0;
    let BULK_SIZE = 500;

    log_watcher.watch(&mut move |line: String| { // This triggers each time a new line is appended to /var/log/test.log
        counter += 1;

        if counter >= BULK_SIZE {
            futures::executor::block_on(async { // This has to be async because log_watcher is not async

                let _response = reqwest::Client::new()
                    .post("http://127.0.0.1/test.php") // <-- This is just for testing, it fails towards the DB too
                    .header("Content-Type", "application/json")
                    .body("{\"test\": true}")
                    .timeout(Duration::from_secs(30))
                    .send() // <-- Freezes here
                    .await;

                if _response.is_ok(){
                    println!("Ok");
                }
            });
            counter = 0;
        }
        LogWatcherAction::None
    });
}

The log file gets about 625 new lines every minute. The crash happends after about ~5500 - ~25000 lines has gone through, or it seems a bit random in general.

I'm suspecting the issue is either something to do with LogWatcher, reqwest, the block_on or the mix of async.

Does anyone have any clue why it randomly freezes?

I'm not familiar with how log-watcher works, but you are mixing sync and async operations in a way that might be causing a problem. If this is the whole application, I'd probably recommend ditching tokio and futures entirely and just use reqwest's [blocking](https://docs.rs/reqwest/latest/reqwest/blocking/index.html) client (instead of async client). Well I just saw you mentioned talking to a database too so maybe nevermind. — kmdreko, Sep 21 '22 at 20:30
This is the main gist of the issue, the whole application is a bit bigger. Removing tokio, and moving async calls inside a `block_on` brings some more issues and challanges `thread 'main' panicked at 'there is no reactor running, must be called from the context of a Tokio 1.x runtime'` — Typewar, Sep 21 '22 at 21:01
Calling futures executor inside tokio runtime is probably not a good idea. To run sync code in tokio you can use `spawn_blocking` (put your log watcher loop in there) and from there, to execute async code either use `tokio::spawn` or send your data through a channel or use sync reqwest client or etc. — stepan, Sep 21 '22 at 21:40
Once the `log_watcher.watch(` scope is opened, it's no longer async inside of it. How would I go about wrapping it with `spawn_blocking` ? Because awaiting tokio spawn wouldn't be possible when inside the `log_watcher.watch(` — Typewar, Sep 21 '22 at 22:03
Alright, I changed the code quite a bit, changed main to be sync instead of async, and replaced the `futures::executor::block_on` with tokio's block_on function instead. It hasn't frozen the past hour atleast, can't confirm just yet. — Typewar, Sep 22 '22 at 05:21

score 0 · Accepted Answer · answered Sep 23 '22 at 21:39

The problem was indeed because of a mix of async with tokio and block_on, NOT directly reqwest.

The problem was solved when changing main to be non-async, and using tokio as the block_on for async calls instead of futures::executor::block_on.

fn main() {
    let mut log_watcher = LogWatcher::register("/var/log/test.log").unwrap();
    let mut counter = 0;
    let BULK_SIZE = 500;

    log_watcher.watch(&mut move |line: String| {
        counter += 1;

        if counter >= BULK_SIZE {
            tokio::runtime::Builder::new_multi_thread()
                .enable_all()
                .build()
                .unwrap()
                .block_on(async {

                    let _response = reqwest::Client::new()
                        .post("http://127.0.0.1/test.php")
                        .header("Content-Type", "application/json")
                        .body("{\"test\": true}")
                        .timeout(Duration::from_secs(30))
                        .send()
                        .await;

                    if _response.is_ok(){
                        println!("Ok");
                    }
            });
            counter = 0;
        }
        LogWatcherAction::None
    });
}

reqwest post request freezes after random amount of time

1 Answers1