Dynamically receive json data in rust with reqwest

Question

Ive been trying to receive json data with reqwest and serde but I keep getting the error:

Error: reqwest::Error { kind: Decode, source: Error("expected value", line: 1, column: 1) }

This is my code so far:

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let url: String = String::from("https://api.slothpixel.me/api/players/leastrio");
    let echo_json: serde_json::Value = reqwest::Client::new()
        .get(url)
        .send()
        .await?
        .json()
        .await?;
    
    println!("{:#?}", echo_json);
    Ok(())
}

reqwest = { version = "0.11", features = ["json"] }
tokio = { version = "1", features = ["full"] }
serde_json = "1"

Are you sure it returns a JSON? Maybe some other format / maybe you didn't include necessary request headers — Alexey S. Larionov, Jul 13 '21 at 07:13
If you change it to use .text() insteda of .json, I get a "error 1020". Maybe you need to tweak a parameter in the client? Doing a fetch with javascript works fine... — ShadowMitia, Jul 13 '21 at 07:47
It looks like this server requires a `User-Agent` to be specified, but `Reqwest` does not add one by default: `curl -v -A '' https://api.slothpixel.me/api/players/leastrio` returns `HTTP 403` while `curl -v -A 'Foo' https://api.slothpixel.me/api/players/leastrio` returns `HTTP 200`. — rodrigo, Jul 13 '21 at 07:49
@rodrigo odd, anyway still a bug the error should be a json asking to provide a User-Agent — Stargateur, Jul 13 '21 at 07:50

score 1 · Accepted Answer · answered Jul 13 '21 at 07:55

1

So I've trie a few things, and it seems you need to add a user agent for it to work. No idea why the documentation doesn't mention it. And I guess reqwest doesn't provied one by default.

reqwest::Client::new()
        .get(url)
        .header("User-Agent", "Reqwest Rust Test")
        .send()
        .await?
        .json()
        .await?;

I used this and it worked!

answered Jul 13 '21 at 07:55

ShadowMitia

2,411
1
19
24

1

If you use a `ClientBuilder` you have a nicer `user_agent()` setter: `reqwest::Client::builder().user_agent("Test").build().unwrap()`. – rodrigo Jul 13 '21 at 08:00

score 0 · Answer 2 · answered Jul 13 '21 at 07:47

It should work a quick test with:

tracing = "0.1"
tracing-subscriber = "0.2"

Adding to main:

let subscriber = tracing_subscriber::FmtSubscriber::builder()
    .with_max_level(tracing::Level::TRACE)
    .finish();

tracing::subscriber::set_global_default(subscriber)
    .expect("setting default subscriber failed");

dbg!(reqwest::Client::new().get(&url).send().await?.text().await);

RUST_LOG=trace cargo run

Jul 13 09:45:59.232 TRACE hyper::client::pool: checkout waiting for idle connection: ("https", api.slothpixel.me)
Jul 13 09:45:59.234 TRACE hyper::client::connect::http: Http::connect; scheme=Some("https"), host=Some("api.slothpixel.me"), port=None
Jul 13 09:45:59.234 DEBUG hyper::client::connect::dns: resolving host="api.slothpixel.me"
Jul 13 09:45:59.277 DEBUG hyper::client::connect::http: connecting to [2606:4700:3036::6815:5b3]:443
Jul 13 09:45:59.301 DEBUG hyper::client::connect::http: connected to [2606:4700:3036::6815:5b3]:443
Jul 13 09:45:59.352 TRACE hyper::client::conn: client handshake Http1
Jul 13 09:45:59.353 TRACE hyper::client::client: handshake complete, spawning background dispatcher task
Jul 13 09:45:59.353 TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: Init, keep_alive: Busy }
Jul 13 09:45:59.353 TRACE hyper::client::pool: checkout dropped for ("https", api.slothpixel.me)
Jul 13 09:45:59.354 TRACE encode_headers: hyper::proto::h1::role: Client::encode method=GET, body=None
Jul 13 09:45:59.355 DEBUG hyper::proto::h1::io: flushed 76 bytes
Jul 13 09:45:59.355 TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: KeepAlive, keep_alive: Busy }
Jul 13 09:45:59.376 TRACE hyper::proto::h1::conn: Conn::read_head
Jul 13 09:45:59.377 TRACE parse_headers: hyper::proto::h1::role: Response.parse([Header; 100], [u8; 953])
Jul 13 09:45:59.377 TRACE parse_headers: hyper::proto::h1::role: Response.parse Complete(937)
Jul 13 09:45:59.378 DEBUG hyper::proto::h1::io: parsed 14 headers
Jul 13 09:45:59.378 DEBUG hyper::proto::h1::conn: incoming body is content-length (16 bytes)
Jul 13 09:45:59.378 TRACE hyper::proto::h1::decode: decode; state=Length(16)
Jul 13 09:45:59.379 DEBUG hyper::proto::h1::conn: incoming body completed
Jul 13 09:45:59.379 TRACE hyper::proto::h1::conn: maybe_notify; read_from_io blocked
Jul 13 09:45:59.379 TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: Init, keep_alive: Idle }
Jul 13 09:45:59.379 TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: Init, keep_alive: Idle }
Jul 13 09:45:59.380 TRACE hyper::client::pool: put; add idle connection for ("https", api.slothpixel.me)
Jul 13 09:45:59.380 DEBUG hyper::client::pool: pooling idle connection for ("https", api.slothpixel.me)
[Jul 13 09:45:59.380 TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: Init, keep_alive: Idle }
src\main.rs:12] reqwest::Client::new().get(&url).send().await?.text().await = Ok(
    "error code: 1020",
)
Jul 13 09:45:59.381 TRACE hyper::proto::h1::dispatch: client tx closed
Jul 13 09:45:59.381 TRACE hyper::client::pool: pool closed, canceling idle interval
Jul 13 09:45:59.382 TRACE hyper::client::pool: checkout waiting for idle connection: ("https", api.slothpixel.me)
Jul 13 09:45:59.382 TRACE hyper::proto::h1::conn: State::close_read()
Jul 13 09:45:59.382 TRACE hyper::client::connect::http: Http::connect; scheme=Some("https"), host=Some("api.slothpixel.me"), port=None
Jul 13 09:45:59.382 TRACE hyper::proto::h1::conn: State::close_write()
Jul 13 09:45:59.382 DEBUG hyper::client::connect::dns: resolving host="api.slothpixel.me"
Jul 13 09:45:59.383 TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Closed, writing: Closed, keep_alive: Disabled }
Jul 13 09:45:59.383 DEBUG hyper::client::connect::http: connecting to [2606:4700:3036::6815:5b3]:443
Jul 13 09:45:59.383 TRACE hyper::proto::h1::conn: shut down IO complete
Jul 13 09:45:59.396 DEBUG hyper::client::connect::http: connected to [2606:4700:3036::6815:5b3]:443
Jul 13 09:45:59.428 TRACE hyper::client::conn: client handshake Http1
Jul 13 09:45:59.428 TRACE hyper::client::client: handshake complete, spawning background dispatcher task
Jul 13 09:45:59.429 TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: Init, keep_alive: Busy }
Jul 13 09:45:59.429 TRACE hyper::client::pool: checkout dropped for ("https", api.slothpixel.me)
Jul 13 09:45:59.430 TRACE encode_headers: hyper::proto::h1::role: Client::encode method=GET, body=None
Jul 13 09:45:59.430 DEBUG hyper::proto::h1::io: flushed 76 bytes
Jul 13 09:45:59.430 TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: KeepAlive, keep_alive: Busy }
Jul 13 09:45:59.451 TRACE hyper::proto::h1::conn: Conn::read_head
Jul 13 09:45:59.451 TRACE parse_headers: hyper::proto::h1::role: Response.parse([Header; 100], [u8; 953])
Jul 13 09:45:59.452 TRACE parse_headers: hyper::proto::h1::role: Response.parse Complete(937)
Jul 13 09:45:59.452 DEBUG hyper::proto::h1::io: parsed 14 headers
Jul 13 09:45:59.452 DEBUG hyper::proto::h1::conn: incoming body is content-length (16 bytes)
Jul 13 09:45:59.453 TRACE hyper::proto::h1::decode: decode; state=Length(16)
Jul 13 09:45:59.453 DEBUG hyper::proto::h1::conn: incoming body completed
Jul 13 09:45:59.453 TRACE hyper::proto::h1::conn: maybe_notify; read_from_io blocked
Jul 13 09:45:59.453 TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: Init, keep_alive: Idle }
Jul 13 09:45:59.454 TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: Init, keep_alive: Idle }
Jul 13 09:45:59.454 TRACE hyper::client::pool: put; add idle connection for ("https", api.slothpixel.me)
Jul 13 09:45:59.454 DEBUG hyper::client::pool: pooling idle connection for ("https", api.slothpixel.me)
Jul 13 09:45:59.454 TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: Init, keep_alive: Idle }
Jul 13 09:45:59.454 TRACE hyper::client::pool: pool closed, canceling idle interval
Jul 13 09:45:59.454 TRACE hyper::proto::h1::dispatch: client tx closed
Jul 13 09:45:59.455 TRACE hyper::proto::h1::conn: State::close_read()
Jul 13 09:45:59.455 TRACE hyper::proto::h1::conn: State::close_write()
Jul 13 09:45:59.455 TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Closed, writing: Closed, keep_alive: Disabled }
Jul 13 09:45:59.456 TRACE hyper::proto::h1::conn: shut down IO complete
Error: reqwest::Error { kind: Decode, source: Error("expected value", line: 1, column: 1) }

error code: 1020 not even a json object contrary to https://api.slothpixel.me/api/players/ that return a json object "error", I suggest a bug report this to https://github.com/slothpixel/core or where it's adequate cause this error is odd.

Dynamically receive json data in rust with reqwest

2 Answers2