unable to convert json to polars dataframe,'read an Array from a non-Array JSON''

Question

i am struggling with conversion from JSON(deserialised from API-provicded data) to polars dataframe.

below are my structs and defined function:


//  structs for option price query
#[derive(Serialize, Deserialize, Debug)]
pub struct OptionResponse {
    pub tables: Vec<OptionTable>,
}

#[derive(Serialize, Deserialize, Debug)]
pub struct OptionTable {
    pub table: OptionDataTable,
}

#[derive(Serialize, Deserialize, Debug)]
pub struct OptionDataTable {
    pub volume: Vec<f64>,
    pub open: Vec<f64>,
    pub close: Vec<f64>,
    pub high: Vec<f64>,
    pub low: Vec<f64>,
}

pub async fn ifind_option_read(option_code: &str, access_token: &str, start_time_str: &str, end_time_str: &str) -> Result<(), Box<dyn std::error::Error>> {
    let url = "https://quantapi.51ifind.com/api/v1/high_frequency";

    // create the high-frequency prices request
    let request_data = maplit::hashmap! {
        "codes" => option_code,
        "indicators" => "volume,open,close,high,low",
        "starttime" => start_time_str,
        "endtime" => end_time_str,
    };
    let request_json = serde_json::to_string(&request_data)?;

    // send the HTTP POST request
    let client = reqwest::Client::new();
    let response = client
        .post(url)
        .header(reqwest::header::CONTENT_TYPE, "application/json")
        .header("access_token", access_token)
        .body(request_json)
        .send()
        .await?;
    let response_json: OptionResponse = response.json().await?;
    let response_body = &response_json.tables[0].table;
    println!("{:?}", &response_body);
    // Convert response_json to byte array
    let json_bytes = serde_json::to_vec(&response_body)?;
    // read the response into df, per method of JsonReader
    let df = JsonReader::new(Cursor::new(json_bytes))
        .with_batch_size(1024)
        .infer_schema_len(None)
        .finish().unwrap();
    println!("{:?}", df);
    Ok(())
}

the response_body returned looks very normal to me and looks like

OptionDataTable { volume: [0.0, 0.0, 0.0], open: [1263.0, 1263.0, 1263.0], close: [1263.0, 1263.0, 1263.0], high: [1263.0, 1263.0, 1263.0], low: [1263.0, 1263.0, 1263.0] }

neverthless the JsonReader cant read it as a df. this is confusing. is this a json format problem?

score 0 · Accepted Answer · answered Jul 04 '23 at 09:39

previously the deserialize function in polars/polars-json/src/json/deserialize.rs has a todo! macro causing this error.


pub fn deserialize(json: &BorrowedValue, data_type: DataType) -> Result<Box<dyn Array>, Error> {
    match json {
        BorrowedValue::Array(rows) => match data_type {
            DataType::LargeList(inner) => Ok(_deserialize(rows, inner.data_type)),
            _ => todo!("read an Array from a non-Array data type"),
        },
        _ => todo!("read an Array from a non-Array JSON"),
    }
}

now it is replaced with Ok(_deserialize(&[json], data_type))

with commit #207e0b9 everything should be fine now.

unable to convert json to polars dataframe,'read an Array from a non-Array JSON''

1 Answers1