I'm completely new to Rust (but experienced coder in general) and I'm struggling with the following problem:
I want to load a Parquet file stored on a Digital Ocean S3 bucket into a polars
DataFrame
object. (Ideally at a later point I will switch toLazyFrame
.)
I'm already using rust-s3
crate to access the S3 successfully, but I'm struggling to find a way to adapt that API to polars (polars seems to be rather opinionated about the backing store being mmap-capable).
I see I may need to use polars own cloud
module to access the S3 rather than be able to reuse my rust-s3
code. This is the start of my solution:
use polars::prelude::*;
async fn load_dataframe() {
let uri = "s3://bucket/path";
let settings = CloudOptions::from_untyped_config(uri, vec![
("endpoint_url", "https://ams3.digitaloceanspaces.com".to_string()),
("access_key", "elided".to_string()),
("secret_key", "elided".to_string()),
]);
let _reader = polars::io::parquet::ParquetAsyncReader::from_uri(uri, Some(settings));
}
but this fails to compile with the following error:
Compiling tranquility v0.1.0 (/work)
error[E0599]: no function or associated item named `from_untyped_config` found for struct `hashbrown::map::HashMap` in the current scope
--> src/trial.rs:7:34
|
7 | let settings = CloudOptions::from_untyped_config(uri, vec![
| ^^^^^^^^^^^^^^^^^^^ function or associated item not found in `hashbrown::map::HashMap<std::string::String, std::string::String, polars::export::ahash::RandomState>`
For more information about this error, try `rustc --explain E0599`.
error: could not compile `tranquility` due to previous error
I've been going round in circles for several days now trying different ways to approach the broader problem, or work through the specific complier error, but I'm hitting a wall, particularly given how to construct the CloudOptions
object seems to be undocumented. I've tried ChatGPT too.
Solutions I don't want to consider because they are not compatible with the project requirements:
- Downloading the parquet file to local storage first.
Versions:
polars
: 0.28.0 (both polars-core & polars-io)rust-s3
: 0.33.0
Thanks in advance.