How to write df as parquet to s3 in polars rust?

Question

I am a bit out of ideas as after doing this

let mut cursor = Cursor::new(Vec::new());
let parquet_bytes = ParquetWriter::new(cursor)
    .with_statistics(true)
    .with_compression(ParquetCompression::Snappy)
    .finish(df)
    .unwrap();

I no longer own cursor to pass to s3 sdk. is there a different way?

score 0 · Answer 1 · answered Apr 11 '23 at 14:58

0

Have you tried passing cursor as a mutable reference? Something like this:

    let s3_client = aws_sdk_s3::Client::new();

    let mut df = df!("a" => &[1,2]).unwrap();
    let mut buf = vec![];
   
    ParquetWriter::new(&mut buf).finish(&mut df).unwrap();

    s3_client
        .put_object()
        .bucket("my-bucket-data")
        .key("tmp.parquet")
        .body(buf.into())
        .send()
        .await
        .unwrap();

answered Apr 11 '23 at 14:58

Jandre Marais

318
2
9

since this write to an in-mem buffer and then spills to the object store, would this be the most performant way to do writes ? – Chitral Verma Apr 12 '23 at 10:04
I'm not aware of alternatives for Polar DF -> AWS S3 parquet – Jandre Marais Apr 13 '23 at 14:46

How to write df as parquet to s3 in polars rust?

1 Answers1