How to read parquet file from AWS S3 bucket using R without downloading it locally?

Question

I'm able to download the parquet file from AWS S3 bucket on local and then read from it (see the below code). But is there any way that I can directly read parquet file from S3 and read, without storing in local?

save_object("Financial_Sample.parquet", file = "Financial_Sample.parquet", bucket = 'my-bucket')
df <- read_parquet("Financial_Sample.parquet")```

score 1 · Answer 1 · answered Oct 28 '21 at 07:14

1

Take a look at the arrow package: https://arrow.apache.org/docs/r/index.html

It can directly read from s3 and even filter before reading using some dplyr verbs.

answered Oct 28 '21 at 07:14

jsch

86
3

How to read parquet file from AWS S3 bucket using R without downloading it locally?

1 Answers1