1

I am trying to use the arrow R package to read a parquet file from s3. The documentation only describes how to specifying AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY when authenticating for access to private s3 bucket.

However, I have to generate keys using okta which additionally provides AWS_SESSION_TOKEN & AWS_SECURITY_TOKEN values as part of my profile.

How can arrow in R be instructed to use the AWS_SESSION_TOKEN value and AWS_SECURITY_TOKEN value as part of authentication? Or, how can arrow in R be instructed to use a profile instead?

I have tried setting environment variables:

Sys.setenv(AWS_ACCESS_KEY_ID="...", AWS_SECRET_ACCESS_KEY="...", AWS_SESSION_TOKEN="...", AWS_SECURITY_TOKEN="...")

library("arrow")

read_parquet("s3://bucket/file.extension?region=us-east-1")

But this results in

AWS Error [code 15]: No response body.

if I try to set the environment variable AWS_DEFAULT_PROFILE then the same error occurs.

Thank you in advance for your consideration and response.

Ramón J Romero y Vigil
  • 17,373
  • 7
  • 77
  • 125
  • 1
    Hmm, I know it picks up credentials from `~/.aws/credentials`, and the aws-sdk-cpp (which arrow uses) also supports picking up credentials from these env vars: https://github.com/aws/aws-sdk-cpp/blob/bb1fdce01cc7e8ae2fe7162f24c8836e9d3ab0a2/aws-cpp-sdk-core/include/aws/core/auth/AWSCredentialsProvider.h#L128. Could you report an issue at https://issues.apache.org/jira/browse/ARROW? – Neal Richardson Jul 13 '22 at 13:00

0 Answers0