3

I am trying to read a file from s3 bucket with is in another AWS Account.

Steps:

  1. Generated access id, access key secret and token using aws sts assume-role command

  2. Set following variables:

    export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=
export AWS_SESSION_TOKEN=

  3. aws s3 ls s3://...

  4. reading file in spark-shell

    sc.hadoopConfiguration.set("fs.s3n.awsAccessKeyId", "XXX")
sc.hadoopConfiguration.set("fs.s3n.awsSecretAccessKey", "XXX")
sc.hadoopConfiguration.set("fs.s3a.session.token", "XXX")
spark.read.parquet("file location")

Up to step 3 its working fine.

I am unable to read the file in spark-shell. I am using spark version: 2.4.3. Please let me know what i am missing. Any help would be appreciated.

Error log: 

    Caused by: com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.model.AmazonS3Exception: The AWS Access Key Id you provided does not exist in our records.
bob
  • 4,595
  • 2
  • 25
  • 35

1 Answers1

0

Seems that your credential was deleted.

You can also see: https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/security-cross-account.html

Paulo Moreira
  • 411
  • 5
  • 13