Multiple S3 credentials in a Spark Structured Streaming application

Question

I want to migrate our Delta lake from S3 to Parquet files in our own on-prem Ceph storage, both accessible through the S3-compliant s3a API in Spark. Is there a possibility to provide different credentials for readStream and writeStream to achieve this?

score 2 · Accepted Answer · answered Feb 07 '22 at 11:50

2

the s3a connector supports per-bucket configuration, so you can declare a different set of secrets, endpoint etc for your internal buckets from your external ones.

consult the hadoop s3a docs for the normative details and examples

answered Feb 07 '22 at 11:50

stevel

12,567
1
39
50

Multiple S3 credentials in a Spark Structured Streaming application

1 Answers1