5

I can't access my files on S3 using a dataframe read : df_read_csv. I get the error: Exception: Unable to locate credentials

This works fine when my dask distributed is running against local worker cores. However, when I import a client with a cluster of additional worker servers it fails. My cluster was created using dask-ec2 on ubuntu with 1 scheduler on the header server and 3 worker servers (all ubuntu).

I am assuming that the failure is because all the workers also need access to S3. I have installed the aws cli on all of them and connected using my key and can list the S3 buckets from the cli. However, for some reason my dataframe read throws an ubuntu error saying that boto can't locate credentials

I have trawled through various posts but can't seem to find anything that helps. This is screenshot of the error:

Error

Cristik
  • 30,989
  • 25
  • 91
  • 127
Matt Barnes
  • 51
  • 3

1 Answers1

3

Hi Matt you should add key and secret to storage_options. In my case I have them saved on settings.py

import dask.dataframe as dd
import settings

df = dd.read_csv("s3://gra-dask-ec2/us_cities.csv",
                  storage_options={'key': settings.AWS_ACCESS_KEY_ID,
                                   'secret': settings.AWS_SECRET_ACCESS_KEY})
rpanai
  • 12,515
  • 2
  • 42
  • 64