Training data in S3 in AWS Sagemaker

Question

I've uploaded my own Jupyter notebook to Sagemaker, and am trying to create an iterator for my training / validation data which is in S3, as follow:

train = mx.io.ImageRecordIter(
        path_imgrec         = ‘s3://bucket-name/train.rec’ …… )

I receive the following exception:

MXNetError: [04:33:32] src/io/s3_filesys.cc:899: Need to set enviroment variable AWS_SECRET_ACCESS_KEY to use S3

I've checked that the IAM role attached with this notebook instance has S3 access. Any clues on what might be needed to fix this?

score 4 · Accepted Answer · answered Mar 29 '18 at 06:37

If your IAM roles are setup correctly, then you need to download the file to the Sagemaker instance first and then work on it. Here's how:

# Import roles
import sagemaker
role = sagemaker.get_execution_role()

# Download file locally
s3 = boto3.resource('s3')
s3.Bucket(bucket).download_file('your_training_s3_file.rec', 'training.rec')

#Access locally
train = mx.io.ImageRecordIter(path_imgrec=‘training.rec’ …… )

Training data in S3 in AWS Sagemaker

1 Answers1