0

I successfully have my: Access Key ID and Secret Access Key.

I have an S3 URL, of this path length/ structure:

s3://parent_filepath/foo/bar/folder_of_interest/

Update: I thought I was able to download everything via. Console, but it turns out you cannot download sub-folders and their contents recursively. So this task would need to be done in my Python application.

How can I download folder_of_interest via. Python, using the below answer to existing post?:

Specifically, what values from my S3 URL do I place into keys: name and location as well as code underneath the dictionary?

Code:

pip install d6tpipe

import d6tpipe
api = d6tpipe.api.APILocal() # keep permissions locally for security

settings = \
{
    'name': 'folder_of_interest',
    'protocol': 's3',
    'location': '',
    'readCredentials' : {
        'aws_access_key_id': '###',
        'aws_secret_access_key': '###'
    }
}

d6tpipe.api.create_pipe_with_remote(api, settings)

pipe = d6tpipe.Pipe(api, 'folder_of_interest')
pipe.scan_remote() # show all files
pipe.pull_preview() # preview
#pipe.pull(['']) # download single file
pipe.pull() # download all files

pipe.files() # show files
#file=open(pipe.dirpath/'') # access file

Traceback:

Welcome to d6tpipe!
auto created profile "default", see docs how to customize profile
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-5-cbb589dcee20> in <module>()
     14 }
     15 
---> 16 d6tpipe.api.create_pipe_with_remote(api, settings)
     17 
     18 pipe = d6tpipe.Pipe(api, 'folder_of_interest')

AttributeError: module 'd6tpipe.api' has no attribute 'create_pipe_with_remote'
  • From your linked answer: `name` - name of file/s3 object, `location` - bucket name – Justinas Aug 26 '21 at 10:01
  • Does this answer your question? [Download file from AWS S3 using Python](https://stackoverflow.com/questions/50100221/download-file-from-aws-s3-using-python) – Justinas Aug 26 '21 at 10:02
  • So I've basing my post's solution on an answer given in that thread. Given my S3 URL: `s3://parent_filepath/foo/bar/dataset_name/`; what values in it (e.g. `foo`) are used for which values in the code? @Justinas –  Aug 26 '21 at 10:06
  • Sorry for my little understanding. I've updated the code in post and now have an error. –  Aug 26 '21 at 10:13

0 Answers0