0

Hello I am trying to download data from one of an Amazon S3 public bucket.

For example https://registry.opendata.aws/noaa-gfs-bdp-pds/

The bucket has web accessible folder and I want to download the files inside the bucket. I know I can do this with AWS CLI tool But I want to know if there anyway to do this with AWs SDK Api (s3 client) (c# visual studio)?

I think the issue is authentication when creating connection to s3 client it requires credentials like access key ,I don't have an AWS account,and the bucket I try to get to is public so
Does anyone know how to access to this public bucket without any credentials via API?

Thanks.

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
Liz Meltzer
  • 1
  • Even if the bucket is public, you need to "know" which object you want to download to, via api/sdk you can execute the function ListObjects if I recall correctly to get the list of it, but if you want to see the bucket as a site like the opendata one, you also need to setup your bucket as a "static website". https://docs.aws.amazon.com/AmazonS3/latest/userguide/WebsiteHosting.html – Juan Fontes Oct 12 '21 at 13:04
  • Yes I want to use function ListObject but this requires to create AmazonS3Client() and pass credentials which I don't have because its public bucket..and I don't have aws account..I'm also this exception unable to get iam security credentials from "ec2 instance metadata service" dont understand why I need any credentials and from where I get them if I want to access public aws bucket ‍♀️ – Liz Meltzer Oct 12 '21 at 13:20
  • @smac2020 you should theoretically be able to make unsigned requests. Both awscli (using `--no-sign-request`) and boto3 (see [here](https://stackoverflow.com/questions/34865927/can-i-use-boto3-anonymously)) support this, for example. Other SDKs may also support this, but I could not immediately find it in the .Net documentation. – jarmod Oct 12 '21 at 14:00
  • @smac2020 if that was a response to me, you're missing the point of my comment. You mentioned earlier that you cannot use the AWS SDK without an AWS account. That's not correct. I'm saying that you can use AWS SDKs (and the AWSCLI) without an AWS account, specifically in the case that unauthenticated/anonymous access is permitted e.g. to an S3 bucket. – jarmod Oct 12 '21 at 14:09
  • Thank you both for answers. @smac2020 do you familiar with any solution for me? – Liz Meltzer Oct 12 '21 at 14:46

1 Answers1

1

If you specify the AnonymousAWSCredentials as the credentials object, any requests that are made to S3 will be unsigned. After that, interacting with the bucket is done like any other call:

using Amazon.Runtime;
using Amazon.S3;
using Amazon.S3.Model;
using System;

namespace S3TestApp
{
    class Program
    {
        static void Main(string[] args)
        {
            var unsigned = new AnonymousAWSCredentials();
            var client = new AmazonS3Client(unsigned, Amazon.RegionEndpoint.USEast1);

            var listRequest = new ListObjectsRequest
            {
                BucketName = "noaa-gfs-bdp-pds",
                Delimiter = "/",
            };

            ListObjectsResponse listResponse;
            do
            {
                listResponse = client.ListObjects(listRequest);
                foreach (var obj in listResponse.CommonPrefixes)
                {
                    Console.WriteLine("PRE {0}", obj);
                }
                foreach (var obj in listResponse.S3Objects)
                {
                    Console.WriteLine("{0} {1}", obj.Size, obj.Key);
                }
                listRequest.Marker = listResponse.NextMarker;
            } while (listResponse.IsTruncated);

        }
    }
}
Anon Coward
  • 9,784
  • 3
  • 26
  • 37