0

aws s3 sync provides a parameter named --no-sign-request, so user can sync a publicly-visible bucket without any auth in advance.

gsutil help anon says that:

gsutil users can access publicly readable data without obtaining credentials. For example, the gs://uspto-pair bucket contains a number of publicly readable objects, so any user can run the following command without first obtaining credentials:

gsutil ls gs://uspto-pair/applications/0800401*

Users can similarly download objects they find via the above gsutil ls command.

However, running this command without login makes gsutil report errors:

➜  mirror-docker git:(v2) gsutil ls 'gs://uspto-pair/applications/0800401*'
Your credentials are invalid. Please run
$ gcloud auth login

Currently, we are setting up a mirror for common buckets on gcloudstorage, and want to find a way to perform sync without auth in advance. Is it possible?

Sunny J
  • 607
  • 3
  • 14
lz96
  • 131
  • 5

2 Answers2

1

Prior using the gsutil tool, you need to authenticate yourself as per this: https://cloud.google.com/storage/docs/quickstart-gsutil#before-you-begin which is one of the requirements. Even if you have a publicly accessible bucket and/or object you cannot unfortunately perform a sync without authentication in advance.

Steeve
  • 111
  • 2
0

@Steeve 's answer that authentication is required is correct. The following is a simple script to sync a publicly-visible bucket to local. Just create a new account with no bill method and a new project/key pair:

#!/bin/bash

# path: sync destination
# access_key_id: https://cloud.google.com/storage/docs/migrating#keys
# access_secret_id: Same as above
# project_id: The project name where you create a new key
# source: source googlestorage path. Example: gs://flutter_intra

mkdir -p "$LUG_path"
env "GS_ACCESS_KEY_ID=$LUG_access_key_id" "GS_ACCESS_SECRET_KEY=$LUG_access_secret_key" ./google-cloud-sdk/bin/gsutil  -o "GSUtil:default_project_id=$LUG_project_id" -m rsync -d -r "$LUG_source" "$LUG_path"
lz96
  • 131
  • 5