Stack Overflow
Stack Overflow

Questions tagged [gsutil]

gsutil is a command-line program that allows you to store, share and manage data hosted at cloud-based storage providers.

gsutil allows you to store, share and manage data hosted at cloud-based storage providers. It provides a simple Unix shell command-like syntax, operating over buckets and objects named with a URIs and wildcards.

868 questions
8
votes
2 answers

Google cloud: Using gsutil to download data from AWS S3 to GCS

One of our collaborators has made some data available on AWS and I was trying to get it into our google cloud bucket using gsutil (only some of the files are of use to us, so I don't want to use the GUI provided on GCS). The collaborators have…
bsmith
  • 389
  • 2
  • 3
  • 13
8
votes
3 answers

ResumableUploadAbortException on gsutil

I keep seeing the following error when using gsutil ResumableUploadAbortException: Upload complete with 6275 additional bytes left in stream The command is very simple, something like gsutil cp -r …
zyxue
  • 7,904
  • 5
  • 48
  • 74
8
votes
5 answers

Google Cloud Storage: Output path does not exist or is not writeable

I am trying to follow this simple Dataflow example from google cloud site. I have successfully installed the dataflow pipeline plugin and gcloud SDK (as well as Python 2.7). I have also set up a project on google cloud and enabled billing and all…
RoshP
  • 151
  • 1
  • 2
  • 9
8
votes
1 answer

Updating from user based configuration file

Whenever I run a bq or gsutil command, I get the below output: Loading user based configuration file: [/home/reggie/.config/gcloud/properties]. User based configuration files are deprecated and will not be read in a future gcloud release. What is…
Reggie Pawle
  • 83
  • 3
8
votes
4 answers

gsutil cp: copy files with -I option to matching subdirectories

I would like to copy a list of files to a bucket while keeping the directory-structure. test.txt: a/b/1.jpg a/c/23.jpg a/d/145.jpg gsutil command: cat file.txt | gsutil -m cp -I 'gs://my-bucket/' This copies the files but ignores the…
Markus Steinhauer
  • 81
  • 1
  • 4
8
votes
1 answer

gsutil cp command copies entire directory path (on Windows) - bug?

I am having a problem where gsutil does not seem to follow the behavior described in the documentation (at least in Windows). The documentation states: When performing recursive directory copies, object names are constructed that mirror the source…
user3610488
  • 81
  • 1
  • 2
  • 6
7
votes
4 answers

Receive error when I installing gcutil in MacOS

I followed the instructions given by the official documents, but stuck in executing ./install.sh. MacOS version:10.14.5 (18F132) Python version:python2.7 After I download the zip file given by the gcloud website, I run the command ./install.sh.It…
Weiziyoung
  • 191
  • 1
  • 2
  • 12
7
votes
1 answer

gsutil not working if calling from pycharm with python and subprocess, "cannot import name _common"

I have written a python script to automatize some gsutil operations. gsutil works fine if I run it by command line. But if I translate the same command with subprocess in python I get an error: BUCKET_NAME = 'datastore-backup' FOLDER_NAME =…
DarioB
  • 1,349
  • 2
  • 21
  • 44
7
votes
1 answer

Set metadata for all objects in a Google Storage bucket

I want to set Content-Type metadata to image/jpeg for all objects of a Google Storage bucket. How to do this?
porton
  • 5,214
  • 11
  • 47
  • 95
7
votes
3 answers

How to get list_blobs to behave like gsutil

I would like to only get the first level of a fake folder structure on GCS. If I run e.g.: gsutil ls 'gs://gcp-public-data-sentinel-2/tiles/' I get a list like…
cpaulik
  • 303
  • 1
  • 3
  • 13
7
votes
1 answer

How to skip existing files in gsutil rsync

I want to copy files between a directory on my local computer disk and my Google Cloud Storage bucket with the below conditions: 1) Copy all new files and folders. 2) Skip all existing files and folders irrespective of whether they have been…
Jyotsna
  • 326
  • 4
  • 15
7
votes
2 answers

result from "gsutil cp" is always in stderr, not stdout

As the title, the results from "gsutil cp" doesn't redirect to stdout, it always redirect to stderr. As an example : gsutil cp existed_file.txt . > >(tee -a out.log) 2> >(tee -a error.log >&2) . In above command, out.log is empty, and error.log has…
voxter
  • 853
  • 2
  • 14
  • 30
7
votes
3 answers

How to check if any given object exist in google cloud storage bucket through bash

I would like to pragmatically check if object exist at a perticular google cloud storage bucket. Based on object availability i would perform further operations. I have gone through https://cloud.google.com/storage/docs/gsutil/commands/stat and…
Virat
  • 137
  • 2
  • 10
7
votes
2 answers

gsutil command to delete old files from last day

I have a bucket in google cloud storage. I have a tmp folder in bucket. Thousands of files are being created each day in this directory. I want to delete files that are older than 1 day every night. I could not find an argument on gsutil for this…
spala
  • 85
  • 1
  • 2
  • 7
7
votes
4 answers

Google Cloud Storage: How to get list of new files in bucket/folder using gsutil

I have a bucket/folder into which a lot for files are coming in every minutes. How can I read only the new files based on file timestamp. eg: list all files with timestamp > my_timestamp
Remis Haroon - رامز
  • 3,304
  • 4
  • 34
  • 62
1 2 3
…
57 58