0

I am trying to copy all files from cloud storage bucket recursively and I am having problem with the -m flag as I have investigated. The command that I am running

gsutil -m cp -r gs://{{ src_bucket }} {{ bucket_backup }}

I am getting something like this:

CommandException: 1 file/object could not be transferred.

where the number of files/objects differs every time.

After investigation I have tried to reduce number of threads/processes which used with the -m option, but this has not helped, so I am looking for some advice about this. I have 170 MiB data on the bucket which is approximately 300k files. I need to download them as fast as possible

UPD: Logs with -L flag

[Errno 2] No such file or directory: '<path>/en_.gstmp' -> '<path>/en'

6 errors like that.

DISCO
  • 135
  • 12

1 Answers1

1

The root of the issue might be that both directory and file of the same name exist in the GCS bucket. Try executing the command with -L flag, so you will get additional logs on the execution and you will be able to find the file that is causing this error. I would suggest you delete that file and make sure there is no directory in the bucket of that name and then upload this file to the bucket again.

Also check if any of the directory created with Jar name. Delete them and processed the copy files.

And check if the required file is already at destination and delete the file at destination and execute copy again.

There are alternatives to copy, for example, it is possible to transfer files using rsync, as described here.

You can also check similar threads: thread1 , thread2 & thread3

Roopa M
  • 2,171
  • 4
  • 12
  • Thank you for the suggestion of adding -L flag, now I have updated question with some logs – DISCO Jan 09 '23 at 14:23
  • Have a look at this [thread](https://stackoverflow.com/q/69443313/18265570)? Are you using Dataflow? – Roopa M Jan 10 '23 at 07:36