0

I'm using gsutil on Windows. Sometimes, when copying files from my PC to the Google Cloud Storage bucket I will get messages such as:

  • "CommandException: Some temporary components were not uploaded successfully. Please retry this upload"
  • "CommandException: n files/objects could not be transferred."

I'm working with the -L parameter but, currently, I cannot effectively trap the mentioned errors - I know this is possible using -c and -L on Linux environments. Instead, I loop thru the gsutil cp -L command about 10 times which, in practice, does the trick and ends up copying all the files.

Is there a better way to handle this command exceptions so I can trap if gsutil fails and retry the operation till it finally copies all the files to the destination bucket? Some kind of errorlevel feature?

tshepang
  • 12,111
  • 21
  • 91
  • 136
user2980216
  • 41
  • 2
  • Can you please clarify: When you say "I loop thru the gsutil cp -L command about 10 times which, in practice, does the trick and ends up copying all the files" -- are you saying that works for you on Linux but not on Windows? Or are you saying it works on Windows but you'd like to find a way that avoids having to do so many retries? – Mike Schwartz Nov 11 '13 at 18:53
  • Hi Mike! I am working on Windows and the fact is that 10 retries of the same gsutil cp -L will be enough to copy all the files but, as you imply on the second part of your question, I'd like to find a better way to trap these command exceptions so I can retry the minimum amount of times. For example, something like: :loop gsutil cp -L cp.log ... if there are command exceptions goto loop :exit I have seen there is a way to handle this in Linux but have not seen it for Windows. – user2980216 Nov 11 '13 at 21:10
  • So just to be clear: gsutil cp -L will only re-copy the files that didn't succeed before (it maintains a manifest of what has succeeded in the log file it writes). So, it actually should be pretty efficient, even with this many times through your loop. I am curious, though - do you know what's failing that forces so many retries? Are you using a network that has intermittent problems? If you're not sure, please try running gsutil -D cp ... That will output a bunch of debugging info. If you post it please remove the Authorization: headers. – Mike Schwartz Nov 11 '13 at 22:59
  • Mike, I am having network related issues during file transfers of 512 MB or more per file. The good news is that I have found that redirecting stderr from gsutil to a file using >2 I can later check for the potential ocurrences of CommandException in that file and loop until no ocurrences are found. Anyway, thank you very much for your time and interest on this issue. – user2980216 Nov 12 '13 at 13:37

0 Answers0