1

We recently did an "on-premises transfer" from an old server being decommissioned into a Google Cloud Storage bucket connected via docker. The total size of the transfer was 365GB. The console shows that the transfer was 100% complete with 365GB worth of data being transferred, but the console lists the transfer as failed because "2 files failed to transfer". However, nowhere does it list which files failed. The error details page has "no results" and the transfer summary page does not list them. I suspect the files are inconsequential but I would like to check. How do I find out which files failed to transfer?

VirtualMitra
  • 93
  • 7

2 Answers2

1

As your question requires looking your logs in more detail to determine why this transfer fails

I recommend you to contact Google Cloud Platform technical support in order to get this issue resolved

You can check these support channels and raise your Cloud Storage technical issues there.

If you just want to see your logs and you don’t require a deeper investigation you just need to find your logs in the selected directory.

As per the public documentation in order to store the logs of the transfer you have to use the --log-dir=logs-directory flag. Then you will be able to inspect your transfer logs to determine why this files failed. If you didn’t use this flag then you will need to run the transfer job again

There is a complete [troubleshooting section]( https://cloud.google.com/storage-transfer/docs/troubleshooting-on-prem ) in the public documentation in the case that you want to know how to solve specific issues with your transfer

Chris32
  • 4,716
  • 2
  • 18
  • 30
  • 1
    I was hoping there was a way I could just look at the logs without having to pay for support. I'm not sure why, if there are logs of my transfer, I can't access them directly? – VirtualMitra Mar 05 '20 at 22:13
  • Hi, yes but you can't only access the logs if you choose to store them. I just updated my answer – Chris32 Mar 06 '20 at 08:14
1

Just to clarify VirtualMitra's comment - in addition to local logs that the Transfer Service generate on the machines where agents are running (this contains mostly info on what happened with individual agent processes), the product tracks per-file status for troubleshooting purposes.

There are two ways to see transfer errors:

  1. You can see a quick, at-a-glance view of a small sampling of errors in the GUI directly. You can follow the instructions here on how to see these error samples in the job details page.
  2. If you want to process all errors, you can see a full list of all errors in the transfer log, in a formatted TSV file with a lot of detailed information on what happened to every file, including when the file was visited, the checksums computed, if the file was copied successfully, and any error messages that weren't resolved via automatic retries. You can find more information on where these logs are stored and the format of the data stored in the logs. These objects are generated on every transfer that is executed and can be loaded into the tool of your choice for detailed troubleshooting/querying if needed (e.g. BigQuery).

All of the above are directly accessible by the transfer user, you don't need to go through support to access this data.

Ash Ahluwalia
  • 43
  • 4