Running gsutil cp ${gsutilURI} ${localPath} will sometimes return a 404 midway through download. Retrying the command will eventually succeed. In the script running the above command, there is a prior existence check with gsutil -q stat ${gsutilURI} that returns successfully. The file being downloaded was uploaded shortly before in the script by a different machine. The download does await completion of the upload.
Example output below
build@xxxxxxxx C:\workspace\Build>gsutil -o GSUtil:parallel_composite_upload_threshold=150M cp gs://xxxxxx/svnCache/567962/trunk.tar ./trunk-567962.tar
Copying gs://xxxxxx/svnCache/567962/trunk.tar...
/ [0 files][ 0.0 B/ 7.1 GiB]
/ [0 files][ 0.0 B/ 7.1 GiB]
-
\
\ [0 files][118.4 MiB/ 7.1 GiB] 118.2 MiB/s
|
/
/ [0 files][335.0 MiB/ 7.1 GiB] 167.2 MiB/s
-
\
\ [0 files][552.8 MiB/ 7.1 GiB] 183.9 MiB/s
|
| [0 files][768.1 MiB/ 7.1 GiB] 191.8 MiB/s
/
-
- [0 files][986.9 MiB/ 7.1 GiB] 207.4 MiB/s
\
\ [0 files][ 1.2 GiB/ 7.1 GiB] 217.7 MiB/s
|
NotFoundException: 404 gs://xxxxxx/svnCache/567962/trunk.tar does not exist.
/
/ [0 files][ 1.4 GiB/ 7.1 GiB] 218.4 MiB/s
-
\
\ [0 files][ 1.6 GiB/ 7.1 GiB] 219.2 MiB/s
|
| [0 files][ 1.8 GiB/ 7.1 GiB] 220.3 MiB/s
/
-
- [0 files][ 2.0 GiB/ 7.1 GiB] 220.8 MiB/s
\
|
| [0 files][ 2.3 GiB/ 7.1 GiB] 220.5 MiB/s
/
/ [0 files][ 2.5 GiB/ 7.1 GiB] 220.6 MiB/s
-
\
\ [0 files][ 2.7 GiB/ 7.1 GiB] 220.2 MiB/s
|
/
/ [0 files][ 2.9 GiB/ 7.1 GiB] 219.7 MiB/s
-
- [0 files][ 3.1 GiB/ 7.1 GiB] 221.3 MiB/s
\
|
| [0 files][ 3.4 GiB/ 7.1 GiB] 223.6 MiB/s
/
NotFoundException: 404 gs://xxxxxx/svnCache/567962/trunk.tar does not exist.
-
- [0 files][ 3.6 GiB/ 7.1 GiB] 222.9 MiB/s
\
\ [0 files][ 3.8 GiB/ 7.1 GiB] 222.2 MiB/s
|
/
/ [0 files][ 4.0 GiB/ 7.1 GiB] 225.2 MiB/s
NotFoundException: 404 gs://xxxxxx/svnCache/567962/trunk.tar does not exist.
-
\
\ [0 files][ 4.2 GiB/ 7.1 GiB] 221.5 MiB/s
|
| [0 files][ 4.4 GiB/ 7.1 GiB] 220.7 MiB/s
/
-
- [0 files][ 4.6 GiB/ 7.1 GiB] 222.6 MiB/s
\
|
| [0 files][ 4.9 GiB/ 7.1 GiB] 224.7 MiB/s
/
/ [0 files][ 5.1 GiB/ 7.1 GiB] 223.2 MiB/s
-
\
CommandException: Some components of .\trunk-567962.tar were not downloaded successfully. Please retry this download.
I would expect to not see 404s during a download of a URI that does exist. It makes some sense because GCS does chunked up/downloads but it doesn't really... Retrying the file download in the script eventually succeeds. Checking for existence of the file through gsutil or in the web portal shows existence. Downloading the file on another machine works fine.
