9

I've followed the gsutil install guide following the Google Cloud instructions and I've updated GCloud components to the latest versions. I've just recently installed and configured GCloud to work with my credentials and project. First, I've used the following command to export a single collection I need from my Firestore :

gcloud firestore export gs://my-project-id.appspot.com --collection-ids=theCollectionINeed

Now I see this export is in my Firebase Console Storage section as a folder. Being the project Owner, I'd like to get this export into my local system. For that, I see I need to use gsutil to be able to copy it. Reading the instructions to download the object from your bucket, I've tried with the following command, but I got this error:

$ gsutil cp -r gs://m-project-id.appspot.com/2020-05-22T02:01:06_86154 .
Copying gs://lucky-level-dev-6ac34.appspot.com/2020-05-22T02:01:06_86154/2020-05-22T02:01:06_86154.overall_export_metadata...
OSError: The filename, directory name, or volume label syntax is incorrect.

I'm running this command using cmd in a Windows 10 environment. I'd like to be able to download this folder from the cloud to my local drive.

Update

After being, I tried to change the bucket folder (object prefix) as suggested:

gsutil mv gs://my-project-id.appspot.com/2020-05-22T02:01:06_86154 gs://my-project-id.appspot.com/2020-06-23_someFolder

But trying again now throws me a new error:

gsutil cp -r gs://my-project-id.appspot.com/2020-05-22_someFolder .
Copying gs://my-project-id.appspot.com/2020-05-22_someFolder/2020-05-22T02:01:06_86154.overall_export_metadata...
OSError: Invalid argument.9.0 B]

Surely I need to change the name of the file too?

halfer
  • 19,824
  • 17
  • 99
  • 186
Metafaniel
  • 29,318
  • 8
  • 40
  • 67
  • 2
    I suspect the issue is that GCS object prefix includes colons and these may cause issues for Windows. I don't have Windows. Please try `gsutil mv gs://m-project-id.appspot.com/2020-05-22T02:01:06_86154 gs://m-project-id.appspot.com/X` to rename the object and then try `gsutil cp -r gs://m-project-id.appspot.com/X .` again. – DazWilkin Jun 23 '20 at 21:24
  • @DazWilkin Thanks for the suggestion. Now I've surpassed that error; sadly now I got the following: `OSError: Invalid argument.9.0 B]` I'll update my question. Any ideas? – Metafaniel Jun 24 '20 at 00:44
  • @DazWilkin I also had to rename the inner file that also included colons in its name. Thanks again! – Metafaniel Jun 24 '20 at 01:17
  • 3
    You may wish to file a bug with Google. The command should work on supported OSs and clearly does not with Windows. https://issuetracker.google.com – DazWilkin Jun 24 '20 at 15:56
  • 1
    For anyone who is interested, this issue is fixed in `gcloud storage` (https://github.com/GoogleCloudPlatform/gsutil/issues/1513) – Betty Dec 03 '22 at 20:48

3 Answers3

10

When doing the cp -r gs:/bucket_name/folder . Gsutil will try to create a folder with "bucket_name" name in the current location, Windows doesn't allow the name of folders to have some special characters including the ':'

You can rename the bucket folder (object prefix) with the command @DazWilkin suggested gsutil mv gs://m-project-id.appspot.com/2020-05-22T02:01:06_86154 gs://m-project-id.appspot.com/new_folder_name and then try again with the new folder name.

Also check that all filenames inside the directory structure don't have the ':' character or any other special one.

Andres S
  • 1,168
  • 7
  • 11
  • Thanks for the suggestion. That really seemed to be the problem, sounds logic but now I've got the following: `OSError: Invalid argument.9.0 B]` Any ideas? – Metafaniel Jun 24 '20 at 00:43
  • I also had to rename the inner file that also had colons and the problem was solved. Please, complement your answer with this addition to get a full answer in case someone don't read the comments. Thanks! – Metafaniel Jun 24 '20 at 01:19
  • Ok I am glad this has been solved! I have updated my answer. – Andres S Jun 24 '20 at 13:18
2

(Posted solution from question author to place it in the answer section).

All I had to do was rename the inner file that also had colons in its name. Renaming it solved the problem!

halfer
  • 19,824
  • 17
  • 99
  • 186
  • 1
    Do you _really_ think this answers the question by itself?? Don't just over moderate... – Metafaniel Jul 13 '20 at 18:42
  • @Metafaniel: it is (partial) answer material, so it needs to go in the answer space. I am happy to delete it if you think it is trivial, or close the question as Typo/Unrepro if the answers are unlikely to be useful to future readers, but I do not think this material belongs in the question. Alternatively you can post this answer yourself (minus the attribution statement) as a self-answer and I can delete the CW copy. – halfer Jul 13 '20 at 20:44
  • Finally, you could ask a question over at _Meta Stack Overflow_ if you want to get second opinions though. – halfer Jul 13 '20 at 20:46
0

In the case of a gcloud firestore export the issue can be avoided by specifying a directory:

# BAD this will auto generate a directory windows doesnt like
gcloud firestore export gs://my-bucket

# GOOD specify a directory
gcloud firestore export gs://my-bucket/firestore-backup

# copy as usual
gsutil -m cp -r "gs://my-bucket/firestore-backup" .
chantey
  • 4,252
  • 1
  • 35
  • 40