0

I have a bucket on GCP and at the top level of this bucket, I have a bunch of folders.

I want to create a new folder and move all of the other ones into it.

However, I've mounted my bucket with gcsfuse and tried traditional Linux mv commands. This is not allowed, apparently.

Likewise, I have also tried gsutil -m mv gs://mybucket/* gs://mybucket/new_folder/ and have received the command error that wildcards are not allowed in this operation.

What's the best option to get this large number of files moved into a new directory?

rocksNwaves
  • 5,331
  • 4
  • 38
  • 77
  • 1
    maybe an 'ls' command to get a list of the files and then iterate over the list doing an individual file by file move? – Kolban Aug 20 '20 at 02:49
  • 3
    1/2) A few concepts to note for Cloud Storage. 1) Objects are immutable, which means you cannot rename then. You must copy objects and delete the original to emulate changing the name. 2) Directories/Folders do not exist. The namespace is flat, all objects are in the root directory. The appearance of folders is just a part of the object name. 3) Cloud Storage supports internal object copy. Be careful not to use a feature which first downloads the object and then uploads it. – John Hanley Aug 20 '20 at 03:22
  • 2
    2/2) 4) Now that those concepts are known, you will need to use a tool such as `gsutil` on each object to initiate the renaming/moving you desire. – John Hanley Aug 20 '20 at 03:22
  • Thanks @JohnHanley. Yeah, I've sorta come to realize those things you are saying, which has lead me to drop the idea of "moving" all of these objects to a folder off of root. You seem to be a real wiz at this stuff. What is it that you do for a living? – rocksNwaves Aug 20 '20 at 14:54

1 Answers1

0

Posting this as a Community Wiki answer, based in the comments provided by @JohnHanley.

A few concepts to note for Cloud Storage.

  1. Objects are immutable, which means you cannot rename then. You must copy objects and delete the original to emulate changing the name.
  2. Directories/Folders do not exist. The namespace is flat, all objects are in the root directory. The appearance of folders is just a part of the object name.
  3. Cloud Storage supports internal object copy. Be careful not to use a feature which first downloads the object and then uploads it.

Considering this information, you will need to use a tool, for example, the gsutil, so you can start to rename and move the files as you would like.

gso_gabriel
  • 4,199
  • 1
  • 10
  • 22