1

Hi I want reach some files in a GCP bucket from the cloud shell terminal (for sftp reasons), gcsfuse successfully mounts the father dir and it has all the directories except the one I need, any ideas what am I doing wrong?

Ezer K
  • 3,637
  • 3
  • 18
  • 34
  • 2
    Check the permissions that gcsfuse is using to access the desired files. There also appears to be the ability to run diagnostics using logging options ... see https://github.com/GoogleCloudPlatform/gcsfuse/blob/master/docs/mounting.md#logging – Kolban Oct 03 '21 at 15:34
  • Are you able to find the desired directory? Did the above github link help you? – Prabir Oct 04 '21 at 12:56
  • nope, ended up using gsutil cp – Ezer K Oct 04 '21 at 13:44

1 Answers1

2

In Google Cloud Storage object names ending in a slash(/) represent a directory, and all other object names represent a file. By default directories are not implicitly defined, they exist only if a matching object ending in a slash(/) exists.

Since the usual file system operations like mkdir will do the right thing, if someone set up a bucket's structure using only gcsfuse then they will not notice anything odd about this. However, if someone uses some other tool to set up objects in Google Cloud Storage (such as the storage browser in the Google Cloud Console), they may notice that not all objects are visible until they create leading directories for them.

For example, let's say someone uploaded an object demo/start.txt by choosing the folder upload option in the storage browser section in Google Cloud Console, then mounted it with gcsfuse. The file system will initially appear empty, since there is no demo/ object. However if they subsequently run mkdir demo, they will now see a directory named demo containing a file named start.txt.

To mitigate this issue gcsfuse supports a flag called --implicit-dirs. When this flag is enabled, name lookup requests use the Google Cloud Storage API's Objects.list operation to search for objects that would implicitly define the existence of a directory with the name in question. So, in the example above, a directory named demo containing a file start.txt would appear.

So in your case I suspect the file you are not able to see is a folder which you have uploaded in Google Cloud Storage bucket. As you have already mounted gcsfuse with a directory, if you mount it again using the flag --implicit-dirs, it will throw an error. So I would suggest you to unmount the directory by running the following command -

fusermount -u /path/to/mount/directory

Then mount the directory again by running the following command -

gcsfuse --implicit-dirs BUCKET_NAME /path/to/mount/directory

You can also create a new directory and mount that directory with gcsfuse without unmounting the existing mounted directory.

Please note that the flag --implicit-dirs has some drawbacks. I would recommend you to go through this github issue to get detailed information about it.

Prabir
  • 1,415
  • 4
  • 10