4

According to the docs:

Cloud Shell provisions 5 GB of free persistent disk storage mounted as your $HOME directory on the virtual machine instance.

I would need more (paid) storage though that I can access from the Cloud Shell environment and that is persistent across my sessions. It's mostly used to store local clones of git repositories and images. I would be the only one to access these files.

It seems that the 5 GB storage is a hard limit, so it won't expand dynamically and bill me for the exceeding amount. It is possible to use Boost Mode, but that does not affect the storage size. And I also can't provision more storage with a custom Cloud Shell environment. I couldn't figure out if I can mount another GCE persistent disk to my $HOME. I was considering gcs-fuse as suggested in this answer but I'm not sure if it is suitable for git repos.

Is there any way to have more storage available in Cloud Shell?

Ani
  • 1,377
  • 1
  • 18
  • 29
  • A thought ... what about using Google Filestore ... https://cloud.google.com/filestore/ ... that provides NFS mountable storage within GCP. Could you setup a Filestore environment and then NFS mount the storage in your Cloud Shell environment? – Kolban Oct 06 '19 at 18:29
  • Same workaround with [Fuse}(https://cloud.google.com/storage/docs/gcs-fuse), where you can mount a Bucket as a filesystem storage. – guillaume blaquiere Oct 06 '19 at 19:28
  • 1
    Refer to my answer below to do this with storage buckets. – mze3e Jul 21 '21 at 03:37

4 Answers4

6

Google Cloud Shell is a container that runs on a hidden Compute Engine instance managed by Google. You can download, modify and redeploy this container to Cloud Shell or to your own container running in the cloud or on your desktop.

The base image of the container is available at gcr.io/cloudshell-images/cloudshell:latest, per this page.

For your use case, I would use Compute Engine with Container OS and run the Cloud Shell container within COS. You can scale the CPUs, memory, and storage to fit your requirements.

You can also set up a Compute Engine instance, install the CLIs, SDKs, and tools and have a more powerful system.

Notes for future readers based upon the first answer:

  1. Filestore is a great product, but pay attention to costs. The minimum deployment is 1 TB at $200+ per month. You will need to mount the NFS share each time Cloud Shell restarts - this can be put into login scripts. Note: I am not sure if you can actually mount an NFS share from Filestore in Cloud Shell. I have never tested this.
  2. You will have the same remount problem with FUSE, plus you will have bandwidth costs to access Cloud Storage.
  3. Cloud Shell is a great product that is well implemented, but when you need to exceed its capabilities it is better to deploy a small/medium size GCE instance. this enables persistent, snapshots, etc.
John Hanley
  • 74,467
  • 6
  • 95
  • 159
  • Thanks for pointing out the minimum deployment of Filestore. It was the reason why I didn't consider it at the first place. I forgot to mention that I don't just use the commandline but also the editor (all on a Chromebook), and I was looking for replacing my local IDE. While the experience is not perfect yet, it's good enough for my needs. I don't know if I can replicate the same in a GCE instance, but I'll just spin up a machine and see. – Ani Oct 07 '19 at 07:03
  • @Ani using the container image that is mentioned above will give you exactly the same tools as is available on the cloud shell (of course, you will need to deal with the fact that you are operating inside a container - e.g. you'll need to persist storage outside of that container). – robsiemb Oct 07 '19 at 23:01
  • 1
    Another note on Cloud Shell. SSH access is set up for port 6000 and not 22. You will probably want to change the internal SSH server to use the standard port 22 or maybe not for security. – John Hanley Oct 07 '19 at 23:04
5

There is another way to have more disk space in the cloud shell. It's to create a cloud storage bucket and map the cloud storage bucket as a folder. This way you can store larger files in the cloud storage bucket and it doesn't require any compute instance.

  1. Go to cloud storage and create a new storage bucket

  2. Copy the storage bucket's name, eg. my_storage_bucket

  3. Go to cloud shell and create a folder in your home folder

    mkdir ~/my_bucket_folder

  4. Mount the storage bucket to this folder

    gcsfuse my_storage_bucket ~/my_bucket_folder

  5. Change directory to your my_bucket_folder

    cd ~/my_bucket_folder

  6. Voila! you have unlimited space!

  7. To unmount please run the following

    fusermount -u ~/my_bucket_folder

mze3e
  • 438
  • 2
  • 5
  • 14
3

I'm using gcsfuse and works fine. You don't have to remount every time if you put the mount command in .customize_environment (run on boot up). 

#!/bin/sh
#.customize_environmnet run in background as root, wait for your user to initialize
sleep 20
sudo -u [USER] gcsfuse -o nonempty -file-mode=777 -dir-mode=777 --uid=1000 --debug_gcs [BUCKET_NAME] /home/[USER]/[FOLDER_NAME]

You can read more at Unlimited persistent disk in google cloud shell

robertsahlin
  • 511
  • 1
  • 4
  • 12
  • Thank you for the new answer and the article link. – John Hanley Feb 03 '21 at 19:18
  • I tried this and get various errors like `too many file handles open` when i try to do anything like a `git checkout` or copy a directory. and its verrryyy slowwww on basic stuff. combined with cloud shell's habit of timing out every 5 minutes, I didn't find this a workable option – dcsan May 16 '21 at 20:18
2

There is no way of adding more storage to the Cloud Shell. You can create a VM and install the Cloud SDK and have as much storage as you'd like but it is not currently possible to add storage space to the Cloud Shell.

Depending on how you plan on using the saved repos, Cloud Storage may be ideal as it has a storage category just perfect archiving.

Filestore will be your best option as it is great for file systems and it is scalable. It fits your needs as you have described.

You can use Cloud Storage with FUSE. Keep in mind that this method, although great, depends on how it will be used as costs are based on storage category.

You can see a brief comparison of the Storage solutions the Cloud Platform has to offer here.

maniSidhu98
  • 527
  • 2
  • 9
  • Thank you. I had a concern regarding pricing of Filestore (min 1 TB), so I didn't consider it (or other SMB/NFS servers). I'll try Cloud Storage with FUSE and see if integrity or bandwidth is going to be concern for my use-cases. Alternatively, I'll spin up a machine with the Cloud Shell image, as suggested in answer by John Hanley, but I doubt I get the same out-of-the-box experience in a browser. – Ani Oct 07 '19 at 07:11
  • can you use Filestore like a FUSE mounted drive? is it faster? GCS buckets really seem more for archiving/backups etc. – dcsan May 16 '21 at 20:20
  • @dcsan, I don't think it can be cost-effective. Filestore is very fast, highly-available (99.99% regional SLA), managed and quite expensive NFS server. You can FUSE (client is on GitHub), but only if you have dozens of VMs that aren't after low latency/high throughput. We're crunching data in batch pipelines, but still roll our own instances of NFS servers. Even at 1TB (the minimum for Filestore) we don't break even, given that an NFS server VM can be shut down if there is no work in the queue, but Filestore cannot be stopped, only deleted with its data. It's for continuous load at a scale. – kkm inactive - support strike Apr 07 '22 at 20:29