3

We want to deploy a k8s cluster which will run ~100 IO-heavy pods at the same time. They should all be able to access the same volume.

What we tried so far:

  • CephFS
    • was very complicated to set up. Hard to troubleshoot. In the end, it crashed a lot and the cause was not entirely clear.
  • Helm NFS Server Provisioner
    • runs pretty well, but when IO peaks a single replica is not enough. We could not get multiple replicas to work at all.
  • MinIO
    • is a great tool to create storage buckets in k8s. But our operations require fs mounting. That is theoretically possible with s3fs, but since we run ~100 pods, we would need to run 100 s3fs sidecars additionally. Thats seems like a bad idea.

There has to be some way to get 2TB of data mounted in a GKE cluster with relatively high availability?

Firestorage seems to work, but it's a magnitude more expensive than other solutions, and with a lot of IO operations it quickly becomes infeasible.

I contemplated creating this question on server fault, but the k8s community is a lot smaller than SO's.

yspreen
  • 1,759
  • 2
  • 20
  • 44
  • 1
    Are you using Google Kubernetes Engine? Have you tried using a storage class other than default in a PersistentVolumeClaim? – Judith Guzman Jan 14 '20 at 21:50
  • Here is a great blog on storage performance https://vitobotta.com/2019/08/06/kubernetes-storage-openebs-rook-longhorn-storageos-robin-portworx/ – Tummala Dhanvi Jan 14 '20 at 22:00
  • Did you try Cloud Filestore - https://cloud.google.com/filestore? This would support your use case however I've heard that others have trouble getting really good throughput from it – stringy05 Jan 14 '20 at 23:50

1 Answers1

1

I think I have a definitive answer as of Jan 2020, at least for our usecase:

| Solution        | Complexity | Performance | Cost           |
|-----------------|------------|-------------|----------------|
| NFS             | Low        | Low         | Low            |
| Cloud Filestore | Low        | Mediocre?   | Per Read/Write |
| CephFS          | High*      | High        | Low            |

* You need to add an additional step for GKE: Change the base image to ubuntu

I haven't benchmarked Filestore myself, but I'll just go with stringy05's response: others have trouble getting really good throughput from it

Ceph could be a lot easier if it was supported by Helm.

yspreen
  • 1,759
  • 2
  • 20
  • 44
  • Why have you discarded entirely using SSD storage with dynamically provisioned PersistentVolumes? Is there some reason regarding your use case that discards this entirely? https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/ssd-pd – Judith Guzman Jan 16 '20 at 17:58
  • 2
    It says ReadWriteMany in the title. Directly connecting disks only allows RWOnce. – yspreen Jan 16 '20 at 21:16
  • Thank you, I was not aware of that fact. – Judith Guzman Jan 16 '20 at 23:21
  • 2
    On the pricing page it seems that Cloud Filestore will run at least $200/mo. Their minimum volume size is 1TB and it costs at least $0.20/GB/mo. – ckeeney Feb 20 '20 at 17:07
  • @ckeeney Good point. If you're fine with HDDs, regular gcp disks are just $40/TB/month and way more flexible regarding size. Also very little operation cost. – yspreen Feb 20 '20 at 21:15