0

I got a challenge to setup a simple object storage server or cdn, especially for storing and loading (via API) any PNG, TXT, and TIFF files, the server have docker and kubernetes both running. I am new to MinIO but managed to install a fresh MinIO then create multiple buckets, setup their access policy and encryption. For disk caching i am able to apply one configuration for multiple bucket, but i cannot figure it out how to apply different configuration to each different bucket since in the console only showing singular configuration.

enter image description here

My objectives are like this:

  • Bucket 1 - thematic data (PNG) - Expiry after 60 days
  • Bucket 2 - topographic data (TIFF) - Expiry after 30 days
  • Bucket 3 - metadata (TXT) - Expiry after 1 days

Already read the documentation but got no solution, any suggestion will be appreciated

Thank you

aswzen
  • 1,512
  • 29
  • 46

1 Answers1

1

Cache configuration is part of a Server deployment. Within a server you cannot apply different configurations for multiple buckets.
Note: caching is meant for gateway deployments to act as a relatively close consumption zone, counter-acting latency for consumers.

If performance and caching requirements are really a must (your distribution is, for example over multiple regions) you can deploy an extra server that acts as a gateway. But you'd end-up with a gateway per configuration that you'd like and even then it's not per bucket.

Personally I would go for a design with multiple server instances if differentiating in caching is really important.

Michaël van der Haven
  • 413
  • 2
  • 10
  • In Redis you can set different Time To Live for each key prefix and i just want the same, but seems MinIO is not working like that – aswzen Mar 29 '22 at 07:44
  • Do you meant that you'd like to have the files/objects removed after a given amount of time? In that case you need to use a retention policy of bucket lifecycle. More info on bucket lifecyles can be found here: https://docs.min.io/docs/minio-bucket-lifecycle-guide.html It can be done per bucket, but also inside of the bucket per prefix. So you can do that in a pretty fine-grained manner. – Michaël van der Haven Mar 29 '22 at 11:20
  • Not like that, the files are still there actually and should be not removed, only the files cache are removed after given amount of time. You know right, cache is using RAM instead of disk storage. Your link about lifecycle is showing a bucket automatic removal. – aswzen Apr 01 '22 at 03:26
  • I was hinted by your note on the TTL in Redis, so I thought you'd like the same behavior (if it's gone from Redis, and not backed by another database: it is pretty much gone forever). The caching in Minio is mainly meant for a distribution setup (e.g. a buffer in another region which is btw a mix between in-memory and disk), and not to act with the same fine-grained options as you'd have when you use Redis as a performance buffer for another database. If you want to have the same behavior with a fine-grained buffering system then Redis as a cache in front of S3 is a pretty common setup as well – Michaël van der Haven Apr 01 '22 at 11:06