4

Rackspace cloud files uses a flat storage system using 'containers' to store files. According to Rackspace there is no limit to the number of files per container.

My question is whether there is a best/most efficient number of files per container to optimize write/fetch performance.

If I have tens of thousands of files to store, should they all go in a single giant container or partitioned into many smaller containers? And if so, what is the optimal container size?

Pete Lunenfeld
  • 1,557
  • 3
  • 19
  • 32

2 Answers2

5

FYI: [Snippets taken from rackspace support]

long story short, the containers are databases, and the more rows in a table, the more time it takes to write them on standard hardware. When a write hasn't been committed to disk, it sits in a queue, and it subject to data loss. It's something we noticed with large containers, and the more objects, the more likely it was, so we instituted the limits to protect the data.

because of the rate limits, your data is safe, it just slows down the writes a bit

the limits starts as low as 50,000 objects, and at that level it limits you to 100 writes per second

by 1,000,000 objects in a container, it's 25 per second

and at 5 million and above, you're down to 4 writes per second

We apologize for the limitations, and will be updating our documentation to more clearly express this.

-This has recently hurt us quite badly. Thought I'd share until they get there API doc's upto date, so others can plan around this issue.

ChrisH
  • 157
  • 8
  • I don't think this is still the case today. We have a legacy app that writes a ton of files to a single container (around 77 million objects right now) and have not noticed anything like a 4-writes-per-second throttle in place. We have a sort of directory structure in place though where each directory shouldn't be reaching something like a million objects so maybe that's what's keeping us from hitting any drastic throttling? – georaldc Nov 29 '19 at 08:40
2

We recommend no more than 1 million objects per container. The system will return a maximum of 10,000 object names per list request by default.

Update 9/20/2013 from Cloud Files development: The 1 million object per container recommendation is no longer accurate since Cloud Files switched to all SSD container servers. Also, the list is limited to 10,000 containers at a time.

Community
  • 1
  • 1
Robert Collazo
  • 165
  • 4
  • 1
    Hi Robert, According to one of your developers the recommendation is more towards 10 million objects per container, Why the vast difference between your recommendations? (http://www.rackspace.com/cloud/blog/2010/01/26/nested-folders-in-rackspace-cloud-files/#comment-10341) – Ittai Dec 27 '11 at 13:10
  • "but performance for some use cases will slow down (and level off) after adding millions of objects". He is just not particularly clear about how bad the slow down is! "There is nothing top stop you from putting a billion items in a container." -Nothing to stop you except it will take around 8 years to enqueue 1 billion objects with the rate limiting they have in place! – ChrisH May 21 '12 at 15:01