Maximum size of object that can be saved in memcached

Question

I need to return a rather big file (11MB) to the user. For certain reasons, I can't just provide a direct url to the file (http://www.sample.com/mybigfile.exe); instead it must be accessed through code.

Instead of having to read it from disk over and over, I thought of saving it in memcached (if this is not a good idea, let me know). Everything seems to work, fine (no errors), but when I try to retrieve the file from memcached I always get None, as if the file wasn't cached.

Is there a size limit to what can be saved?

Here's the code:

def download_demo():
    """
    Returns the demo file
    """
    KEY = "xyz"
    TIME = 86400 #24 hours

    buff = memc.get(KEY)
    if not buff: 
        file = open(FILENAME, 'r')
        buff = file.read()
        memc.set(KEY, buff, TIME)

    print "Content-Type:application/x-download\nContent-Disposition:attachment;filename=%s\nContent-Length:%s\n\n%s" %    (os.path.split(FILENAME)[-1], len(buff), buff)

If memcache is running locally you will probably not really win much by reading the content from it, since, assuming there is enough RAM available, the file system will already have it cached in RAM. — radiospiel, Mar 28 '17 at 06:52

score 43 · Answer 1 · edited Oct 03 '18 at 10:40

As of memcache 1.4.2, this is a user-configurable parameter: ReleaseNotes142 * memcached @ Github

Configurable maximum item size

Many people have asked for memcached to be able to store items larger than 1MB, while it's generally recommended that one not do this, it is now supported on the commandline.

A few enlightened folk have also asked for memcached to reduce the maximum item size. That is also an option.

The new -I parameter allows you to specify the maximum item size at runtime. It supports a unit postfix to allow for natural expression of item size.

Examples:
memcached -I 128k # Refuse items larger than 128k. 
memcached -I 10m  # Allow objects up to 10MB

Like in the other answer, these links are also dead now – e4c5 Jun 27 '16 at 03:45 — e4c5, Jun 27 '16 at 03:45

score 40 · Accepted Answer · edited Oct 03 '18 at 10:35

40

There are two entries about that in the memcached FAQ :

What is the maximum size of an object you can store in memcached? Is that configurable?

The answer to the first one is (quoting, emphasis mine) :

The maximum size of a value you can store in memcached is 1 megabyte. If your data is larger, consider clientside compression or splitting the value up into multiple keys.

So I'm guessing your 11MB file is quite too big to fit in one memcached entry.

Increasing the size of an object is possible, as per other answers.

edited Oct 03 '18 at 10:35

naXa stands with Ukraine

35,493
19
190
259

answered Sep 17 '09 at 19:07

Pascal MARTIN

395,085
80
655
663

5

Note: Memcached now allows you to change the size limit at launch – NoviceCoding Jun 20 '12 at 22:08
21

These links are all dead now – Assaf Lavie Aug 20 '12 at 12:39
3

In newer versions of Memcached you can set the limit up to 128 mb using the -I option (linux.die.net/man/1/memcached). You might consider switching to Redis for large object storing. – maxwell2022 Feb 24 '14 at 22:59
2

Wow! The answer is so old that these links died twice. – naXa stands with Ukraine Oct 03 '18 at 10:32

score 24 · Answer 3 · edited Oct 03 '18 at 10:41

To summarise the necessary steps:

1) Update Memcache to version 1.4.2 or later.

2) Add the flag -I 15M (or however many megabytes) to your memcache run command.

That's either command line, or in Ubuntu, add the line

-I 15M

to anywhere in /etc/memcached.conf and restart the service.

3) Add the necessary flag to the client in memcache.

import memcache
memc = memcache.Client(['localhost'], server_max_value_length=1024*1024*15)
memc.set(KEY, buff, TIME)

If you don't have direct access to the memcache client (i.e. working through a framework), then just hack the memcache code directly.

On Ubuntu, it's /usr/local/lib/python2.7/dist-packages/memcache.py. Change the line:

SERVER_MAX_ITEM_LENGTH = 1024 * 1024

to

SERVER_MAX_ITEM_LENGTH = 1024 * 1024 * 15

Obviously, you'll need to do this hack over again if you update memcache, but it's a very simple and quick fix.

The argument you want to pass to the python memcache Client is actually server_max_value_length see [memcache.py:163](https://github.com/linsomniac/python-memcached/blob/master/memcache.py#L163) — Zach Dwiel, Nov 08 '13 at 01:22

score 2 · Answer 4 · edited Jan 21 '16 at 12:58

2

The maximum data size is 1mb per item

http://code.google.com/p/memcached/wiki/FAQ#What_is_the_maximum_data_size_you_can_store?_%281_megabyte%29

Update: This is a 2009 answer. And in that date the info was accurate and the source was official. Now in 2014 memcached can store 128mb instead of 1mb (but the developers didn't bother to update the official FAQ). The only reference someone could find id an obscure page that probably will be dead in one year.

edited Jan 21 '16 at 12:58

ChrisF

134,786
31
255
325

answered Sep 17 '09 at 19:06

The Disintegrator

4,147
9
35
43

2

This is a dead link now – maxwell2022 Feb 24 '14 at 00:21
I can't find any new source there. Looks like it was bad advertising? – The Disintegrator Feb 24 '14 at 22:46
In newer versions of memcached you can set the limit to 128 mb using the `-I` option (http://linux.die.net/man/1/memcached). I personally switch to Redis for large object storage. – maxwell2022 Feb 24 '14 at 22:58
1

This is a 2009 answer. And in that date the info was accurate and the source was official. Now in 2014 some morons are downvoting all answers because (five years after the fact) memcached can store 128mb instead of 1mb and the developers didn't bother to update the official FAQ. The only reference someone could find id an obscure page that probably will be dead in one year. Way to go. And now this is MY fault? – The Disintegrator Mar 05 '14 at 23:22
@TheDisintegrator just keep your answer updated and be polite. Focus on the content, not the person. [Code of Conduct](https://stackoverflow.com/conduct). – naXa stands with Ukraine Oct 03 '18 at 10:45

score 1 · Answer 5 · answered May 06 '15 at 12:16

Following article explained why it limit max size of a single file to 1M.

http://www.mikeperham.com/2009/06/22/slabs-pages-chunks-and-memcached/

Basicly, there are small pieces of memory pages in memcache, inside which objects reside.

And the default page size seems to be 1M, so the max object a page can contain is limited to 1M.

Even though you can config it to increase the size, but I think this might have some kind of performance trade-off.

score 1 · Answer 6 · answered Jun 24 '16 at 02:26

If you are using python memcached client, please also make sure to increase the limit in client memcache.py file along with the memcached server.

~/anaconda/lib/python2.7/site-packages/memcache.py

SERVER_MAX_KEY_LENGTH = 250 
SERVER_MAX_VALUE_LENGTH = 1024 * 1024 * 15

you can see in memcached console ( telnet localhost 11211 )

stats slabs STAT 48:chunk_size 3677344

score 1 · Answer 7 · answered Nov 17 '16 at 13:28

Nobody seems to address this part of the question:

Instead of having to read it from disk over and over, I thought of saving it in memcached (if this is not a good idea, let me know).

This is not a good idea. All modern file systems have efficient caching mechanisms. Reading a file from disk that was read a few seconds earlier will typically still be in memory cache. This is much faster than accessing memcached over a network.

If you are running memcached on 127.0.0.1, the memory it claimed will only be used for that single file, wereas if you simply rely on the caching of your operating system the memory can be used for various purposes depending on the jobs at hand.

+1. You're right to note Memcache can be unnecessary here. But I will say "it depends". I already work with distributed cache to speed up access to a (file) system. It was very complex, but it may be a good trade-off depending on the use case (number of requests, size of the data, SLA of the system, cost, ...) — mcoolive, Apr 04 '18 at 09:46

ns15 · Answer 8 · 2022-10-03T14:14:29.623

2022 Update - For open source Memcached,

the minimum value for max-item-size(-I, --max-item-size) is 1KiB
the maximum value is 1 GiB
the default value is 1 MiB

https://github.com/memcached/memcached/blob/597645db6a2b138710f01ffe5e92e453117b987a/memcached.c#L5213

if (unit == 'k' || unit == 'K')
    size_max *= 1024;
if (unit == 'm' || unit == 'M')
    size_max *= 1024 * 1024;

https://github.com/memcached/memcached/blob/597645db6a2b138710f01ffe5e92e453117b987a/memcached.c#L5730

if (settings.item_size_max > (ITEM_SIZE_MAX_UPPER_LIMIT)) {
    fprintf(stderr, "Cannot set item size limit higher than a gigabyte.\n");
    exit(EX_USAGE);
}

In case of cloud memcache variant this value will differ - For example here is the limit on google cloud memcache - gcloud docs

Note: If you are planning to increase the max-item-size value then inspect/increase the amount of memory allocated for the cache(maxbytes : this is the overall Memcached memory). use the -m option to specify the amount of RAM to be allocated (in megabytes). Default is 64MB.

Both settings are available to inspect in memcached statistics (size in bytes):

stats settings
STAT maxbytes 67108864
STAT item_size_max 1048576

Maximum size of object that can be saved in memcached

8 Answers8

Configurable maximum item size

Linked