3

I need to cache lists 10MB+ size, is it safe to do it in Django filesystem cache?

Or is there a better way to do it?

Bob
  • 5,809
  • 5
  • 36
  • 53

1 Answers1

1

I would examine carefully the Python DiskCache.

Django is Python’s most popular web framework and ships with several caching backends. Unfortunately the file-based cache in Django is essentially broken. The culling method is random and large caches repeatedly scan a cache directory which slows linearly with growth. Can you really allow it to take sixty milliseconds to store a key in a cache with a thousand items?

In Python, we can do better. And we can do it in pure-Python!

raratiru
  • 8,748
  • 4
  • 73
  • 113
  • I'm finding that diskcache fails to write 5-10% of the time (I'm making writes from multiple concurrent processes and each write could be up to 15MB). Any ideas why? – Neil May 11 '20 at 05:14
  • @Neil I am lightly using it for several Bytes for each write. Could it be a disk IO timeout? I do not know how to measure this, however the logs could provide some info. It is better to open an issue but before take a look at [this issue](https://github.com/grantjenks/python-diskcache/issues/139) in case it is relevant. – raratiru May 12 '20 at 11:38
  • 1
    I actually discovered later that it's not write failures, but read failures some small amount of time. I think it may be SQLite operational errors that are ignored in diskcache, what do you think? See [this issue](https://stackoverflow.com/questions/61724247/django-filesystem-file-based-cache-failing-to-write-data-5-10-of-the-time) – Neil May 12 '20 at 17:30