Python - the best way to serialize data with high performance?

Question

I need to serialize a data which requires high performance. Separated thread will be accessing it each second and must to load data to memory. There will be about 1000 - 10000 dictionary-like entries about user sessions (id, sessid, login date). Some data will be frequently updated because login time has some time.

These data will be shared between python server and Django application. I think to use pickle or its faster version cPickle. I also found marshal.

What is the best way to do that? Is cPickle efficient enough? Or maybe marshal?

EDIT: Very important thing is access time. It's going to be a realtime websocket server so I need very low delays. Is it faster to access cPickle data once a second or to connect to database like Redis?

score 4 · Answer 1 · answered Apr 08 '12 at 18:53

4

A better approach may be to use some in-memory cache - memcached, if your needs are simple - or something with a bit more of a feature set, like redis.

answered Apr 08 '12 at 18:53

Burhan Khalid

169,990
18
245
284

cPickle will be very slow compared to redis - which is designed foremost for speed. – Burhan Khalid Jan 17 '13 at 07:10
Unfortunately, it turns out that redis is actually very slow. See [redis vs. read file](https://stackoverflow.com/q/11216647/2529619) and [redis vs. sqlite3](https://stackoverflow.com/q/11216647/2529619) That's because redis is designed as a remote data store, and each command makes an expensive network roundtrip or an IPC. – ChaimG Jan 21 '21 at 20:49

score 4 · Answer 2 · edited Jun 02 '15 at 16:39

redis still requires serializing any complex python object, so redis doesn't solve this problem unless you represent all your data as simple keys and simple values. redis is not a deserialization solution, it's just a data store for strings. and in any case, redis is one of the slower options: https://charlesleifer.com/blog/completely-un-scientific-benchmarks-of-some-embedded-databases-with-python/

score 0 · Answer 3 · answered Apr 08 '12 at 19:03

0

Use a real database in memory. Don't use pickle, cPickle or marshal, or anything like that.

answered Apr 08 '12 at 19:03

Ali Afshar

40,967
12
95
109

Complex objects will be pickled anyway with Redis – fjsj Sep 05 '20 at 23:02

Python - the best way to serialize data with high performance?

3 Answers3