Why am I getting this PicklingError?

Question

I'm working on a Python/Django/Wagtail project, and I have some api to return some paginated articles. It goes like this:

In URLs:

url(r'^morearticles/', views.get_live_articles),

In Views:

def get_live_articles(request):
  context = {'articles': getLiveArticles(request) }
  return render(request, 'app/components/articles-live.html', context, content_type="text/html; charset=utf-8")

The getLiveArticles function looks like this:

def getLiveArticles(request):

 # Articles
 a = get_articles() #this is getting the articles correctly
 p = Paginator(a, 4)
 page_n = request.GET.get('page')

 try:
     articles = p.page(page_n)
 except Exception, e:
     articles = []

 return articles

However when hitting the api endpoint I get this:

    Traceback (most recent call last):
  File "/Users/john/.virtualenvs/upgrade/lib/python2.7/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/Users/john/.virtualenvs/upgrade/lib/python2.7/site-packages/django/core/handlers/base.py", line 249, in _legacy_get_response
    response = self._get_response(request)
  File "/Users/john/.virtualenvs/upgrade/lib/python2.7/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/Users/john/.virtualenvs/upgrade/lib/python2.7/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/Users/john/app/app/views.py", line 90, in get_live_articles
    context = {'articles': getLiveArticles(request) }
  File "/Users/john/.virtualenvs/upgrade/lib/python2.7/site-packages/cache_utils/decorators.py", line 48, in wrapper
    cache.set(key, value, timeout, **backend_kwargs)
  File "/Users/john/.virtualenvs/upgrade/lib/python2.7/site-packages/django/core/cache/backends/locmem.py", line 75, in set
    pickled = pickle.dumps(value, pickle.HIGHEST_PROTOCOL)
PicklingError: Can't pickle <class 'wagtail.wagtailcore.blocks.base.RichTextBlockMeta'>: attribute lookup wagtail.wagtailcore.blocks.base.RichTextBlockMeta failed

I've came upon the Pickling error before, but never quite understood what it is about. Any idea of what could be causing this?

Let me know if I should provide more information. After debugging I think the issue has to be contained in these chunks of code, but I might be wrong.

EDIT: One of the fields of the objects has been turned into a StreamField object containing the main content lately. May that have something to do?

I think this is cache related (had same issue lately), where some objects can't be pickled for caching? So my question is: Do you use cache and please give us more traceback — Giorgi Jambazishvili, Jun 05 '18 at 18:11
Then for test purposes, please try to disable cache temporarily and replicate problem, I guess this time error would go away — Giorgi Jambazishvili, Jun 05 '18 at 18:15
That fixed it! Would you answer that so I can mark it as the correct answer? — Pablo Viacheslav, Jun 05 '18 at 18:16
Anon downvoters didn't explain themselves, so I'll just say in a comment: USE `dill`. It's better. — JacobIRR, Jun 05 '18 at 18:21
@JacobIRR, I think that, your answer did not answer question - cause of exception, instead you've proposed some `dill` package, that's why, I personally, downvoted — Giorgi Jambazishvili, Jun 05 '18 at 18:26

score 2 · Accepted Answer · answered Jun 05 '18 at 18:23

I think this is cache related (had same issue lately), where some objects can't be pickled for caching, and as you've confirmed you are using cache. So disabling the cache would solve the problem.

I suggest to use pickle compatible object types. On the other hand in Python documentation there is some guidelines how to implement / customize the pickling / unpickling logic.

Why am I getting this PicklingError?

1 Answers1