Python 3.5 dill pickling/unpickling on different servers: "KeyError: 'ClassType'"

Question

See updates at the bottom

--

A similar question was asked here, but never resolved: pickling and unpickling user-defined class

I'm working on a project which necessitates pickling user defined classes, and sending them to a remote server where they are unpickled and called. We use the Dill library to accomplish this, and have had a lot of success.

Unfortunately, I've run into an issue I'm having a hard time debugging. I create and pickle a class as follows:

import dill, base64
import time, random

class periodicSource(object):
    def __call__(self):
        while True:
            time.sleep(0.1)
            yield random.uniform(20,100)

periodic_src = periodicSource()
a = base64.b64encode(dill.dumps(periodic_src)).decode("ascii")
print(a)

It creates an ascii representation of the dilled class.

gANjZGlsbC5kaWxsCl9jcmVhdGVfdHlwZQpxAChjZGlsbC5kaWxsCl9sb2FkX3R5cGUKcQFYCQAAAENsYXNzVHlwZXEChXEDUnEEWA4AAABwZXJpb2RpY1NvdXJjZXEFaAFYBgAAAG9iamVjdHEGhXEHUnEIhXEJfXEKKFgIAAAAX19jYWxsX19xC2NkaWxsLmRpbGwKX2NyZWF0ZV9mdW5jdGlvbgpxDChoAVgIAAAAQ29kZVR5cGVxDYVxDlJxDyhLAUsASwFLA0tjQyl4IgB0AABqAQBkAQCDAQABdAIAagMAZAIAZAMAgwIAVgFxAwBXZAAAU3EQKE5HP7mZmZmZmZpLFEtkdHERKFgEAAAAdGltZXESWAUAAABzbGVlcHETWAYAAAByYW5kb21xFFgHAAAAdW5pZm9ybXEVdHEWWAQAAABzZWxmcReFcRhYHwAAADxpcHl0aG9uLWlucHV0LTIwLTdhNGU5MDIwYWM2Yz5xGWgLSwdDBgABAwENAXEaKSl0cRtScRx9cR0oWAYAAAByYW5kb21xHmNkaWxsLmRpbGwKX2ltcG9ydF9tb2R1bGUKcR9oFIVxIFJxIVgEAAAAdGltZXEiaB9YBAAAAHRpbWVxI4VxJFJxJXVoC05OfXEmdHEnUnEoWAoAAABfX21vZHVsZV9fcSlYCAAAAF9fbWFpbl9fcSpYBwAAAF9fZG9jX19xK05YDQAAAF9fc2xvdG5hbWVzX19xLF1xLXV0cS5ScS8pgXEwLg==

When I go to deserialize it on the other server:

a = 'gANjZGlsbC5kaWxsCl9jcmVhdGVfdHlwZQpxAChjZGlsbC5kaWxsCl9sb2FkX3R5cGUKcQFYCQAAAENsYXNzVHlwZXEChXEDUnEEWA4AAABwZXJpb2RpY1NvdXJjZXEFaAFYBgAAAG9iamVjdHEGhXEHUnEIhXEJfXEKKFgIAAAAX19jYWxsX19xC2NkaWxsLmRpbGwKX2NyZWF0ZV9mdW5jdGlvbgpxDChoAVgIAAAAQ29kZVR5cGVxDYVxDlJxDyhLAUsASwFLA0tjQyl4IgB0AABqAQBkAQCDAQABdAIAagMAZAIAZAMAgwIAVgFxAwBXZAAAU3EQKE5HP7mZmZmZmZpLFEtkdHERKFgEAAAAdGltZXESWAUAAABzbGVlcHETWAYAAAByYW5kb21xFFgHAAAAdW5pZm9ybXEVdHEWWAQAAABzZWxmcReFcRhYHwAAADxpcHl0aG9uLWlucHV0LTIwLTdhNGU5MDIwYWM2Yz5xGWgLSwdDBgABAwENAXEaKSl0cRtScRx9cR0oWAYAAAByYW5kb21xHmNkaWxsLmRpbGwKX2ltcG9ydF9tb2R1bGUKcR9oFIVxIFJxIVgEAAAAdGltZXEiaB9YBAAAAHRpbWVxI4VxJFJxJXVoC05OfXEmdHEnUnEoWAoAAABfX21vZHVsZV9fcSlYCAAAAF9fbWFpbl9fcSpYBwAAAF9fZG9jX19xK05YDQAAAF9fc2xvdG5hbWVzX19xLF1xLXV0cS5ScS8pgXEwLg=='
a = dill.loads(base64.b64decode(a.encode()))
print(a)

I get the following error:

/home/streamsadmin/anaconda3/bin/python /home/streamsadmin/git/streamsx.topology/test/python/topology/deleteme2.py

Traceback (most recent call last):
 File "/home/streamsadmin/git/streamsx.topology/test/python/topology/deleteme2.py", line 40, in <module>
   a = dill.loads(base64.b64decode(a.encode()))
 File "/home/streamsadmin/anaconda3/lib/python3.5/site-packages/dill/dill.py", line 277, in loads
   return load(file)
 File "/home/streamsadmin/anaconda3/lib/python3.5/site-packages/dill/dill.py", line 266, in load
   obj = pik.load()
 File "/home/streamsadmin/anaconda3/lib/python3.5/site-packages/dill/dill.py", line 524, in _load_type
   return _reverse_typemap[name]
KeyError: 'ClassType'

I would expect this if I were using different version of Python on the remote system, but they're the same:

Server 1:

>>> import sys
>>> sys.version
'3.5.2 |Anaconda 4.1.1 (64-bit)| (default, Jul  2 2016, 17:53:06) \n[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]'

Server 2:

>>> import sys
>>> sys.version
'3.5.2 |Anaconda 4.2.0 (64-bit)| (default, Jul  2 2016, 17:53:06) \n[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]'

Additionally, both versions of Dill are 0.2.6. Any ideas how I could debug this?

EDIT: I've think it might be something with my environment. I'm using Python 3.5, but list the builtin types:

>>> import types
>>> dir(types)
  ['BuiltinFunctionType',
   'BuiltinMethodType',
   'ClassType',
   'CodeType',
   ...
  ]

It seems ClassType is in the output which should NOT be the case since ClassType was removed in Python 3.5. This is exceedingly strange.

I'm running on a system that has both Python 2.7 and Python 3.5 installed. Could the 2.7 installation somehow be polluting the 3.5 installation?

I'm not familiar with Dill, but with pickle you need to have all functions (and I think classes) available when you unpickle as it just stores references to where those are. — fuzzything44, Mar 22 '17 at 19:20
Pickle does that. Dill manages to serialize dependencies along with the class, so that's not the issue. — wheelium, Mar 22 '17 at 19:25
If I do this on my linux mint with python 3.4, I get a different string but it works. if I use your string, I get the same error you do. My encoded string is too big to post here, so see http://pastebin.com/Pu6MCkhg — tdelaney, Mar 22 '17 at 20:04
The string you posted decodes successfully: "<__main__.periodicSource object at 0x7f600282b5c0>" But the question remains of why the one server is serializing differently. — wheelium, Mar 22 '17 at 20:37
Hm, I just realized that the versions of Anaconda are different. One is Anaconda 4.2.0, and the other is Anaconda 4.1.1. Although, the versions of Dill are still the same at 0.2.6. I'm downloading Anaconda 4.1.1 now to see if their versions of dill are, in fact, compatible. — wheelium, Mar 22 '17 at 21:02
Nope. The issue still persists when deserializing with Anaconda 4.1.1. — wheelium, Mar 22 '17 at 21:49
Hi, I'm the `dill` author. I am not seeing your error -- however I'm using the latest `dill` and python `3.5.3` everywhere. Actually, I've never seen the error you are seeing... but the traceback says that the `ClassType` is not registered in the `typemap`... which is super weird. — Mike McKerns, Mar 23 '17 at 14:17
Unpickling errors are harder to debug than pickling errors. So, I suggest two things for a start. (1) add `dill.extend(True)` before the `dill.loads`, just to ensure that all `dill` types are loaded into the registry, and (2) add `dill.detect.trace(True)` before the `dill.dumps` to see what exactly is going into the serialization. — Mike McKerns, Mar 23 '17 at 14:17
A third suggestion is to open an issue on the `dill` GitHub page. This seems like it might take some back-and-forth, and that's probably a better venue for that type of conversation than SO. — Mike McKerns, Mar 23 '17 at 14:18
Another note: you can test it it's an issue of code differences between the first and second server by simply saving the serialized string to your clipboard... starting a new session on the first server, and pasting the string into the session. If it works (using two sessions and one server), then it's an issue of some difference between installs on the servers (or something like that). — Mike McKerns, Mar 23 '17 at 14:22
Hi Mike, thank you for the advice. I'll create a GH issue on the dill page and follow up here with the results. — wheelium, Mar 23 '17 at 18:16
@MikeMcKerns, I've determined it's probably not a Dill issue. Take a look at the top-level edit to this issue. It seems that, somehow, `ClassType` is finding it's way into the `types` `__dict__` and confusing dill. — wheelium, Mar 23 '17 at 18:41
That could definitely be the cause... it's a bit unexpected. Note that in the pickle registry, `dill` uses `ClassType = type` in python 3.x... so you could re-register the function in the dispatch table manually if you need to. However, I'd look to finding why it's kicked out in the first place. — Mike McKerns, Mar 23 '17 at 20:41
@MikeMcKerns, Eureka! That did it. I added `dill.dill._reverse_typemap['ClassType'] = type` just before deserialization. Hm, I don't think `dill` actually does define `ClassType` anywhere. In `dill` 0.2.6 in `dill.py` at line 486 where `_reverse_typemap` is defined, I don't see a `ClassType` key. — wheelium, Mar 23 '17 at 22:37
So breaking it down -- __Server A__ has `types.ClassType` defined, and __server B__ does not. __Server A__ serializes an object and passes it to __server B__. On deserialization, __server B__ complains about `ClassType` not being defined in the `_reverse_typemap`. A workaround is to manually set `dill.dill._reverse_typemap['ClassType'] = type` before deserialization on __server B__. — wheelium, Mar 23 '17 at 22:46

score 19 · Accepted Answer · edited May 01 '19 at 19:56

The culprit is cloudpickle. By default in Python 3.5, types.ClassType is left unset.

>>> import types
>>> dir(types)
['BuiltinFunctionType', 'BuiltinMethodType', 'CodeType', ...]

When cloudpickle is imported, suddenly, types.ClassType becomes defined.

>>> import cloudpickle
>>> dir(types)
['BuiltinFunctionType', 'BuiltinMethodType', 'ClassType', 'CodeType', ...]

Server A uses dill to serialize objects, and also imports cloudpickle. Therefore it includes a reference to ClassType during serialization.

Server B does NOT import cloudpickle, and then tries to find a reference to ClassType during deserialization and fails. Raising the error:

Traceback (most recent call last):
 File "/home/streamsadmin/git/streamsx.topology/test/python/topology/deleteme2.py", line 40, in <module>
   a = dill.loads(base64.b64decode(a.encode()))
 File "/home/streamsadmin/anaconda3/lib/python3.5/site-packages/dill/dill.py", line 277, in loads
   return load(file)
 File "/home/streamsadmin/anaconda3/lib/python3.5/site-packages/dill/dill.py", line 266, in load
   obj = pik.load()
 File "/home/streamsadmin/anaconda3/lib/python3.5/site-packages/dill/dill.py", line 524, in _load_type
   return _reverse_typemap[name]
KeyError: 'ClassType'

On our system, we can't remove cloudpickle from our environment, so we had to do the following workaround.

On server B, right after we import dill and sometime before the first call to dill.loads, we invoke the following line of code:

dill._dill._reverse_typemap['ClassType'] = type

This defines ClassType appropriately. And causes dill deserialization to work as expected.

Thank you for this. A minor change: dill._dill._reverse_typemap['ClassType'] = type — Prayson W. Daniel, Apr 11 '19 at 13:14

score 2 · Answer 2 · answered Sep 06 '18 at 10:25

I am sure cloudpickle is causing the problem. You can debug it step by step.

First Check if classType Exists in your builtin types

import types dir(types)

if it exist than it should have worked for you, if not than move to next steps.

import cloudpickle and now check again. You will have classType in buildin types
excute below code

dill.dill._reverse_typemap['ClassType'] = type

it should work for you :)

But if you are still getting error AttributeError: module 'dill' has no attribute 'dill'

than use this one dill._dill._reverse_typemap['ClassType'] = type because dill.dill is moved to dill._dill

Python 3.5 dill pickling/unpickling on different servers: "KeyError: 'ClassType'"

2 Answers2