0

I have an app that want to serialize only classes/functions which are: no Python primitive data type no Numpy data type no pandas data type.

So, is it possible to filter object to be saved in dill ? (filter by loop on the type)

Thanks,

tensor
  • 3,088
  • 8
  • 37
  • 71

1 Answers1

2

While this is not a complete solution (i.e. you'd probably want to include more of the modules with pandas data types, numpy data types… and also you might want to be more selective for the built-in types by filtering by type instead of module)… I think it sort of gets you what you want.

>>> import dill
>>> import numpy
>>> import pandas
>>> 
>>> target = numpy.array([1,2,3])
>>> dill.dumps(target) if not dill.source.getmodule(type(target)) in [numpy, pandas.core.series, dill.source.getmodule(int)] else None
>>> 
>>> target = [1,2,3]             
>>> dill.dumps(target) if not dill.source.getmodule(type(target)) in [numpy, pandas.core.series, dill.source.getmodule(int)] else None
>>> 
>>> target = lambda x:x   
>>> dill.dumps(target) if not dill.source.getmodule(type(target)) in [numpy, pandas.core.series, dill.source.getmodule(int)] else None
>>> 
>>> class Foo(object): 
...   pass
... 
>>> target = Foo()     
>>> dill.dumps(target) if not dill.source.getmodule(type(target)) in [numpy, pandas.core.series, dill.source.getmodule(int)] else None
'\x80\x02cdill.dill\n_create_type\nq\x00(cdill.dill\n_load_type\nq\x01U\x08TypeTypeq\x02\x85q\x03Rq\x04U\x03Fooq\x05h\x01U\nObjectTypeq\x06\x85q\x07Rq\x08\x85q\t}q\n(U\r__slotnames__q\x0b]q\x0cU\n__module__q\rU\x08__main__q\x0eU\x07__doc__q\x0fNutq\x10Rq\x11)\x81q\x12}q\x13b.'
>>>

However, if you are asking if dill has such a filtering method, then the answer is no.

Mike McKerns
  • 33,715
  • 8
  • 119
  • 139
  • Thanks, Just need to create a big list of all primitives types, numpy, and pandas ones. It can be used as regression testing, to throw in dill a list of complex/no standard class and see which ones passed / failed. This list can be maintained by community (make it easier to detect which class is serialized / not serialized). – tensor Dec 27 '16 at 11:05
  • @Tensor: `dill` includes a list of python objects from the standard library that is can and can't serialize. If you do some variant of `dill.load_types()` and then look at the types in `dill.types` you'll see what `dill` can (and, depending on the kwds used in `load_types`, can't serialize). This is not a complete list. There's even a more comprehensive listing that is maintained in `dill._objects`, and that is still primarily the standard library objects. I would be interested in seeing what you come up with, and please feel free to submit a PR. – Mike McKerns Dec 27 '16 at 13:58