5

Under python 3.7, dill fails to pickle sympy.Float objects when the "recurse" setting is set to True:

dill.settings["recurse"] = True
import sympy
tst = sympy.Float(3.3)
dill.detect.errors(tst)

This fails with RecursionError('maximum recursion depth exceeded in comparison'). Does anybody know a workaround?

The error disappears for Python >=3.8, and also does not occur in the absence of the "recurse" setting. Unfortunately I need to support Python 3.7 and am forced to use the "recurse" mode.

jaykay
  • 51
  • 2

2 Answers2

2

Change pickle protocol from 3 to 4:

import dill
dill.settings["recurse"] = True
dill.settings["protocol"] = 4  # Add this
import sympy
tst = sympy.Float(3.3)
dill.detect.errors(tst)

In Python 3.8 and above, pickle sets DEFAULT_PROTOCOL = 4.

Explanation

  1. In pickle save_reduce, if protocol < 4, a func is saved instead of the Float class:
if self.proto >= 4:
    save(cls)
    save(args)
    save(kwargs)
    write(NEWOBJ_EX)
else:
    func = partial(cls.__new__, cls, *args, **kwargs)
    save(func)
    save(())
    write(REDUCE)

  1. With "recurse" = True, dill recursively traces and pickles objects referred to in the global dictionary of functions (starting with cls.__new__, i.e. Float.__new__).

  2. sympy has circular references in Float.__new__ → _convert_numpy_types → sympify → _convert_numpy_types (repeated) → ...

Proof of concept

You can reproduce the error with:

def a():
    b

def b():
    a

import dill
dill.settings["recurse"] = True
dill.detect.errors(a)
aaron
  • 39,695
  • 6
  • 46
  • 102
1

I'm the dill author. I'm not seeing the reported behavior. Maybe you are using an older version of Python, dill, or sympy.

Python 3.7.13 (default, May 10 2022, 11:13:40) 
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import dill
>>> import sympy
>>> dill.__version__
'0.3.6.dev0'
>>> sympy.__version__
'1.10.1'
>>> tst = sympy.Float(3.3)
>>> dill.copy(tst)
3.30000000000000
>>> dill.settings['recurse'] = True
>>> dill.copy(tst)
3.30000000000000
>>> dill.detect.errors(tst)
>>>

Here, copy(tst) is the same as loads(dumps(tst)). If your versions are older than mine, and after you update you still see the same behavior (or it otherwise persists for you), then please do let me know in the comments (and fill out an issue on the dill GitHub page).

Note: We have a release that is imminent, so dill-0.3.6 will be out in a matter of days.

Mike McKerns
  • 33,715
  • 8
  • 119
  • 139
  • Thank you - first of all for what must be an incredible amount of work developing and maintaining dill and pathos. Fantastic packages! I seem to be stuck here, as I keep seeing the problem described (same sympy version as you, dill on 0.3.5.1). – jaykay Aug 12 '22 at 01:41
  • What if you update to `dill` master? I believe this bug was fixed very recently. – Mike McKerns Aug 12 '22 at 03:57
  • Thanks, dill 0.3.6 works perfectly (and also resolves another recursion problem, the origin of which I failed to pinpoint). Looking forward to the release as our package has dill as a dependency and we are not yet ready to give up on Python 3.7 support. – jaykay Aug 13 '22 at 13:30