Why do I get this TypeError while using Python's copy.deepcopy?

Question

I've just run into this error with copy.deepcopy:

import copy
import datetime


class Hours(datetime.timedelta):
    # Using __new__ because timedelta is immutable
    # See https://stackoverflow.com/a/22531773/3358488.
    def __new__(cls, hours):
        return datetime.timedelta.__new__(cls, hours=hours)


h1 = Hours(2)
h2 = copy.deepcopy(h1)  # TypeError: __new__() takes 2 positional arguments but 4 were given

Here's the full traceback:

Traceback (most recent call last):
  File "/Users/xxxx/Documents/PyCharmProjects/Flow/backend/deepcopy_test.py", line 11, in <module>
    h2 = copy.deepcopy(h1)  # TypeError: __new__() takes 2 positional arguments but 4 were given
  File "/Users/xxxx/.conda/envs/qtdesktopapp/lib/python3.8/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/Users/xxxx/.conda/envs/qtdesktopapp/lib/python3.8/copy.py", line 264, in _reconstruct
    y = func(*args)
TypeError: __new__() takes 2 positional arguments but 4 were given

Note that timedelta initialization's takes anywhere from 0 to 6 arguments:

class datetime.timedelta(days=0, seconds=0, microseconds=0, milliseconds=0, minutes=0, hours=0, weeks=0)

Have I used copy.deepcopy incorrectly or have I found a bug in it?

What is your intent with defining `__new__`? Evidently your signature for it is not correct. Are you confusing it for `__init__`? — bgfvdu3w, Jul 02 '22 at 17:34
@bgfvdu3w that's because timedelta is immutable, please see https://stackoverflow.com/a/22531773/3358488 — user118967, Jul 02 '22 at 17:42
Regarding the PPS, you get all the zeros because you gave the parameters default values of 0. — wjandrea, Jul 02 '22 at 18:02
@wjandrea: but the method does capture a non-zero value when I use `*args`, so why aren't those non-zero values captured by the positional arguments. The default `0` should not be used if values are being provided... — user118967, Jul 02 '22 at 18:20
@user118967 Sorry, I totally misunderstood! The actual reason is that the new parameters don't match the usage. `Hours(2)` means `Hours(days=2)`, but `days` is not saved anywhere. If you do `Hours(hours=2)` instead, you'll see `0 7200 0 0 0 0 0`. — wjandrea, Jul 02 '22 at 18:33
@wjandrea: sorry, I still don't understand. It seems that somewhere `deepcopy` is calling `__new__` with `0 7200 0 0 0 0 0` arguments. If I use `*args*`, I capture that. But if I have all those positional arguments with default `0`, I only get `0`s, which I don't understand since the default should not kick in if `7200` is being provided. — user118967, Jul 02 '22 at 19:17
@user118967 Sorry, I think you're confusing two different situations. If you add `print(h1); print(h2)`, it should be clearer, and [here's a gist](https://gist.github.com/wjandrea/9a5759f776969000195dd2217d58105e) with a full explanation. — wjandrea, Jul 02 '22 at 19:48
@wjandrea I see! Thank you very much for taking the time to create the gist! — user118967, Jul 04 '22 at 19:32

Samwise · Accepted Answer · 2022-07-02T17:43:28.007

Add a *args to your __new__ and add a print so you can see what arguments it's getting called with:

import copy
import datetime


class Hours(datetime.timedelta):
    def __new__(cls, hours, *args):
        print(cls, hours, *args)
        return datetime.timedelta.__new__(cls, hours=hours)


h1 = Hours(2)
h2 = copy.deepcopy(h1)

prints:

<class '__main__.Hours'> 2
<class '__main__.Hours'> 0 7200 0

deepcopy doesn't know that you modified __new__ so it's trying to construct a copy the same way it'd copy a timedelta object, by passing the positional args days, seconds, and microseconds.

If you want deepcopy to work correcty, you need to override the __deepcopy__ method in your class to match the __new__ override:

class Hours(datetime.timedelta):
    def __new__(cls, hours):
        return datetime.timedelta.__new__(cls, hours=hours)

    def __deepcopy__(self, _memo):
        return Hours(self.seconds // 3600)


h1 = Hours(2)
h2 = copy.deepcopy(h1)
assert h1 is not h2  # different objects
assert h1 == h2      # but they're still equivalent

This works in this case, but it doesn't work for negatives or partial hours. For example try `Hours(-1.501)`, which `timedelta` represents as `-1 day, 22:29:56.400000`. This solution reduces it to just `22:00:00`. I think it'd be better to lean into the parent constructor instead, i.e. `def __deepcopy__(self, _memo): return datetime.timedelta.__new__(type(self), self.days, self.seconds, self.microseconds)`. — wjandrea, Jul 02 '22 at 19:25

Why do I get this TypeError while using Python's copy.deepcopy?

1 Answers1