Copying nested custom objects: alternatives to deepcopy

Question

I'm looking to make a deep copy of a class object that contains a list of class objects, each with their own set of stuff. The objects don't contain anything more exciting than ints and lists (no dicts, no generators waiting to yield, etc). I'm performing the deep copy on between 500-800 objects a loop, and it's really slowing the program down. I realize that this is already inefficient; it currently can't be changed.

Example of how it looks:

import random
import copy

class Base:
    def __init__(self, minimum, maximum, length):
        self.minimum = minimum
        self.maximum = maximum
        self.numbers = [random.randint(minimum, maximum) for _ in range(length)]
        # etc

class Next:
    def __init__(self, minimum, maximum, length, quantity):
        self.minimum = minimum
        self.maximum = maximum
        self.bases = [Base(minimum, maximum, length) for _ in range(quantity)]
        # etc

Because of the actions I'm performing on the objects, I can't shallow copy. I need the contents to be owned by the new variable:

> first = Next(0, 10, 5, 10)
> second = first
> first.bases[0].numbers[1] = 4
> print(first.bases[0].numbers)
> [2, 4, 3, 3, 8]
> print(second.bases[0].numbers)
> [2, 4, 3, 3, 8]
>
> first = Next(0, 10, 5, 10)
> second = copy.deepcopy(first)
> first.bases[0].numbers[1] = 4
> print(first.bases[0].numbers)
> [8, 4, 7, 9, 9]
> print(second.bases[0].numbers)
> [8, 11, 7, 9, 9]

I've tried a couple of different ways, such as using json to serialize and reload the data, but in my tests it's not been nearly fast enough, because I'm stuck reassigning all of the variables each time. My attempt at pulling off a clever self.__dict__ = dct hasn't worked because of the nested objects.

Any ideas for how to efficiently deep copy multiply-nested Python objects without using copy.deepcopy?

You might want to think about ways of storing your data items as primitives (e.g. ints and strings) rather than Python objects. Copying those would be far more efficient. Perhaps the array module or even the cython language would be useful. — Paul Cornelius, Aug 19 '16 at 00:28

score 8 · Accepted Answer · edited May 23 '17 at 12:08

8

Based on cherish's answers here, pickle.loads(pickle.dumps(first)) works about twice as fast per call. I had written it off initially because of an unrelated error when testing it, but on retesting it, it performs well within my needs.

edited May 23 '17 at 12:08

Community

1
1

answered Sep 09 '16 at 14:32

Noah Bogart

1,703
1
16
24

score 5 · Answer 2 · answered Aug 19 '16 at 00:17

5

One of the first things that copy.deepcopy looks for is if the object defines it's own __deepcopy__ method So instead of having it figure out how to copy object every time just define your own process.

It would require you have a way of defining a Base object without any element of random-ness for the copies to use, but if you can find a more efficient process of copying your objects you should define it as a __deepcopy__ method to speed up copying process.

answered Aug 19 '16 at 00:17

Tadhg McDonald-Jensen

20,699
5
35
59

I'm not sure how much a custom `__deepcopy__` can help here. The default implementation already copies all attributes, and it avoids calling `__init__` on the new object, so you don't run into any problems with the random stuff. I think the only clear ways to improve performance is to either dramatically change the data structures (e.g. using `numpy` arrays or something, rather than lists of objects), or changing the whole algorithm to avoid the need for all the copying. – Blckknght Aug 19 '16 at 01:24
I tried this out, and couldn't get the speed gains I was hoping for (no doubt an error on my part in how I implemented it). I found that using `pickle.loads(pickle.dumps(first))` worked about twice as fast as deepcopy for my purposes. Thanks for the help! – Noah Bogart Sep 09 '16 at 14:25

Copying nested custom objects: alternatives to deepcopy

2 Answers2

Linked