55

I have a game state in Python with about 1000 objects (planetary systems + stars + planets), and I need to copy it and apply a bunch of transformations to it when requested. However, at about 1 request/second, this is taking up 24.63% of my runtime. How can I make it go fast? Note that copying less is not an option, since the transforms touch just about everything.

EDIT: got it down to 8% with judicious implementation of __deepcopy__ on things. Still, not good enough. (Good enough is 1% or less, I plan on throwing many more things at this.) timeit says 41.8ms per deepcopy().

Vadim Kotov
  • 8,084
  • 8
  • 48
  • 62
Electro
  • 2,994
  • 5
  • 26
  • 32
  • Alternatively to copying the state, you could create an action queue: Use the current state to determine the next actions and effects for all the objects without applying them right away, then apply all those actions in one batch, then calculate the actions for the next 'turn', etc. – tobias_k Jul 15 '14 at 11:21
  • @tobias_k That's actually exactly what my pile of transforms is doing, but I need to do it afresh quite often, hence the copying. – Electro Jul 15 '14 at 11:22
  • Then I don't understand the question: Are you applying different sets of actions to _the same_ state. maybe to see which ones come out best? – tobias_k Jul 15 '14 at 11:23
  • Yup. Well, not to see which ones come out best, but that is what I am doing. – Electro Jul 15 '14 at 11:24
  • Another idea: Maybe you can create some sort of "diff-state", i.e. like "this is x and this is y and everything else is the same as in parent state z". – tobias_k Jul 15 '14 at 11:27
  • Yeah, but to access the output without modifying the initial state, I still need a copy. – Electro Jul 15 '14 at 11:28
  • But you would not need to copy the entire state, if those actions change only, say, 10% of it. You specify what those 10% will be in the "copy" of the state and keep a reference back to the original state for the remaining 90%. – tobias_k Jul 15 '14 at 11:29
  • Why not use 'copy()' that do a shallow-copy ? I don't see the point where you need a deepcopy(). – Antoine Jul 15 '14 at 11:31
  • @tobias_k That might work, but it would be a lot of work to maintain the internal references. – Electro Jul 15 '14 at 11:33
  • @Antoine I need a deep copy, since the state object has objects I need copies of. – Electro Jul 15 '14 at 11:34

5 Answers5

71

Actually, deepcopy is very slow. But we can use json, ujson, or cPickle. we can use json/cPickle to dump an object, and load it later. This is my test:

Total time: 3.46068 s
File: test_deepcopy.py
Function: test at line 15
Line #   Hits          Time Per Hit   % Time  Line Contents
==============================================================
15                                             @profile
16                                             def test():
17       100       957585   9575.9     27.7        b = deepcopy(a)
18       100          862      8.6      0.0        c = copy(a)
19       100        42295    422.9      1.2        d = ujson.loads(ujson.dumps(a))
20       100        85040    850.4      2.5        e = json.loads(json.dumps(a))
21       100      2323465  23234.7     67.1        f = pickle.loads(pickle.dumps(a, -1))
22       100        51434    514.3      1.5        g = cPickle.loads(cPickle.dumps(a, -1))

as what we can see, json/ujson/cPickle is faster than deepcopy, but pickle...

cherish
  • 1,370
  • 1
  • 11
  • 16
  • 1
    That is scary. What does `deepcopy` do that `cPickle` doesn't? – Electro Apr 09 '15 at 11:05
  • 2
    I don't know... you can find something useful [here](https://docs.python.org/2/library/copy.html) :) – cherish Apr 13 '15 at 01:37
  • 6
    I did some testing and found that deepcopy outperformed json on lists of dictionaries while json outperformed deepcopy on large nested dictionaries. – wrkyle Mar 16 '18 at 00:11
  • 11
    Please note that for python3, `pickle` is what's listed as `cPickle` here. – Karl Bartel Jan 06 '20 at 16:46
  • What is scary is what `deepcopy` is doing to take so many time. I've checked `pickle` (python3) in a backtracking algorithm and works perfect. – Ivan Dec 04 '20 at 18:50
  • 3
    Be careful with `json.loads(json.dumps(a))` (probably for `ujson` as well), they will always consider keys as strings. If a key is not a string, it'll convert to a string. – Rikard Olsson Jan 20 '22 at 13:35
9

If you create your own class to hold these objects you can create your own methods that work with copy and deep copy. http://www.rafekettler.com/magicmethods.html#copying (Broken Link)

New Link for a github repository https://github.com/RafeKettler/magicmethods

class MyClass():
    def __copy__(self):
        copy_object = MyClass()
        return copy_object

    def __deepcopy__(self, memodict={}):
        copy_object = MyClass()
        copy_object.value = self.value
        return copy_object

if __name__ == "__main__":
    my_inst = MyClass()
    print(copy.deepcopy(my_inst))

Here is a similar description from the previous broken link.

Copying

Sometimes, particularly when dealing with mutable objects, you want to be able to copy an object and make changes without affecting what you copied from. This is where Python's copy comes into play. However (fortunately), Python modules are not sentient, so we don't have to worry about a Linux-based robot uprising, but we do have to tell Python how to efficiently copy things.

__copy__(self)

Defines behavior for copy.copy() for instances of your class. copy.copy() returns a shallow copy of your object -- this means that, while the instance itself is a new instance, all of its data is referenced -- i.e., the object itself is copied, but its data is still referenced (and hence changes to data in a shallow copy may cause changes in the original).

__deepcopy__(self, memodict={})

Defines behavior for copy.deepcopy() for instances of your class. copy.deepcopy() returns a deep copy of your object -- the object and its data are both copied. memodict is a cache of previously copied objects -- this optimizes copying and prevents infinite recursion when copying recursive data structures. When you want to deep copy an individual attribute, call copy.deepcopy() on that attribute with memodict as the first argument. What are some use cases for these magic methods? As always, in any case where you need more fine-grained control than what the default behavior gives you. For instance, if you are attempting to copy an object that stores a cache as a dictionary (which might be large), it might not make sense to copy the cache as well -- if the cache can be shared in memory between instances, then it should be.

justengel
  • 6,132
  • 4
  • 26
  • 42
  • That is actually what I was just pursuing. However, I only got it down to 8% of my runtime. Things learned: `deepcopy`ing a `sortedcontainers.SortedListWithKey` is slow, listify it first. Also copying `itertools.count()` is slow, which probably applies to generators in general. – Electro Jul 15 '14 at 12:07
  • I found a similar description for copying and added it to the bottom of my answer. I also found that there was a github repository for that link. – justengel Jan 16 '17 at 14:02
4

I've made a fast experiment comparing both deepcopy/json/ujson for several cases and my results contradicts @cherish's ones on certain cases, posting the little experiment here:

import ujson
import timeit
import json
import random
import string
import copy
import ujson
import sys


def random_string(N):
    return ''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(N))


def random_json(width=5, height=5, levels=1):
    dct = {}
    lst = [random_string(4) for i in range(width)]
    lst2 = [random.randint(0, 10000) for i in range(width)]
    lst3 = [bool(random.randint(0, 1)) for i in range(width)]
    for j in range(height):
        dct[str(j)] = lst
        dct[str(width+j)] = lst2
        dct[str(2*width+j)] = lst3

    for i in range(levels):
        new_dct = {}
        for j in range(height):
            new_dct[str(j)] = dct
        dct = json.loads(json.dumps(new_dct))

    return new_dct

if __name__ == "__main__":
    print(sys.version)
    levels = 3
    for i in range(15):
        dataset = random_json(i, i, levels)
        print("Comparing deepcopy/ujson/json using random dataset({},{},{}), length {}".format(i,i,levels, len(json.dumps(dataset))))
        print(timeit.timeit('copy.deepcopy(dataset)',
                            setup='from __main__ import copy, dataset', number=10))
        print(timeit.timeit('ujson.loads(ujson.dumps(dataset))',
                            setup='from __main__ import ujson, dataset', number=10))
        print(timeit.timeit('json.loads(json.dumps(dataset))',
                            setup='from __main__ import json, dataset', number=10))
        print()

And the results would be:

3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:04:45) [MSC v.1900 32 bit (Intel)]
Comparing deepcopy/ujson/json using random dataset(0,0,3), length 2
2.6842977659931844e-05
0.00012039864979822371
7.776568527950847e-05

Comparing deepcopy/ujson/json using random dataset(1,1,3), length 63
0.0002731667726569534
3.552747043226263e-05
0.00012987264191349377

Comparing deepcopy/ujson/json using random dataset(2,2,3), length 1106
0.0011858280130946362
0.00034974820892205325
0.0007093651596308467

Comparing deepcopy/ujson/json using random dataset(3,3,3), length 6834
0.0042218477363672215
0.0021178319874343293
0.003378267688436718

Comparing deepcopy/ujson/json using random dataset(4,4,3), length 26572
0.011379054029782284
0.006288757016181971
0.009920059244030693

Comparing deepcopy/ujson/json using random dataset(5,5,3), length 79210
0.028879491215043435
0.027906433274870912
0.029595961868760734

Comparing deepcopy/ujson/json using random dataset(6,6,3), length 183678
0.047142979515255284
0.04682125853300759
0.06791747047568517

Comparing deepcopy/ujson/json using random dataset(7,7,3), length 395528
0.08239215142913198
0.09871347134571351
0.15347433002098887

Comparing deepcopy/ujson/json using random dataset(8,8,3), length 764920
0.1351954464835896
0.19448842613700734
0.3020533693660834

Comparing deepcopy/ujson/json using random dataset(9,9,3), length 1356570
0.24560258734724671
0.44074906118659407
0.5705849913806413

Comparing deepcopy/ujson/json using random dataset(10,10,3), length 2287770
0.3237815755327835
0.61104051671153
0.8698565598118777

Comparing deepcopy/ujson/json using random dataset(11,11,3), length 3598750
0.4958284828467452
0.9472223636741877
1.2514314609961668

Comparing deepcopy/ujson/json using random dataset(12,12,3), length 5636414
0.6261448233909714
1.4066722957969802
1.8636325417418167

Comparing deepcopy/ujson/json using random dataset(13,13,3), length 8220800
0.8396582099444547
2.061675688670409
2.755659427352441

Comparing deepcopy/ujson/json using random dataset(14,14,3), length 12018290
1.0951926990258762
2.96703050743886
4.088875914783021

Conclusion from this little experiment is:

  • When dictionaries are small ones time(ujson)<time(json)<time(deepcopy)
  • When dictionaries are big ones time(deepcopy)<time(ujson)<time(json)

So it depends the number of copies you're making per second and which type of dictionary you're dealing with, you'll prefer switching between deepcopy or ujson.

BPL
  • 9,632
  • 9
  • 59
  • 117
4

The built-in module marshal is faster than ujson and supports more native types like set, tuple, and non-string dict keys. It's not portable across Python versions (similarly to pickle), but for locally cloning data that's not an issue.

To show this, I took @BPL's test program from another answer and added marshal to the compared cases, and ran it on an ARMv6-compatible processor.

Added case:

print(timeit.timeit('marshal.loads(marshal.dumps(dataset))',
       setup='from __main__ import marshal, dataset', number=1))

Results (Marshal is the fastest for all cases):

2.7.14 (default, Mar  6 2019, 13:27:55)
[GCC 7.3.0]
Comparing deepcopy/marshal/ujson/json using random dataset(0,0,1), length 2
0.000588178634644
0.000134944915771
0.000258922576904
0.00113606452942
()
Comparing deepcopy/marshal/ujson/json using random dataset(0,0,3), length 2
0.000546932220459
0.000134944915771
0.000180006027222
0.00120401382446
()
Comparing deepcopy/marshal/ujson/json using random dataset(0,0,5), length 2
0.000545978546143
0.000128984451294
0.000185966491699
0.00106000900269
()
Comparing deepcopy/marshal/ujson/json using random dataset(0,2,1), length 50
0.00154900550842
0.000281810760498
0.000414848327637
0.00174903869629
()
Comparing deepcopy/marshal/ujson/json using random dataset(0,2,3), length 242
0.00655102729797
0.000789880752563
0.00133085250854
0.00432300567627
()
Comparing deepcopy/marshal/ujson/json using random dataset(0,2,5), length 1010
0.0514280796051
0.0015549659729
0.00413513183594
0.0148711204529
()
Comparing deepcopy/marshal/ujson/json using random dataset(0,4,1), length 172
0.00250005722046
0.000365018844604
0.000761985778809
0.00263404846191
()
Comparing deepcopy/marshal/ujson/json using random dataset(0,4,3), length 2892
0.0329101085663
0.00363397598267
0.0110101699829
0.0262169837952
()
Comparing deepcopy/marshal/ujson/json using random dataset(0,4,5), length 46412
0.616458892822
0.0826110839844
0.189103841782
0.504135131836
()
Comparing deepcopy/marshal/ujson/json using random dataset(2,0,1), length 2
0.000693082809448
0.000132083892822
0.000182867050171
0.00107002258301
()
Comparing deepcopy/marshal/ujson/json using random dataset(2,0,3), length 2
0.000566005706787
0.000132083892822
0.000180959701538
0.00107598304749
()
Comparing deepcopy/marshal/ujson/json using random dataset(2,0,5), length 2
0.000562906265259
0.000128984451294
0.000184059143066
0.00118517875671
()
Comparing deepcopy/marshal/ujson/json using random dataset(2,2,1), length 258
0.00405406951904
0.000534057617188
0.00124287605286
0.00309610366821
()
Comparing deepcopy/marshal/ujson/json using random dataset(2,2,3), length 1058
0.026270866394
0.00180387496948
0.00363302230835
0.0096640586853
()
Comparing deepcopy/marshal/ujson/json using random dataset(2,2,5), length 4338
0.0778729915619
0.00682806968689
0.0151469707489
0.0468928813934
()
Comparing deepcopy/marshal/ujson/json using random dataset(2,4,1), length 716
0.00720596313477
0.00100684165955
0.0215280056
0.0062358379364
()
Comparing deepcopy/marshal/ujson/json using random dataset(2,4,3), length 11468
0.112984895706
0.0238728523254
0.0448131561279
0.0874760150909
()
Comparing deepcopy/marshal/ujson/json using random dataset(2,4,5), length 183628
1.83552503586
0.407335042953
0.617804050446
1.65498495102
()
Comparing deepcopy/marshal/ujson/json using random dataset(4,0,1), length 2
0.000571012496948
0.000132083892822
0.000189781188965
0.00121593475342
()
Comparing deepcopy/marshal/ujson/json using random dataset(4,0,3), length 2
0.000757932662964
0.000131130218506
0.000180959701538
0.00144195556641
()
Comparing deepcopy/marshal/ujson/json using random dataset(4,0,5), length 2
0.00056791305542
0.000132083892822
0.000184059143066
0.00107407569885
()
Comparing deepcopy/marshal/ujson/json using random dataset(4,2,1), length 430
0.00451302528381
0.00053596496582
0.00142502784729
0.00343203544617
()
Comparing deepcopy/marshal/ujson/json using random dataset(4,2,3), length 1730
0.0259549617767
0.00232696533203
0.00387692451477
0.0187470912933
()
Comparing deepcopy/marshal/ujson/json using random dataset(4,2,5), length 7026
0.112207174301
0.0119769573212
0.0211799144745
0.0547370910645
()
Comparing deepcopy/marshal/ujson/json using random dataset(4,4,1), length 1684
0.00609397888184
0.00121903419495
0.00452899932861
0.00959086418152
()
Comparing deepcopy/marshal/ujson/json using random dataset(4,4,3), length 26828
0.19367814064
0.0293428897858
0.0688338279724
0.140627145767
()
Comparing deepcopy/marshal/ujson/json using random dataset(4,4,5), length 433484
3.54843020439
0.590909004211
1.09412097931
2.72070598602
zcoop98
  • 2,590
  • 1
  • 18
  • 31
Ritzen Yang
  • 49
  • 1
  • 1
    Thanks. `marshal` is the fastest of all the mentioned alternatives in this thread, slightly faster than `cPickle` as well. – Marius Feb 12 '22 at 21:56
  • Worth noting that marshal doesn't support cyclical objects that refer to themselves (pickle does I believe), but json serialization won't either. – Will S Aug 07 '23 at 13:29
1

You can provide your own copy functions to the objects such that you won't need deep copy. deep copy inspects every object to check what needs to be copied. This is an expensive operation.

Bort
  • 2,423
  • 14
  • 22
  • I do need a deep copy of everything. The bits which are transformed are also the most numerous ones, and are hence causing `deepcopy()` to do more work. – Electro Jul 15 '14 at 11:15
  • @Electro I think what he meant is to write your own deep copy method that does copy everything, just without the reclection/introspection part. – tobias_k Jul 15 '14 at 11:25
  • I fail reading comprehension. Oops. Hmm, but is the introspection really the most expensive operation of `deepcopy()`? – Electro Jul 15 '14 at 11:28
  • 2
    Yes, `deepcopy()` has to make sure no reference loops will appear etc... There is also a lot of bookkeeping involved. See [here](http://stackoverflow.com/questions/3043369/deepcopy-and-python-tips-to-avoid-using-it), [here](http://stackoverflow.com/questions/10128351/any-alternative-to-a-very-slow-deepcopy-in-a-dfs) and [here](http://writeonly.wordpress.com/2009/05/07/deepcopy-is-a-pig-for-simple-data/). – Bort Jul 15 '14 at 11:35