2

I'm trying to copy large dictionary in python (10MB when saved as raw json)

What is the fastest way to do it?

Is there any c extension that perform this copy fast?

So far i found ujson.loads(ujson.dumps(my_dict)) to be the fastest option which looks strange (how translating dict to string and then from string to new dict is faster then some pure copy)

Here is an example of the methods i tried and their running time for small dictionary (the results of course are more clear with larger dictionary):

x = {'a':1,'b':2,'c':3,'d':4, 'e':{'a':1,'b':2}}

#this function only handle dict of dicts
def fast_copy(d):
    output = d.copy()
    for key, value in output.items():
        output[key] = fast_copy(value) if isinstance(value, dict) else value        
    return output



from copy import deepcopy
import ujson


%timeit deepcopy(x)
13.5 µs ± 146 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

%timeit fast_copy(x)
2.57 µs ± 31.6 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

%timeit ujson.loads(ujson.dumps(x))
1.67 µs ± 14.8 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
Idok
  • 3,642
  • 4
  • 21
  • 18
  • `dict` constructor accepts dictionary as an argument. You don't have to create copy manually simply do `y = dict(x)`. – Sohaib Farooqi May 22 '18 at 10:49
  • looks similar. i'm focusing on c extension solution... The answer there is not fully correct... the suggested solution is compared to json library. ujson is faster than the suggested solution which is mind blowing .... – Idok May 22 '18 at 10:51
  • @bro-grammer dict(x) is doing shallow copy. i need deep copy. y = dict(x) ; id(y['e']) == id(x['e']) – Idok May 22 '18 at 10:52

0 Answers0