0

I'm trying to take a dictionary and (1) raise each key to the power of its value, and (2) multiply together the results of (1).

This needs to be done to several million rows of data. Any thoughts on how to speed up the process? I am using pandas data frame's map function to apply the function toTheExp to my column of data already. This step is still pretty slow though. Currently I'm trying something like:

import ast
from numpy import array
from functools import reduce
import operator

def toTheExp(d):
    return reduce(operator.mul, array(d.keys())**array(d.values()), 1)

toTheExp({2:3,4:5,6:7})
>>> 2293235712
JoeDanger
  • 3,325
  • 4
  • 13
  • 14

2 Answers2

2

Since you're importing numpy, I assume it's fine to use it over pandas. 

def to_the_exp(d):
    x = np.fromiter(d.iterkeys(), np.float, count=len(d))
    p = np.fromiter(d.itervalues(), np.float, count=len(d))
    return np.prod(x**p)

so that:

>>> to_the_exp({2:3,4:5,6:7})
2293235712

A quick speed test with 10 million entries:

>>> %timeit to_the_exp(d)
1 loops, best of 3: 2.99 s per loop

and your code gives:

>>> %timeit toTheExp(d)
1 loops, best of 3: 7.96 s per loop

Your code isn't so slow, really. Perhaps the bottleneck is somewhere else? Also, the product of tens of millions of numbers is generally either zero, one, or infinity... Just want to make sure that you're doing something sensical here.

jme
  • 19,895
  • 6
  • 41
  • 39
  • nice solution! Yeah, its millions of rows, each containing a dict of smallish size. so Its applying the function to several million dicts. Not a dict of several million entries – JoeDanger Dec 02 '14 at 03:19
  • Ah, I see. That makes more sense. Well I'm afraid that this solution won't net you much of a speedup, then, seeing as much of the time is going to be spent iterating over your rows. I'd think about getting the data out of the dictionaries somehow and putting everything into one big numpy array so that the whole process can be vectorized. – jme Dec 02 '14 at 03:24
2

What about using comprehension, for example:

from functools import reduce
from operator import mul

print(reduce(mul,(k**v for k,v in {2:3,4:5,6:7}.items()),1)
#2293235712
Marcin
  • 215,873
  • 14
  • 235
  • 294