Pad 0's Of Numpy Array To Nearest Power Of Two

Question

Question:

How to quickly add 0's to a large array (~ 600 000 entries) at the beginning to bring the length of the array to the next power of two. (2^n) Is there a faster solution besides np.concatinate()?

What I've already tried:

Using the np.concatenate(0, arr) function until the length of the array is equal to the next power of two. The code I have works, it just takes a very very long time.

Here's the pad left function:

def PadLeft(arr):
    nextPower = NextPowerOfTwo(len(arr))
    deficit = int(math.pow(2, nextPower) - len(arr))
    #for x in range(1, int(deficit)):
    for x in range(0, deficit):    
        arr = np.concatenate(([0], arr))
    return arr

Here's the next power of two function:

def NextPowerOfTwo(number):
    # Returns next power of two following 'number'
    return math.ceil(math.log(number,2))

My implementation:

arr = np.ones(())
a = PadLeft(arr)

Thanks!

Initialize an output array with zeros and then set values into it? — Divakar, Apr 10 '16 at 18:50
The arrays I'm dealing with come in at different lengths so I would like it to be done post! — jackzellweger, Apr 10 '16 at 18:52
I meant something like : `out = np.zeros(deficit+len(arr),dtype=arr.dtype); out[deficit:] = arr` to simulate output from `PadLeft(arr)` IIUC. — Divakar, Apr 10 '16 at 18:57

Oliver W. · Accepted Answer · 2016-04-11T00:11:08.750

6

Rather than extending the old array in a for loop with a single element, why not add the entire set of zeroes at once?

arr = np.concatenate((np.zeros(deficit, dtype=arr.dtype), arr))

So don't use the for-loop. That's where your code is running slowly, as it is making a new array every iteration, which is far less efficient than making the required size array once and then filling it as needed, which can be done in several ways. This is just one, one that's close to your own solution.

The reason dtype=arr.dtype is added, is because np.zeros will return an array that is of the np.float dtype by default. If the datatype of arr was "less" than that (in a casting sense), the result will be cast to the "broader" datatype, being float, which is usually not what you would want (because it happens automatically). This valid point was made by Divakar in the comments below.

edited Apr 11 '16 at 00:11

answered Apr 10 '16 at 18:52

Oliver W.

13,169
3
37
50

To keep the datatype as the input array, use `,dtype=arr.dtype` with `np.zeros` maybe? Timed this against an initialization based one and seems like this is just as fast or even faster. – Divakar Apr 10 '16 at 19:13
I thought it would indeed be similar to your initialization suggestion (make it into an answer, please? I'd like to upvote, because it is a *good* answer), because "behind the screens" the same/similar operations would need to be performed. `dis` would probably give closure for that, but I don't have a python-shell here... – Oliver W. Apr 10 '16 at 19:25
Added. But would make sense I guess if you could add that dtype thing in your post for cases when `arr` is an int array, because `np.zeros(deficit)` would default to a float array and after concatenation would still be float . – Divakar Apr 10 '16 at 19:42
@Divakar, done and thank you for adding your solution. Upvoted ;-) – Oliver W. Apr 10 '16 at 23:19

Imanol Luengo · Answer 2 · 2016-04-10T19:54:16.827

5

There is numpy.pad which does exactly that.

For a 1D array:

arr = np.pad(arr, (deficit,0), mode='constant')

It reads as (left, right) padding.

For a 2D arrray:

arr = np.pad(arr, ((0,0), (deficit,0)), mode='constant')

The second parameter reads as ((top, bottom), (left, right)). Which pads the array with deficit to the left.

edited Apr 10 '16 at 19:54

answered Apr 10 '16 at 19:35

Imanol Luengo

15,366
2
49
67

score 4 · Answer 3 · edited May 23 '17 at 12:24

Making use of NumPy entirely, here's an approach with initialization -

def NextPowerOfTwo(number):
    # Returns next power of two following 'number'
    return np.ceil(np.log2(number))

def PadLeft_with_initialization(arr):
    nextPower = NextPowerOfTwo(len(arr))
    deficit = int(np.power(2, nextPower) - len(arr))
    out = np.zeros(deficit+len(arr),dtype=arr.dtype)
    out[deficit:] = arr
    return out

Runtime test

Let's time the proposed solution in this post and np.concatenate based one as listed in Oliver W.'s solution :

def PadLeft_with_concatente(arr): # Oliver W.'s solution
    nextPower = NextPowerOfTwo(len(arr))
    deficit = int(np.power(2, nextPower) - len(arr))
    return np.concatenate((np.zeros(deficit,dtype=arr.dtype), arr))

Timings -

In [226]: arr = np.random.randint(0,9,(600000))

In [227]: %timeit PadLeft_with_concatente(arr)
100 loops, best of 3: 5.21 ms per loop

In [228]: %timeit PadLeft_with_initialization(arr)
100 loops, best of 3: 6.75 ms per loop

Being cleaner and faster, I think Oliver W.'s solution with np.concatenate would be the way to go.

Pad 0's Of Numpy Array To Nearest Power Of Two

3 Answers3

Linked