Initialise numpy array of unknown length

Question

I want to be able to 'build' a numpy array on the fly, I do not know the size of this array in advance.

For example I want to do something like this:

a= np.array()
for x in y:
     a.append(x)

Which would result in a containing all the elements of x, obviously this is a trivial answer. I am just curious whether this is possible?

What may be a more efficient approach is to allocate some large array, and double the size of it every time you reach capacity. — wim, Apr 24 '13 at 01:09

score 126 · Accepted Answer · answered Apr 12 '12 at 11:04

126

Build a Python list and convert that to a Numpy array. That takes amortized O(1) time per append + O(n) for the conversion to array, for a total of O(n).

    a = []
    for x in y:
        a.append(x)
    a = np.array(a)

answered Apr 12 '12 at 11:04

Fred Foo

355,277
75
744
836

3

Or better still: `a = np.array([x for x in y])`; or just `a = np.array(list(y))` – A T Jan 19 '18 at 23:58

score 19 · Answer 2 · answered Apr 12 '12 at 10:57

19

You can do this:

a = np.array([])
for x in y:
    a = np.append(a, x)

answered Apr 12 '12 at 10:57

alexisdm

29,448
6
64
99

6

That takes linear time per append. – Fred Foo Apr 12 '12 at 11:02
13

This approach copies the array every append, which is O(sum(range(n))). On my laptop, this method was 42 times slower than @larsman's method: Building a list following larsmans method exactly takes me 1000 loops, best of 3: 1.53 ms per loop . Following this method exactly takes me 10 loops, best of 3: 64.8 ms per loop. – Alex Gaudio Feb 14 '13 at 21:59
1

Code should be readable, not fast, unless speed is your bottleneck. Why optimize code you'll run once? – Marcos Pereira May 20 '20 at 08:01
Nice comment @AlexGaudio . I really didn't know that. Thank you! – Prasad Raghavendra Jun 20 '20 at 14:55

Mr_and_Mrs_D · Answer 3 · 2017-04-05T13:14:11.300

7

Since y is an iterable I really do not see why the calls to append:

a = np.array(list(y))

will do and it's much faster:

import timeit

print timeit.timeit('list(s)', 's=set(x for x in xrange(1000))')
# 23.952975494633154

print timeit.timeit("""li=[]
for x in s: li.append(x)""", 's=set(x for x in xrange(1000))')
# 189.3826994248866

edited Apr 05 '17 at 13:14

answered Mar 27 '17 at 22:34

Mr_and_Mrs_D

32,208
39
178
361

BenDundee · Answer 4 · 2013-12-04T13:50:03.233

1

For posterity, I think this is quicker:

a = np.array([np.array(list()) for _ in y])

You might even be able to pass in a generator (i.e. [] -> ()), in which case the inner list is never fully stored in memory.

Responding to comment below:

>>> import numpy as np
>>> y = range(10)
>>> a = np.array([np.array(list) for _ in y])
>>> a
array([array(<type 'list'>, dtype=object),
       array(<type 'list'>, dtype=object),
       array(<type 'list'>, dtype=object),
       array(<type 'list'>, dtype=object),
       array(<type 'list'>, dtype=object),
       array(<type 'list'>, dtype=object),
       array(<type 'list'>, dtype=object),
       array(<type 'list'>, dtype=object),
       array(<type 'list'>, dtype=object),
       array(<type 'list'>, dtype=object)], dtype=object)

edited Dec 04 '13 at 13:50

answered Apr 24 '13 at 01:07

BenDundee

4,389
3
28
34

I made a change here: list(_) and that worked great – WestCoastProjects Dec 04 '13 at 03:37
To be clear @javadba, you don't need to do that--I'm sure there's some Pythonistas who would take offense :) – BenDundee Dec 04 '13 at 13:48
1

This is not a matter of style. without the list(_) it does not even work at last for the case i have that y is an array itself – WestCoastProjects Dec 04 '13 at 16:54
The OP says "which would result in `a` containing all the elements of x", so you def do need to do `list(_)` – Akaisteph7 Jul 19 '19 at 14:48

score 1 · Answer 5 · answered Aug 08 '16 at 21:19

1

a = np.empty(0)
for x in y:
    a = np.append(a, x)

answered Aug 08 '16 at 21:19

chiefenne

565
2
15
30

This won't work when x has a non-scalar dimension. For example, if x = np.ones((3,5)), it will fail. – Shubham Agrawal Mar 02 '21 at 07:57

Shubham Agrawal · Answer 6 · 2021-03-02T07:56:10.350

I wrote a small utility function. (most answers above are good. I feel this looks nicer)

def np_unknown_cat(acc, arr):
  arrE = np.expand_dims(arr, axis=0)
  if acc is None:
    return arrE
  else:
    return np.concatenate((acc, arrE))

You can use the above function as the following:

acc = None  # accumulator
arr1 = np.ones((3,4))
acc = np_unknown_cat(acc, arr1)
arr2 = np.ones((3,4))
acc = np_unknown_cat(acc, arr2)

score -1 · Answer 7 · answered Aug 10 '22 at 05:56

list1 = []
size = 1
option = "Y"
for x in range(size):
    ele = input("Enter Element For List One : ")
    list1.append(ele)
while(option == "Y"):
    option = input("\n***Add More Element Press Y ***: ")
    if(option=="Y"):
        size = size + 1
        for x in range(size):
            ele = input("Enter Element For List Element : ")
            list1.append(ele)
            size = 1
    else:
        break;
print(list1)

Take:

list1 = []    # Store Array Element 
size = 1      # Rune at One Time 
option = "Y"  # Take User Choice

Implementation:

For Loop at a range of size variable which runs one time because size = 1
use While loop
- take input from users if they want to add press "Y" else "N"
- in the while loop use the if else statement if the option is "Y" then increase size with which helps run 2 time array because size = size + 1 run another for loop at a range of size
- else break

Initialise numpy array of unknown length

7 Answers7

Linked