np.savetxt triggers ValueError. Why?

Question

Earlier in the program, I calculate the matrices P and K. I need the eigenvalues of PKP. Since this takes a while (it's a 10000 x 10000 matrix), I want to save the result someplace so it's easily accessible even if I restart the computer.

The last two lines of my program are:

eigs=np.linalg.eig(P@K@P)
np.savetxt('eigdata.txt',eigs)

This triggers the error:

ValueError: could not broadcast input array from shape (10000,10000) into shape (10000)

Deleting the last time removes the error, so the problem is definitely with np.savetxt.

How do I fix this?

Thanks

Right before the Value Error... File "C:\Users\(my name)\Anaconda3\lib\site-packages\numpy\core\numeric.py", line 538, in asarray return array(a, dtype, copy=False, order=order) — J.D., Aug 02 '19 at 08:37
can you please provide shapes of P and K? seems like error caused by dot product P@K@P — alexey, Aug 02 '19 at 08:44
Try to use also ```np.savez_compressed('eigs.npz', eigs)```. I think that could be a best option to save huge arrays. Then load it with ```np.load('eigs.npz')['arr_0']``` — Victor Ruiz, Aug 02 '19 at 08:46
P and K are both 10000 x 10000. Ok, will try Victor Ruiz's suggestion. — J.D., Aug 02 '19 at 08:48
Just tried closing and reopening Python, then adding the line ```np.load('eigs.npz')['arr_0']``` Got the error message ValueError: Cannot load file containing pickled data when allow_pickle=False — J.D., Aug 02 '19 at 09:10

alexey · Answer 1 · 2019-08-02T08:55:36.210

1

np.linalg.eig outputs two np.ndarray one of shape (1000,), the second of shape (1000, 1000). You should either save them into different files or use np.savez or np.savez_compressed istead:

np.savez('eigdata', *eigs)

And restore them later:

w, v = np.load('eigdata.npz').values()

edited Aug 02 '19 at 08:55

answered Aug 02 '19 at 08:52

alexey

706
5
9

Better ```np.savez_compressed``` for huge arrays. – Victor Ruiz Aug 02 '19 at 08:55
As said in my reply to my question... ValueError: Cannot load file containing pickled data when allow_pickle=False – J.D. Aug 02 '19 at 10:17
well, it's slightly a different problem. try passing `allow_pickle=True` into call that crashes – alexey Aug 02 '19 at 10:19
Changed it to np.load('eigs.npz',allow_picle=True). Got eigOSError: Failed to interpret file 'eigs.npz' as a pickle – J.D. Aug 02 '19 at 10:29
use `allow_pickle=True` in while saving as well – alexey Aug 02 '19 at 11:10
Alas, np.savez_compressed('eigs.npz',eigs,allow_pickle=True) yielded the error message ValueError: could not broadcast input array from shape (10000,10000) into shape (10000) – J.D. Aug 03 '19 at 01:51
you missed a start `np.savez_compressed('eigs.npz', *eigs, allow_pickle=True)` – alexey Aug 03 '19 at 11:02

hpaulj · Answer 2 · 2019-08-03T08:21:43.787

Using a simple example from eig:

In [332]: a = np.array([[1, 1j], [-1j, 1]]) 
     ...: w,v = np.linalg.eig(a)                                                                             

In [334]: np.savez('eig.npz', w=w, v=v)                                                                      


In [335]: d = np.load('eig.npz')                                                                             
In [336]: list(d.keys())                                                                                     
Out[336]: ['w', 'v']
In [337]: d['w']                                                                                             
Out[337]: array([2.+0.j, 0.+0.j])
In [338]: d['v']                                                                                             
Out[338]: 
array([[ 0.        +0.70710678j,  0.70710678+0.j        ],
       [ 0.70710678+0.j        , -0.        +0.70710678j]])

Or without the keyword names:

In [339]: np.savez('eig.npz', w,v)                                                                           
In [340]: d = np.load('eig.npz')                                                                             
In [341]: list(d.keys())                                                                                     
Out[341]: ['arr_0', 'arr_1']
In [342]: d['arr_0']                                                                                         
Out[342]: array([2.+0.j, 0.+0.j])

i suspect that if you get a pickle error, it's because you are saving an object like a tuple or dictionary:

In [345]: np.savez('eig.npz', {'w':w, 'v':v})                                                                
In [346]: d = np.load('eig.npz')                                                                             
In [347]: list(d.keys())                                                                                     
Out[347]: ['arr_0']
In [348]: d['arr_0']                                                                                         
...
ValueError: Object arrays cannot be loaded when allow_pickle=False

Recent numpy versions have become picky about this pickling parameter.

Or you could save the arrays to two files:

In [370]: np.save('eig_w.npy',w); np.save('eig_v.npy',v)                                                     
In [371]: np.load('eig_w.npy')                                                                               
Out[371]: array([2.+0.j, 0.+0.j])
In [372]: np.load('eig_v.npy')                                                                               
Out[372]: 
array([[ 0.        +0.70710678j,  0.70710678+0.j        ],
       [ 0.70710678+0.j        , -0.        +0.70710678j]])

===

With:

In [373]: eigs = np.linalg.eig(a)
In [375]: np.savez('eig.npz', eigs) 
ValueError: could not broadcast input array from shape (2,2) into shape (2)

That error is produced by a np.array(eigs) step. np.save is intended to saving arrays. Given a tuple it tries to turn it into an array.

Saving *eigs is like the In [339] example above:

In [385]: np.savez('eig.npz', *eigs)                                                                         
In [386]: d = np.load('eig.npz')                                                          
In [387]: list(d.keys())                                                                                     
Out[387]: ['arr_0', 'arr_1']
In [388]: d['arr_1']                                                                                         
Out[388]: 
array([[ 0.        +0.70710678j,  0.70710678+0.j        ],
       [ 0.70710678+0.j        , -0.        +0.70710678j]])
In [389]: d['arr_0']                                                                                         
Out[389]: array([2.+0.j, 0.+0.j])

np.savetxt triggers ValueError. Why?

2 Answers2

Linked