1

I am trying to train a CNN using Lasagne and NoLearn. After, the learning is over, I want to store the neuralnetwork parameters.

Specifically,

nn = net1.fit(X_train, y_train) # train neural net

with open('nn.pickle', 'wb') as f:
    pickle.dump(nn2, f, -1)

I am not being able to use pickle() because the size of object (nn) is some tens of GB. How can I store such a large sized object on disc?

Avisek

ForceBru
  • 43,482
  • 10
  • 63
  • 98
user3215384
  • 61
  • 3

2 Answers2

2

Instead of saving the entire network using pickle, you can save just its parameters using:

values = lasagne.layers.get_all_param_values(net1)

In the official Lasagne tutorial, you can see that get_all_param_values returns a list of numpy arrays representing the parameter values. You can save them using numpy.savez, able to store several arrays into a single file in uncompressed .npz format. You can also compress them using numpy.savez_compressed.

In order to load the parameters into your CNN, first you have to generate the network architecture (obviously it must be the same you previously trained), then you can load from disk the parameters using np.load and finally assign them to the network calling:

lasagne.layers.set_all_param_values(net1, values)

Federico Pala
  • 21
  • 4
0

If you want to store a Python object to disc, you need serialization. When you want serialization, you need pickle or ... it's 'brother' cPickle which can sometimes be thousands times faster than pickle.

ForceBru
  • 43,482
  • 10
  • 63
  • 98