While I am training a network in Keras+Theano, it is giving error.

Question

I was trying to train a network in keras with theano backed. I am beginner in keras and theano. It is showing a lot of error, could you please if someone knows how to solve it, please share it here?

train images/masks shape:
(20, 1, 565, 565)
train images range (min-max): 0.0 - 1.0
train masks are within 0-1

patches per full image: 9500

train PATCHES images/masks shape:
(190000, 1, 48, 48)
train PATCHES images range (min-max): 0.0 - 1.0
Traceback (most recent call last):
  File "./src/NN_training.py", line 167, in <module>
    model = get_net(n_ch, patch_height, patch_width)  #the model
  File "./src/NN_training.py", line 36, in get_net
    conv1 = Dropout(0.2)(conv1)
  File "/home/ss/anaconda2/envs/kerasenv/lib/python2.7/site-packages/keras/engine/topology.py", line 572, in __call__
    self.add_inbound_node(inbound_layers, node_indices, tensor_indices)
  File "/home/ss/anaconda2/envs/kerasenv/lib/python2.7/site-packages/keras/engine/topology.py", line 635, in add_inbound_node
    Node.create_node(self, inbound_layers, node_indices, tensor_indices)
  File "/home/ss/anaconda2/envs/kerasenv/lib/python2.7/site-packages/keras/engine/topology.py", line 166, in create_node
    output_tensors = to_list(outbound_layer.call(input_tensors[0], mask=input_masks[0]))
  File "/home/ss/anaconda2/envs/kerasenv/lib/python2.7/site-packages/keras/layers/core.py", line 111, in call
    x = K.in_train_phase(dropped_inputs, lambda: x)
  File "/home/ss/anaconda2/envs/kerasenv/lib/python2.7/site-packages/keras/backend/theano_backend.py", line 1167, in in_train_phase
    x = x()
  File "/home/ss/anaconda2/envs/kerasenv/lib/python2.7/site-packages/keras/layers/core.py", line 110, in dropped_inputs
    return K.dropout(x, self.p, noise_shape, seed=self.seed)
  File "/home/ss/anaconda2/envs/kerasenv/lib/python2.7/site-packages/keras/backend/theano_backend.py", line 1297, in dropout
    random_tensor = rng.binomial(x.shape, p=retain_prob, dtype=x.dtype)
  File "/home/ss/anaconda2/envs/kerasenv/lib/python2.7/site-packages/theano/sandbox/rng_mrg.py", line 1359, in binomial
    x = self.uniform(size=size, dtype=dtype, nstreams=nstreams)
  File "/home/ss/anaconda2/envs/kerasenv/lib/python2.7/site-packages/theano/sandbox/rng_mrg.py", line 1321, in uniform
    rstates = self.get_substream_rstates(nstreams, dtype)
  File "/home/ss/anaconda2/envs/kerasenv/lib/python2.7/site-packages/theano/sandbox/rng_mrg.py", line 1223, in get_substream_rstates
    multMatVect(rval[0], A1p72, M1, A2p72, M2)
  File "/home/ss/anaconda2/envs/kerasenv/lib/python2.7/site-packages/theano/sandbox/rng_mrg.py", line 64, in multMatVect
    [A_sym, s_sym, m_sym, A2_sym, s2_sym, m2_sym], o, profile=False)
  File "/home/ss/anaconda2/envs/kerasenv/lib/python2.7/site-packages/theano/compile/function.py", line 320, in function
    output_keys=output_keys)
  File "/home/ss/anaconda2/envs/kerasenv/lib/python2.7/site-packages/theano/compile/pfunc.py", line 479, in pfunc
    output_keys=output_keys)
  File "/home/ss/anaconda2/envs/kerasenv/lib/python2.7/site-packages/theano/compile/function_module.py", line 1777, in orig_function
    defaults)
  File "/home/ss/anaconda2/envs/kerasenv/lib/python2.7/site-packages/theano/compile/function_module.py", line 1641, in create
    input_storage=input_storage_lists, storage_map=storage_map)
  File "/home/ss/anaconda2/envs/kerasenv/lib/python2.7/site-packages/theano/gof/link.py", line 690, in make_thunk
    storage_map=storage_map)[:3]
  File "/home/ss/anaconda2/envs/kerasenv/lib/python2.7/site-packages/theano/gof/vm.py", line 1003, in make_all
    no_recycling))
  File "/home/ss/anaconda2/envs/kerasenv/lib/python2.7/site-packages/theano/gof/op.py", line 970, in make_thunk
    no_recycling)
  File "/home/ss/anaconda2/envs/kerasenv/lib/python2.7/site-packages/theano/gof/op.py", line 879, in make_c_thunk
    output_storage=node_output_storage)
  File "/home/ss/anaconda2/envs/kerasenv/lib/python2.7/site-packages/theano/gof/cc.py", line 1200, in make_thunk
    keep_lock=keep_lock)
  File "/home/ss/anaconda2/envs/kerasenv/lib/python2.7/site-packages/theano/gof/cc.py", line 1143, in __compile__
    keep_lock=keep_lock)
  File "/home/ss/anaconda2/envs/kerasenv/lib/python2.7/site-packages/theano/gof/cc.py", line 1587, in cthunk_factory
    key = self.cmodule_key()
  File "/home/ss/anaconda2/envs/kerasenv/lib/python2.7/site-packages/theano/gof/cc.py", line 1282, in cmodule_key
    compile_args=self.compile_args(),
  File "/home/ss/anaconda2/envs/kerasenv/lib/python2.7/site-packages/theano/gof/cc.py", line 955, in compile_args
    ret += c_compiler.compile_args()
  File "/home/ss/anaconda2/envs/kerasenv/lib/python2.7/site-packages/theano/gof/cmodule.py", line 1879, in compile_args
    native_lines = get_lines("%s -march=native -E -v -" % theano.config.cxx)
  File "/home/ss/anaconda2/envs/kerasenv/lib/python2.7/site-packages/theano/gof/cmodule.py", line 1848, in get_lines
    shell=True)
  File "/home/ss/anaconda2/envs/kerasenv/lib/python2.7/site-packages/theano/misc/windows.py", line 36, in subprocess_Popen
    proc = subprocess.Popen(command, startupinfo=startupinfo, **params)
  File "/home/ss/anaconda2/envs/kerasenv/lib/python2.7/subprocess.py", line 390, in __init__
    errread, errwrite)
  File "/home/ss/anaconda2/envs/kerasenv/lib/python2.7/subprocess.py", line 916, in _execute_child
    self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory

Thanks

You need to provide your network definition. It seems that it's too large to fit in your memory. — Marcin Możejko, Mar 15 '17 at 14:10
@Patric thanks for your comment, `190000` is the number of patches extracted from only 20 images that each patch has 48x48 size. How can I solve this issue? — S.EB, Mar 16 '17 at 09:45
@S.EB you can change a machine with large memory or reduce the network, such as using small input size (batchsize, image height/width) or adding pooling layer after convolution. — Patric, Mar 16 '17 at 11:03

While I am training a network in Keras+Theano, it is giving error.

0 Answers0