R mxnet create a 3 input 1 output regression with 3x3 hidden layers

Question

I have a CSV file with 4 columns. 3 inputs and one output. Already normalized. I can use nnet and neuralnet to train a network with 3 inputs, 3 hidden layers with 3 nodes each and one output. It works.

I would like to do the same with MXNET but the parameter for a "FullyConected" has to be hidden = 1 when doing a regression. Any other value just throws an Error message.

How do I build a network as the one in the title or in this image?

NeuralNet Plot

This is the code:

csvIn <- read.csv("normalized.csv")

require(mxnet)

inputData <- csvIn[,1:3]
outputData <- csvIn[,4]

lcinm <- data.matrix(inputData, rownames.force = "NA")
lcoutm <- data.matrix(outputData, rownames.force = "NA")
lcouta <- as.numeric(lcoutm)

data <- mx.symbol.Variable("data")
fc1 <- mx.symbol.FullyConnected(data, name="fc1", num_hidden=3)
act1 <- mx.symbol.Activation(fc1, name="sigm1", act_type="sigmoid")
fc2 <- mx.symbol.FullyConnected(act1, name="fc2", num_hidden=3)
act2 <- mx.symbol.Activation(fc2, name="sigm2", act_type="sigmoid")
fc3 <- mx.symbol.FullyConnected(act2, name="fc3", num_hidden=3)
softmax <- mx.symbol.LinearRegressionOutput(fc3, name="softmax")

mx.set.seed(0)
mxn <- mx.model.FeedForward.create(array.layout = "rowmajor", softmax, X = lcinm, y = lcouta, learning.rate=0.07, eval.metric=mx.metric.rmse)

This is the error message:

Start training with 1 devices
[08:54:33] C:/Users/qkou/mxnet/dmlc-core/include/dmlc/logging.h:235: [08:54:33] src/ndarray/ndarray.cc:231: Check failed: from.shape() == to->shape() operands shape mismatch
Error in exec$update.arg.arrays(arg.arrays, match.name, skip.null) : 
  [08:54:33] src/ndarray/ndarray.cc:231: Check failed: from.shape() == to->shape() operands shape mismatch

Input data (3 nodes)

> lcinm
                  INA          INV        INC
     [1,] 0.327172792 0.1842063931 0.50227366
     [2,] 0.328585645 0.1911366252 0.50394467
     [3,] 0.329998499 0.1980668574 0.50557458
     [4,] 0.333367019 0.1994041603 0.50606766
     [5,] 0.338691205 0.2007416800 0.50656075
     [6,] 0.344015391 0.2020789830 0.50705383
     [7,] 0.345432095 0.2021049795 0.50698534
     [8,] 0.346848798 0.2021309760 0.50691686
     [9,] 0.348355970 0.2026784188 0.50617724
    [10,] 0.349953611 0.2032256450 0.50542391

Output data (1 node)

> lcouta
   [1] 0.6334235 0.6336314 0.6338394 0.6339434 0.6339434 0.6339434
   [7] 0.6306156 0.6272879 0.6241681 0.6212562 0.6183444 0.6170965

score 0 · Answer 1 · answered Jan 17 '17 at 22:38

0

For FullyConnected API, num_hidden is the number of hidden units of this layer. To define multiple hidden layers in one network, you can do something like this:

>>> import mxnet as mx
>>> net = mx.symbol.Variable('data')
>>> net = mx.symbol.FullyConnected(data=net, name='fc1', num_hidden=128)
>>> net = mx.symbol.Activation(data=net, name='relu1', act_type="relu")
>>> net = mx.symbol.FullyConnected(data=net, name='fc2', num_hidden=64)
>>> net = mx.symbol.Activation(data=net, name='relu2', act_type="relu")
>>> net = mx.symbol.FullyConnected(data=net, name='fc3', num_hidden=32)
>>> net = mx.symbol.SoftmaxOutput(data=net, name='out')

This is written in python. You should be able to do similar things in R.

answered Jan 17 '17 at 22:38

kevinthesun

336
1
5

Hi, thanks. but this copy paste that is all over the place is for RELU which is for Classification (0/1) not for Regression like "sigmoid" or "tanh" – David Jan 19 '17 at 22:57
You can change the activation function type as you want. net = mx.symbol.Activation(data=net, name='tanh1', act_type="tanh") – kevinthesun Jan 19 '17 at 23:17
If you change the activation type then it does not work (it throws an error) unless num_hidden=1. – David Jan 19 '17 at 23:21
num_hidden doesn't relate to type of activation function, since act function is only for element-wise multiplication. Can you post you code? – kevinthesun Jan 19 '17 at 23:26
it is identical to yours but instead of "relu" says "sigmoid" try that and you should get an error. If you don't I'll post my code and the output. – David Jan 20 '17 at 11:24
I didn't get any error by using tanh or sigmoid instead of relu in python. – kevinthesun Jan 20 '17 at 18:54
I have edited the post to add the code and some more input – David Jan 23 '17 at 12:02

score 0 · Accepted Answer · answered Jan 24 '17 at 05:51

Try the following:

data <- mx.symbol.Variable("data")
fc1 <- mx.symbol.FullyConnected(data, name="fc1", num_hidden=3)
act1 <- mx.symbol.Activation(fc1, name="sigm1", act_type="sigmoid")
fc2 <- mx.symbol.FullyConnected(act1, name="fc2", num_hidden=3)
act2 <- mx.symbol.Activation(fc2, name="sigm2", act_type="sigmoid")
fc3 <- mx.symbol.FullyConnected(act2, name="fc3", num_hidden=3)
act3 <- mx.symbol.Activation(fc3, name="sigm3", act_type="sigmoid")
fc4 <- mx.symbol.FullyConnected(act3, name="fc4", num_hidden=1)
linear_reg_ output <- mx.symbol.LinearRegressionOutput(fc4, name="output")

Here fc4 is actually the output of the whole network. We use LinearRegressionOutput as final output to enable the network to optimize for squared loss.You can also directly use fc4 as output and write you own loss function.

Thanks! that worked! So the issue is that I can't connect a 3 node layer to the output so you connected it to a 1 node hidden and then to the output? — David, Jan 25 '17 at 12:05
Yes. Actually fc4 and linear_reg_output together form the output. — kevinthesun, Feb 02 '17 at 14:13

R mxnet create a 3 input 1 output regression with 3x3 hidden layers

2 Answers2