Confusion about neural networks activation functions

Question

I followed a tutorial about an image classifier using Python and Tensorflow.

I'm now trying to apply deep learning to a custom situation. I made a simulation program of sellers/buyers where the customers buy a stone following its wishes. The stones have a color, a size and a percentage of curve. The nearest of the customer wished values the stone is, the more the customer is able to pay. For the seller, the rarest the stone is, the higher the price should be. Then the program generates 100.000 purchases of a stone to feed a neural network which will try to beat others sellers. The dataset is looking like that :

I'm now trying to create my neural network. In the tutorial, he is using two Conv2D layers with a relu activation function and a MaxPooling2D, then a Flatten layer, a Dense layer and finally another Dense layer with a sigmoid activation function.

After reading some documentation, I found that the Conv2D layer is for a matrix but my data is already flat, so I prefer to use only Dense layers.

My first question is : does my neural network need a dense layer with a relu function like that :

model.add(Dense(64, activation='relu', input_dim(3)))

If my program generates only positives values ?

My second question is : does my neural network need a sigmoid function if I already normalized my data to make them between 0 and 1 by dividing them like this ? :

X[:,0] /= 256.0
X[:,1] /= 50.0
X[:,2] /= 100.0

These values are the max value of each column. So do I need a sigmoid function ?

Actually my neural network looks like this :

model = Sequential()
model.add(Dense(64, activation='relu', input_dim(3)))
model.add(Dense(64, activation='relu'))
model.add(Dense(1,  activation='sigmoid'))

But I'm confused about the efficientness of my model. Does my neural network could work ? If not, what kind of layers and activation functions I have to use ?

You do need the relu. Your data being positive does not matter. The nonlinearity is still present because your weights may become negative. (Consider an image baswd network which is always positive, and those networks usually have many relu funcs.) Also you do need the sigmoid output. Still your original data's output range does not matter here, since the network's output may still produce values out of that range otherwise. — WiseDev, Jun 17 '19 at 15:35

score 3 · Accepted Answer · answered Jun 17 '19 at 15:48

My first question is : does my neural network need a dense layer with a relu function like that :

Yes. Your network requires the ReLUs even if your data is only positive. The idea of ReLUs (and activation functions in general) is that they add a certain complexity, such that the classifier may learn to generalize.

Consider a CNN that takes images as inputs. The input data here consists of only positive values as well ( [0-1] or [0-255]) and they usually have many and many layers with the ReLU nonlinearity.

If my program generates only positives values ?

Your confusion is that your actual input-output relationship produces only positive values, but your classifier still contains weights that can be negative, so your layer outputs could still be negative otherwise.

Also, if you were not to have any nonlinearities like ReLU, there would be no point in having multiple layers, as they would add no complexity to your classifier.

second question is : does my neural network need a sigmoid function if I already normalized my data to make them between 0 and 1 by dividing them like this?

Yes. You also need the sigmoid. Same reasoning as above. Your data may be positive, but your output layer would still be able to produce negative values or otherwise values outside your expected range.

Having a linear output activation function would make learning nearly impossible, especially if your output range is within [0,1].

score 0 · Answer 2 · answered Jun 17 '19 at 14:51

0

it is fine if you normalize like that (/ max), also you could use sigmoid on input but that will be sort of less accurate on max and min values. But i dont understand why do you use Conv2D layer since it is full connected with only 4 input. Also if you generate dataset full random, the network will not learn anything

answered Jun 17 '19 at 14:51

user8426627

903
1
9
19

My dataset is random but the wishes are constants. Basically the buyers would buy a stone only if its suits their wishes, so I think that the network can evaluate the market trend ? And I'm not using a Conv2D layer especially because my dataset doesn't need it, I was just aksing if I was right about it ? Only dense layers are useful in my case I think, but which kind of activation function I need to use ? – M. Ozn Jun 17 '19 at 14:58
1

yes you do not have convo here, since all fields are interconnected. Take one hidden layer with sig or relu, no activation at the output. If you take real dataset about wishes that would work, and you extract the real worls relation. If you just generate it it will extract just generated relation -> stupid run. The taks is actualy realy simple, maybe you add some other fields, not just 4. And if the relation is simple, the NN is an overkill and you may use just KNN classifier there. – user8426627 Jun 17 '19 at 15:04

Confusion about neural networks activation functions

2 Answers2