1

I am trying to write some simple neural network using pytorch. I am new to this library. I faced with two ways of implementing the same idea: a layer with some fixed activation function (e.g. tanh).

The first way to implement it:

l1 = nn.Tanh(n_in, n_out)

The second way:

l2 = nn.Linear(n_in, n_out) # linear layer, that do nothing with its input except summation

but in forward propagation use:

import torch.nn.functional as F
x = F.tanh(l2(x)) # x - value that propagates from layer to layer

What are the differences between those mechanisms? Which one is better for which purposes?

Wasi Ahmad
  • 35,739
  • 32
  • 114
  • 161
MefAldemisov
  • 867
  • 10
  • 21

1 Answers1

1

An activation function is just a non-linear function and it doesn't have any parameters. So, your first approach doesn't make any sense!

However, you can use a sequential wrapper to combine a linear layer with tanh activation.

model = nn.Sequential(
    nn.Linear(n_in, n_out),
    nn.Tanh()
)
output = model(input)
Wasi Ahmad
  • 35,739
  • 32
  • 114
  • 161