2

What does the negative slope in a LeakyReLU function refer to?

The term "negative slope" is used in the documentation of both TensorFlow and Pytorch, but it does not seem to point to reality.
The slope of a LeakyReLU function for both positive and negative inputs is generally non-negative.

The Pytorch and TensorFlow docs provide examples of setting the negative slope, and they both use a positive value. TensorFlow explicitly enforces non-negative values. (See below)

Are they just wrong or am I missing something?

Pytorch docs:

CLASStorch.nn.LeakyReLU(negative_slope=0.01, inplace=False)

Tensorflow LeakyReLU:

Args alpha Float >= 0. Negative slope coefficient. Default to 0.3.

iacob
  • 20,084
  • 6
  • 92
  • 119
user25004
  • 1,868
  • 1
  • 22
  • 47
  • 3
    I've always read as the slope for the negative part. You can suggest edits to their repository using more accurate terminology in the doc. I guess they'd welcome that. – Berriel Aug 20 '21 at 19:34
  • 2
    I believe this question doesn't require an answer as the reasoning behind the name (even though `negative_slope` is not perfectly accurate) has been given. You could go on their GitHub repository and see if you can/want to change it... Voting to close. – Ivan Aug 20 '21 at 19:50
  • 1
    Question was edited, now I think is ok, reopened it. – zabop Sep 01 '21 at 14:33
  • The question hasn't changed, nor the reasons for closing it in the first place. I don't see why it has been reopened. Needless to say, it has already received an answer: above in the comments and below. What more to say? – Ivan Sep 01 '21 at 14:36

1 Answers1

2

negative_slope in this context means the negative half of the Leaky ReLU's slope. It is not describing a slope which is necessarily negative.

When naming kwargs it's normal to use concise terms, and here "negative slope" and "positive slope" refer to the slopes of the linear splines spanning the negative [-∞,0] and positive (0,∞] halves of the Leaky ReLU's domain.

iacob
  • 20,084
  • 6
  • 92
  • 119
  • @jacob, it is not only in kwargs naming of LeakyReLU that negative_slope term is used. It is used more than that in their documentation. Eg Tensorflow uses a similar terminology when explaining the meaning of alpha as well: "Args alpha Float >= 0. Negative slope coefficient. Default to 0.3." Without ever clarifying what they meant. – user25004 Sep 01 '21 at 23:12