Python: Define the softmax function

Question

import numpy as np

def softmax(x):
    row_num = x.shape[0]
    col_num = x.shape[1]
    for m in row_num:
        for n in col_num:
            new_x[m,n] = np.exp(x[m,n])/sum(x[:,n])

    return new_x

logits = [1.0, 2.0, 3.0]
logits2 = np.array([
    [1, 2, 3, 6],
    [2, 4, 5, 6],
    [3, 8, 7, 6]])

print(softmax(logits1))
print(softmax(logits2))

Above is the function for softmax (it is used to turn logits to probabilities)

I want to obtain the solution shown as below:

[ 0.09003057  0.24472847  0.66524096]

[
    [ 0.09003057  0.00242826  0.01587624  0.33333333]
    [ 0.24472847  0.01794253  0.11731043  0.33333333]
    [ 0.66524096  0.97962921  0.86681333  0.33333333]
]

However, error was revealed that "'int' object is not iterable". In addition, I want to see a more efficient code for this function with less complexity.

Miriam Farber · Answer 1 · 2017-04-05T04:07:00.447

2

This will do the job:

logits = np.array([1.0, 2.0, 3.0])
logits2 = np.array([
    [1, 2, 3, 6],
    [2, 4, 5, 6],
    [3, 8, 7, 6]])
def softmax(x):
    r=np.exp(x - np.max(x))
    return r/r.sum(axis=0)

You get the error

"'int' object is not iterable"

because row_num (and similarly col_num) is a number, so you cannot iterate over it. You need to add range (that is, for m in range(row_num)).

There are also other issues. For example, x.shape[1] is not necessarily defined (it is not defined for logits) so it will also throw an error. new_x is not defined as well.

edited Apr 05 '17 at 04:07

answered Apr 05 '17 at 04:00

Miriam Farber

18,986
14
61
76

Thanks for your solution. But when I run your code, it revealed "'list' object has no attribute 'shape'" – FortranFun Apr 05 '17 at 04:06
2

@FortranFun In my solution I didn't use shape, so I guess you run your solution after you added range. The problem in this case is that logits is one dimensional vector, so logits.shape is a tuple with one number, so you cannot access to logits.shape[1]. Also, write logits = np.array([1.0, 2.0, 3.0]) (your logits is not numpy array, while logits2 is. you need to make it a numpy array so that it will have shape attribute). – Miriam Farber Apr 05 '17 at 04:09

score 1 · Answer 2 · answered Apr 05 '17 at 04:32

1

The most efficient code should be:

import numpy as np
def softmax(x):
    return np.exp(x) / np.sum(np.exp(x), axis=0)

answered Apr 05 '17 at 04:32

Jeremy_Tamu

725
1
8
21

2

There might be an issue if x has large numbers, such as x=[100,100,1000]. Your code will return array([ 0., 0., nan]), even though the correct answer is actually besically array([ 0., 0., 1.]) – Miriam Farber Apr 05 '17 at 04:37

Nolan Conaway · Answer 3 · 2019-01-03T17:35:49.900

EDIT. As of version 1.2.0, scipy includes softmax as a special function:

https://scipy.github.io/devdocs/generated/scipy.special.softmax.html

In general, it is better to use a vectorized implementation rather than rely on for loops. You can take advantage of numpy's broadcasting to do this. There are a number of other questions possessing correct implementations of such a function (for example: here, here).

For the sake of associating an answer with the question, I'll paste in my general softmax function operating over an arbitrary axis, including a tricky max subtraction bit. I also wrote a more detailed blog post about it.

def softmax(X, theta = 1.0, axis = None):
    """
    Compute the softmax of each element along an axis of X.

    Parameters
    ----------
    X: ND-Array. Probably should be floats. 
    theta (optional): float parameter, used as a multiplier
        prior to exponentiation. Default = 1.0
    axis (optional): axis to compute values along. Default is the 
        first non-singleton axis.

    Returns an array the same size as X. The result will sum to 1
    along the specified axis.
    """

    # make X at least 2d
    y = np.atleast_2d(X)

    # find axis
    if axis is None:
        axis = next(j[0] for j in enumerate(y.shape) if j[1] > 1)

    # multiply y against the theta parameter, 
    y = y * float(theta)

    # subtract the max for numerical stability
    y = y - np.expand_dims(np.max(y, axis = axis), axis)

    # exponentiate y
    y = np.exp(y)

    # take the sum along the specified axis
    ax_sum = np.expand_dims(np.sum(y, axis = axis), axis)

    # finally: divide elementwise
    p = y / ax_sum

    # flatten if X was 1D
    if len(X.shape) == 1: p = p.flatten()

    return p

score 0 · Answer 4 · answered Nov 22 '20 at 06:32

My implementation of softmax function in numpy module is like this:

import numpy as np
def softmax(self,x,axis=0):    
    ex = np.exp(x - np.max(x,axis=axis,keepdims=True))
    return ex / np.sum(ex,axis=axis,keepdims=True) 
np.softmax = softmax.__get__(np)

Then it is possible to use softmax function as a typical numpy built-in function. like :

output = model(x)  # output.shape : (50000,10)
softmaxed_value = np.softmax(output,axis=1)

Python: Define the softmax function

4 Answers4