encounter error during reproducing the Maximum Likelihood Estimation for normal distribution

Question

I managed to reproduce this post and am trying to understand its logic.

here is the code.

x = [4, 5, 7, 8, 8, 9, 10, 5, 2, 3, 5, 4, 8, 9]
# Plot the Maximum Likelihood Functions for different values of mu 
# and sigma
def plot_ll(x):
    plt.figure(figsize=(5,8))
    plt.title("Maximim Likelihood Functions")
    plt.xlabel("Mean Estimate")
    plt.ylabel("Log Likelihood")
    plt.ylim(-40, -30)
    plt.xlim(0, 12)
    mu_set = np.linspace(0, 16, 1000)
    sd_set = [.5, 1, 1.5, 2.5, 3, 3.5]
    max_val = max_val_location = None
    for sd_hat in sd_set:
        ll_array = []
        for mu_hat in mu_set:
            temp_mm = 0
            for smp in x:
                temp_mm += np.log(norm.pdf(smp, mu_hat, sd_hat)) # The LL function
            ll_array.append(temp_mm)
            if (max_val is None):
                max_val = max(ll_array)
            elif max(ll_array) > max_val:
                max_val = max(ll_array)
                max_val_location = mu_hat
        # Plot the results
        plt.plot(mu_set, ll_array, label="sd: %.1f" % sd_hat)
        print("The max LL for sd %.2f is %.2f" % (sd_hat, max(ll_array)))
    plt.axvline(x=max_val_location, color='black', ls='-.')
    plt.legend(loc='lower left')
plot_ll(x)

I have mastered norm.pdf, log likelihood implementation.

temp_mm is used to cache the likelihood of x for mu = mu_hat and sd = sd_hat.

ll_array is to cache all the likelihood for each element in the sample x.

max(ll_array) is to find the max likelihood.

why mu_hat is considered as the location? whose location?

Luke Lefebure · Answer 1 · 2019-04-21T13:29:14.540

The max_val_location variable refers to the value of mu that corresponds to the largest log likelihood, so it's the "location" in the mu_set that produces the largest log likelihood. I find this implementation a little overcomplicated though.

Let's take a closer look at this if/elif block. The point of this is to keep track of the maximum value of the LL seen so far (max_val), and the value of mu that produced that maximum value of the LL. The use of max here is unnecessary and inefficient since we are taking a running maximum and only need to check if the latest value of the LL is larger than the largest value that we've seen so far. Also, there should be a max_val_location = mu_hat under the if block as well. Better yet, this could be one if that combines the two conditions.

if (max_val is None):
    max_val = max(ll_array)
elif max(ll_array) > max_val:
    max_val = max(ll_array)
    max_val_location = mu_hat

I've rewritten the example slightly to hopefully make it more clear, including renaming max_val_location to max_val_parameters.

x = [4, 5, 7, 8, 8, 9, 10, 5, 2, 3, 5, 4, 8, 9]
# Plot the Maximum Likelihood Functions for different values of mu 
# and sigma
def plot_ll(x):
    plt.figure(figsize=(5,8))
    plt.title("Maximim Likelihood Functions")
    plt.xlabel("Mean Estimate")
    plt.ylabel("Log Likelihood")
    plt.ylim(-40, -30)
    plt.xlim(0, 12)
    mu_set = np.linspace(0, 16, 1000)
    sd_set = [.5, 1, 1.5, 2.5, 3, 3.5]
    max_val = None
    max_val_parameters = None # Keeps track of the (mu, sigma) that produces
                              # the maximum value of the LL
    for sd_hat in sd_set:
        ll_array = []
        for mu_hat in mu_set:
            temp_mm = 0
            for smp in x:
                temp_mm += np.log(norm.pdf(smp, mu_hat, sd_hat)) # The LL function
            ll_array.append(temp_mm)
            if max_val is None or temp_mm > max_val:
                # This `temp_mm` is the largest value of the LL that we've
                # seen so far, so keep track of its value and the (mu, sd)
                # that produced this value.
                max_val = temp_mm
                max_val_parameters = (mu_hat, sd_hat)
        # Plot the results
        plt.plot(mu_set, ll_array, label="sd: %.1f" % sd_hat)
        print("The max LL for sd %.2f is %.2f" % (sd_hat, max(ll_array)))
    plt.axvline(x=max_val_parameters[0], color='black', ls='-.')
    plt.legend(loc='lower left')
plot_ll(x)

encounter error during reproducing the Maximum Likelihood Estimation for normal distribution

1 Answers1