0

The closed-form analytical solution for the entropy of a variable X that follows the t-distribution, derived here, is

enter image description here

Seeing that python has functions scipy.special.digamma and scipy.special.beta, how can the above formula be implemented in code?

What confuses me is that the functions just mentioned do not take the degrees of freedom parameter nu (v) as an input according to the documentation. A running example would help

develarist
  • 1,224
  • 1
  • 13
  • 34
  • 1
    Hi, are you looking for the python formulation of the equation above? Or, is it that you want to know how does that function works? If it's the first, I can share a simple code with you. If it's the second, then from what I understood the formula above is obtained by integrating the Shannon formula of entropy, applied to the Student-t probability density. Beta and Digmma functions do take the degree of freedom as an input only as a result of the integration. They are just two mathematical formulas used to simplify the expression. – Yassine Majdoub Nov 17 '20 at 23:17
  • Both would be very helpful – develarist Nov 17 '20 at 23:17

2 Answers2

2

By its definition, the entropy is defined by Shanoon as: enter image description here

Now if you apply this formula to the Student-t distribution, you will notice that this one already contains the degree of freedom parameter (v): enter image description here

As a result of the integration you will have in the approximation both Betta and Digmma. If you can the calculation, honestly I couldn't, you will find out that these take v as an input just as a result of the calculation. It is not in their definition.

v varies between 1 (Cauchy distribution) and infinity (the normal distribution).

To simplify the calculations, I used the code below:

import numpy as np
import scipy.special as sc
v = float(input('Degre of freedom '))
v1 = (1+v)/2
v2 = v/2
Entropy_of_Variable_X = v1*(sc.digamma(v1)-sc.digamma(v2))+np.log(np.sqrt(v)*sc.beta(v2,0.5))
print('Entropy of the variable X, of degree of freedom equal to : ', v, 'is ', Entropy_of_Variable_X)

You can pass it a list or something like that to calculate the entropy for multiple distribution.

Yassine Majdoub
  • 154
  • 9
  • isn't the second formula just a re-statement of the t-distribution pdf, except where the inverse beta function B replaces what normally are Gamma functions? https://wikimedia.org/api/rest_v1/media/math/render/svg/7fb35627dbb7e3dec4f14d60b0b58ea399966f46 – develarist Nov 17 '20 at 23:56
  • Yes. It's the same formula – Yassine Majdoub Nov 18 '20 at 12:57
1

You can also use the differential entropy of multivariate student t-distribution, where, dim is dimensional, dof is degree of freedom and cmtx is covariance.

import numpy as np
import scipy.special as sc
def compute_true_entropy(dim=1, dof=3, std=False):
    cmtx = np.identity(dim)
    B0 = 0.5*np.log(np.linalg.det(cmtx))
    B1 = sc.gamma((dim+dof)/2)/((sc.gamma(dof/2))*((np.pi*dof)**(dim/2)))
    B2 = ((dof+dim)/2)*(sc.digamma((dof+dim)/2) - sc.digamma((dof)/2))
    entropy = B0 - np.log(B1) + B2
   return entropy
M.cadirci
  • 26
  • 5
  • thanks for extending to the multivariate case. how would the function be changed to accomodate the `std` input. Would that be a vector of different variables' standard deviations if they are not =1? currently, `std` is absent within the function – develarist Dec 16 '20 at 12:52