Binary entropy function

In information theory, the binary entropy function, denoted ${\displaystyle \operatorname {H} (p)}$ or ${\displaystyle \operatorname {H} _{\text{b}}(p)}$, is defined as the entropy of a Bernoulli process with probability ${\displaystyle p}$ of one of two values. It is a special case of ${\displaystyle \mathrm {H} (X)}$, the entropy function. Mathematically, the Bernoulli trial is modelled as a random variable ${\displaystyle X}$ that can take on only two values: 0 and 1, which are mutually exclusive and exhaustive.

If ${\displaystyle \operatorname {Pr} (X=1)=p}$, then ${\displaystyle \operatorname {Pr} (X=0)=1-p}$ and the entropy of ${\displaystyle X}$ (in shannons) is given by

${\displaystyle \operatorname {H} (X)=\operatorname {H} _{\text{b}}(p)=-p\log _{2}p-(1-p)\log _{2}(1-p)}$,

where ${\displaystyle 0\log _{2}0}$ is taken to be 0. The logarithms in this formula are usually taken (as shown in the graph) to the base 2. See binary logarithm.

When ${\displaystyle p={\tfrac {1}{2}}}$, the binary entropy function attains its maximum value. This is the case of an unbiased coin flip.

${\displaystyle \operatorname {H} (p)}$ is distinguished from the entropy function ${\displaystyle \mathrm {H} (X)}$ in that the former takes a single real number as a parameter whereas the latter takes a distribution or random variable as a parameter. Sometimes the binary entropy function is also written as ${\displaystyle \operatorname {H} _{2}(p)}$. However, it is different from and should not be confused with the Rényi entropy, which is denoted as ${\displaystyle \mathrm {H} _{2}(X)}$.