Directed information

Directed information is an information theory measure that quantifies the information flow from the random string ${\displaystyle X^{n}=(X_{1},X_{2},\dots ,X_{n})}$ to the random string ${\displaystyle Y^{n}=(Y_{1},Y_{2},\dots ,Y_{n})}$. The term directed information was coined by James Massey and is defined as

${\displaystyle I(X^{n}\to Y^{n})\triangleq \sum _{i=1}^{n}I(X^{i};Y_{i}|Y^{i-1})}$

where ${\displaystyle I(X^{i};Y_{i}|Y^{i-1})}$ is the conditional mutual information ${\displaystyle I(X_{1},X_{2},...,X_{i};Y_{i}|Y_{1},Y_{2},...,Y_{i-1})}$.

Directed information has applications to problems where causality plays an important role such as the capacity of channels with feedback, capacity of discrete memoryless networks, capacity of networks with in-block memory, gambling with causal side information, compression with causal side information, real-time control communication settings, and statistical physics.