Information Theory Connection: Entropy and KL Divergence

Information theory offers a powerful lens through which to view and quantify aspects of Bayesian inference. At its core, Bayesian learning is about updating our beliefs (represented by probability distributions) in light of new data. Information theory provides the tools to measure the uncertainty inherent in these beliefs and the "distance" or divergence between different belief states. Two fundamental concepts here are entropy and Kullback-Leibler (KL) divergence.

Shannon Entropy: Quantifying Uncertainty

Shannon entropy measures the average level of "information", "surprise", or "uncertainty" inherent in a random variable's possible outcomes. For a discrete random variable $X$ with probability mass function $p(x)$, the entropy $H(X)$ is defined as:

$$H(X) = - \sum_{x \in \mathcal{X}} p(x) \log p(x)$$

For a continuous random variable $X$ with probability density function $p(x)$, the differential entropy is:

$$H(X) = - \int_{-\infty}^{\infty} p(x) \log p(x) dx$$

The base of the logarithm determines the units (base 2 gives bits, base $e$ gives nats). Higher entropy implies greater uncertainty about the outcome of $X$. A distribution sharply peaked around a single value has low entropy, while a uniform distribution over a wide range has high entropy.

In Bayesian modeling:

• Prior Uncertainty: The entropy of the prior distribution $P(\theta)$ quantifies our initial uncertainty about the parameters $\theta$ before observing any data. Broad, uninformative priors generally have higher entropy. The principle of maximum entropy is sometimes used to select priors that encode minimal assumptions.
• Posterior Uncertainty: The entropy of the posterior distribution $P(\theta | \mathcal{D})$ quantifies our remaining uncertainty after observing data $\mathcal{D}$. Successful learning should ideally lead to a posterior with lower entropy than the prior, indicating that the data has reduced our uncertainty about $\theta$.

Kullback-Leibler (KL) Divergence: Measuring Differences Between Distributions

While entropy measures the uncertainty of a single distribution, KL divergence quantifies how one probability distribution $P$ differs from a second, reference probability distribution $Q$. It's often interpreted as the information lost when $Q$ is used to approximate $P$, or the relative entropy of $P$ with respect to $Q$.

For discrete distributions $P(x)$ and $Q(x)$:

$$D_{KL}(P || Q) = \sum_{x \in \mathcal{X}} p(x) \log \frac{p(x)}{q(x)}$$

For continuous distributions $p(x)$ and $q(x)$:

$$D_{KL}(p || q) = \int_{-\infty}^{\infty} p(x) \log \frac{p(x)}{q(x)} dx$$

Important properties of KL divergence:

1. Non-negativity: $D_{KL}(P || Q) \ge 0$, with equality if and only if $P = Q$.
2. Asymmetry: In general, $D_{KL}(P || Q) \neq D_{KL}(Q || P)$. This means it's not a true distance metric, but rather a directed divergence. The choice of which distribution comes first matters.

The asymmetry has important implications. Minimizing $D_{KL}(P || Q)$ encourages $Q$ to be non-zero wherever $P$ is non-zero (it tries to cover $P$). Minimizing $D_{KL}(Q || P)$ encourages $Q$ to be zero wherever $P$ is zero (it tries to be contained within $P$).

KL divergence $D_{KL}(P || Q)$ quantifies the difference between distribution P (blue) and distribution Q (pink). It measures the inefficiency of using Q when the true distribution is P. Note the asymmetry, $D_{KL}(P || Q)$ would yield a different value than $D_{KL}(Q || P)$.

The Role in Bayesian Inference

The connection between information theory and Bayesian methods becomes particularly apparent when dealing with the computational challenges mentioned earlier, especially intractable posterior distributions $P(\theta | \mathcal{D})$.

Variational Inference (VI): This is a major application area for KL divergence. VI reframes Bayesian inference as an optimization problem. We seek an approximation $Q(\theta)$ from a tractable family of distributions (e.g., Gaussians) that is "closest" to the true, often intractable, posterior $P(\theta | \mathcal{D})$. "Closest" is typically measured using KL divergence. Specifically, VI aims to minimize $D_{KL}(Q(\theta) || P(\theta | \mathcal{D}))$. Directly minimizing this is still hard because it involves the unknown posterior. However, minimizing this KL divergence is equivalent to maximizing a quantity called the Evidence Lower Bound (ELBO):

$$\text{ELBO}(Q) = \mathbb{E}_{Q}[\log P(\mathcal{D}, \theta)] - \mathbb{E}_{Q}[\log Q(\theta)]$$

Maximizing the ELBO pushes $Q(\theta)$ to be close to the true posterior $P(\theta | \mathcal{D})$ in the KL sense. We will explore this relationship in detail when we cover Variational Inference techniques in Chapter 3.

Model Comparison and Selection: While formal Bayesian model comparison often relies on the marginal likelihood $P(\mathcal{D})$ or Bayes factors, information criteria like AIC (Akaike Information Criterion) and DIC (Deviance Information Criterion) have connections to KL divergence. They provide ways to estimate a model's expected out-of-sample predictive accuracy, implicitly considering a balance between model fit and complexity, concepts related to information gain and distribution divergence.

In summary, entropy provides a way to quantify the uncertainty in our Bayesian models' priors and posteriors. KL divergence provides a fundamental tool for comparing probability distributions, forming the mathematical bedrock for approximation techniques like Variational Inference, which are essential for applying Bayesian methods to complex, high-dimensional problems where exact computation of the posterior is infeasible. Understanding these information-theoretic measures is therefore not just a theoretical exercise; it's foundational for implementing and interpreting many advanced Bayesian techniques.