Parameter Learning in Bayesian Networks

Once the structure $G$ of a Bayesian Network (BN) is defined, either through domain expertise or structure learning algorithms, the next significant step is to quantify the probabilistic relationships between connected nodes. This involves learning the parameters of the model, which typically define the Conditional Probability Distributions (CPDs) associated with each variable given its parents in the graph. In contrast to simply finding a single best estimate for these parameters (like Maximum Likelihood Estimation), the Bayesian approach treats the parameters themselves as random variables and aims to compute their posterior distribution given the observed data.

Parameters as Random Variables

In a Bayesian Network, the parameters, collectively denoted as $\theta$, represent the conditional probabilities that define the model. For a network with variables $X_1, \dots, X_n$, the parameters $\theta$ consist of sets $\theta_{i | pa(X_i)}$ for each variable $X_i$, specifying the probability distribution $P(X_i | pa(X_i), \theta_{i | pa(X_i)})$ for each configuration of its parents $pa(X_i)$.

The core idea of Bayesian parameter learning is to start with a prior distribution $P(\theta | G)$ over these parameters, reflecting our beliefs before observing any data $D$. We then use Bayes' theorem to update these beliefs based on the data, resulting in a posterior distribution $P(\theta | D, G)$:

$$P(\theta | D, G) = \frac{P(D | \theta, G) P(\theta | G)}{P(D | G)}$$

Here, $P(D | \theta, G)$ is the likelihood of observing the data $D$ given a specific set of parameters $\theta$ and the graph structure $G$. The term $P(D | G)$ is the marginal likelihood or evidence, often intractable to compute directly, but serves as a normalizing constant.

A fundamental property of Bayesian Networks simplifies this process considerably. Given a complete dataset (no missing values), the likelihood function decomposes according to the graph structure:

$$P(D | \theta, G) = \prod_{i=1}^{n} \prod_{j=1}^{q_i} P(\theta_{ij} | D)$$

where $\theta_{ij}$ represents the parameters for variable $X_i$ when its parents $pa(X_i)$ are in their $j$-th configuration (out of $q_i$ possible configurations). Furthermore, if we assume parameter independence in the prior, meaning $P(\theta|G) = \prod_i \prod_j P(\theta_{ij})$, then the posterior also decomposes:

$$P(\theta | D, G) = \prod_{i=1}^{n} \prod_{j=1}^{q_i} P(\theta_{ij} | D)$$

This decomposition implies we can learn the parameters for each local CPD independently, breaking down a potentially massive learning problem into smaller, manageable ones.

Learning Parameters for Discrete Variables

Let's consider the common case where all variables in the BN are discrete. Each variable $X_i$ can take one of $r_i$ states. The parameters $\theta_{ijk}$ represent the probability that variable $X_i$ takes its $k$-th state, given that its parents $pa(X_i)$ are in their $j$-th configuration:

$$\theta_{ijk} = P(X_i = k | pa(X_i) = j, \theta_{ij})$$

where $\sum_{k=1}^{r_i} \theta_{ijk} = 1$ for all $i, j$.

The likelihood contribution for a specific CPD $P(X_i | pa(X_i)=j)$ given the data $D$ follows a multinomial distribution based on the counts $N_{ijk}$ - the number of times in the dataset $D$ where $X_i=k$ and $pa(X_i)=j$.

For the prior distribution over the parameters $\theta_{ij} = (\theta_{ij1}, \dots, \theta_{ij r_i})$, a convenient and widely used choice is the Dirichlet distribution. The Dirichlet distribution $Dir(\alpha_{ij1}, \dots, \alpha_{ij r_i})$ is defined over the probability simplex (where components are non-negative and sum to 1). Its probability density function is:

$$P(\theta_{ij} | \alpha_{ij}) = \frac{\Gamma(\sum_{k=1}^{r_i} \alpha_{ijk})}{\prod_{k=1}^{r_i} \Gamma(\alpha_{ijk})} \prod_{k=1}^{r_i} \theta_{ijk}^{\alpha_{ijk} - 1}$$

The hyperparameters $\alpha_{ijk} > 0$ can be interpreted as pseudo-counts or prior counts, reflecting prior belief about the occurrences of each state $k$ for $X_i$ given parent configuration $j$. The sum $\alpha_{ij} = \sum_k \alpha_{ijk}$ represents the strength of this prior belief (equivalent sample size).

The Dirichlet prior is conjugate to the multinomial likelihood. This means that if the prior is Dirichlet and the likelihood is multinomial, the posterior distribution is also Dirichlet. Specifically, the posterior for $\theta_{ij}$ is:

$$P(\theta_{ij} | D, G) = Dir(\alpha_{ij1} + N_{ij1}, \dots, \alpha_{ij r_i} + N_{ij r_i})$$

This elegant result means the posterior hyperparameters are simply the prior hyperparameters updated by the corresponding counts observed in the data. Learning parameters amounts to counting occurrences in the data and adding them to the prior pseudo-counts.

Parameter Estimation: Exploring the Posterior

While the Bayesian approach yields a full posterior distribution $P(\theta | D, G)$, often we need a single point estimate for the parameters, for example, to populate a CPT for inference. Common choices include:

1. Posterior Mean (Bayesian Estimate): This is the expected value of the parameters under the posterior distribution. For a Dirichlet posterior $Dir(\alpha'_{ij1}, \dots, \alpha'_{ij r_i})$ where $\alpha'_{ijk} = \alpha_{ijk} + N_{ijk}$, the posterior mean is:

   $$\mathbb{E}[\theta_{ijk} | D, G] = \frac{\alpha'_{ijk}}{\sum_{l=1}^{r_i} \alpha'_{ijl}} = \frac{\alpha_{ijk} + N_{ijk}}{\sum_{l=1}^{r_i} (\alpha_{ijl} + N_{ijl})}$$

   Notice how this estimate smooths the empirical frequencies $N_{ijk} / \sum_l N_{ijl}$ (which is the Maximum Likelihood Estimate, MLE) using the prior pseudo-counts $\alpha_{ijk}$. This helps prevent zero probabilities for events not seen in the data, especially important with smaller datasets. A common non-informative prior choice is the Laplace smoothing prior where $\alpha_{ijk} = 1$ for all $i, j, k$.

2. Maximum A Posteriori (MAP) Estimate: This finds the parameter values that maximize the posterior probability density. For the Dirichlet posterior, the MAP estimate is:

   $$\theta_{ijk}^{MAP} = \frac{\alpha'_{ijk} - 1}{\sum_{l=1}^{r_i} (\alpha'_{ijl} - 1)} = \frac{\alpha_{ijk} + N_{ijk} - 1}{\sum_{l=1}^{r_i} (\alpha_{ijl} + N_{ijl} - 1)}$$

   provided all $\alpha'_{ijk} > 1$. If any $\alpha'_{ijk} \le 1$, the mode is at the boundary of the simplex.

The choice between posterior mean and MAP depends on the application, but the posterior mean is often preferred for its averaging nature and connection to minimizing squared error loss.

Example: Prior to Posterior Update

Consider a simple parameter $\theta$ representing the probability of a coin landing heads. Let's assume a Beta prior (which is a Dirichlet distribution with K=2 states: heads, tails), say $Beta(\alpha_H=2, \alpha_T=2)$. This reflects a prior belief centered at 0.5 but with some uncertainty.

Suppose we flip the coin 10 times and observe $N_H=7$ heads and $N_T=3$ tails. The posterior distribution becomes:

$$P(\theta | D) = Beta(\alpha_H + N_H, \alpha_T + N_T) = Beta(2 + 7, 2 + 3) = Beta(9, 5)$$

The posterior mean estimate for $\theta$ (probability of heads) is:

$$\mathbb{E}[\theta | D] = \frac{9}{9 + 5} = \frac{9}{14} \approx 0.643$$

The MLE would be $7/10 = 0.7$. The Bayesian estimate is pulled slightly towards the prior mean (0.5).

We can visualize this update:

The plot shows how the probability density shifts and becomes narrower (more confident) after observing data. The prior (blue) is centered at 0.5, while the posterior (pink) peaks near 0.64, reflecting the observed data (7 heads, 3 tails).

This principle extends directly to the multinomial parameters in CPTs of Bayesian Networks, where each row of a CPT corresponding to a specific parent configuration is updated independently using its relevant prior counts and observed data counts.

Challenges and Extensions

While elegant for complete data and conjugate priors, Bayesian parameter learning faces challenges:

• Missing Data/Latent Variables: The simple counting approach breaks down. Iterative methods like Expectation-Maximization (EM) or sampling techniques (Gibbs sampling, covered in MCMC chapters) are needed to handle the uncertainty introduced by missing values or hidden variables.
• Non-conjugate Priors: If the chosen prior is not conjugate to the likelihood, the posterior distribution may not have a closed-form solution, again necessitating approximate inference methods (MCMC, VI).
• Continuous Variables: For continuous variables, different likelihoods (e.g., Gaussian) and corresponding conjugate priors (e.g., Normal-Wishart for Gaussian parameters) are used, involving similar principles but different mathematical forms.

In summary, Bayesian parameter learning provides a principled way to estimate the parameters of a Bayesian Network by combining prior knowledge with observed data. The use of conjugate priors like the Dirichlet distribution greatly simplifies calculations for discrete networks with complete data, allowing for efficient updates and providing a full posterior distribution that captures parameter uncertainty. This uncertainty is a distinct advantage over point estimates like MLE and can be propagated through subsequent inference tasks.