Evaluation Metrics for GANs

Evaluating the output of a Generative Adversarial Network isn't as straightforward as calculating accuracy or loss in supervised learning. Since the generator's goal is to produce realistic and diverse samples mimicking a target distribution, we need metrics that assess both the quality (fidelity) of individual generated images and the variety (diversity) of the entire generated set. Simply looking at samples can be subjective and doesn't scale well, while the generator and discriminator losses during training often don't correlate strongly with the perceived quality of the final output. Therefore, specialized quantitative metrics are necessary to provide objective comparisons between different GAN models or training checkpoints.

The core challenge lies in comparing probability distributions: the distribution of real data, $p_{data}$, and the distribution implicitly defined by the generator, $p_g$. We want to measure how "close" $p_g$ is to $p_{data}$. Two prominent metrics have emerged as standards in the field: the Inception Score (IS) and the Fréchet Inception Distance (FID).

Inception Score (IS)

The Inception Score aims to capture both fidelity and diversity using a pre-trained image classification model, typically Inception V3 trained on ImageNet. The intuition is twofold:

1. Fidelity: Images generated by a good GAN should be clearly recognizable and contain meaningful objects. When passed through the Inception classifier, the conditional probability distribution $p(y|x)$ (the probability of image $x$ belonging to class $y$) should have low entropy. This means the classifier is confident about assigning the image to a specific class.
2. Diversity: The generator should produce images covering a wide variety of classes present in the dataset. Therefore, the marginal probability distribution $p(y) = \int p(y|x) p_g(x) dx$ (the overall distribution of classes across all generated images) should have high entropy. This indicates that the generator isn't stuck producing images of only a few classes (mode collapse).

These two ideas are combined using the Kullback-Leibler (KL) divergence between the conditional and marginal distributions, averaged over all generated samples $x \sim p_g$:

$$IS = \exp(\mathbb{E}_{x \sim p_g} [ D_{KL}(p(y|x) || p(y)) ])$$

A higher Inception Score is generally considered better. However, IS has limitations. It primarily measures whether generated images look like any of the ImageNet classes, not necessarily the specific classes in the target dataset if it's different from ImageNet. It also doesn't directly compare the generated images to real images from the target distribution and can be susceptible to adversarial examples within classes. Furthermore, it has been shown that IS doesn't always correlate well with human perception of image quality, especially regarding diversity within a class.

Fréchet Inception Distance (FID)

The Fréchet Inception Distance has become a more popular and widely adopted metric because it addresses some of the shortcomings of the IS. FID compares the statistics of generated images directly to the statistics of real images from the target dataset. It operates in the feature space of a pre-trained Inception V3 model.

Here's how FID is calculated:

1. Feature Extraction: Select a specific layer from the pre-trained Inception V3 network (commonly the final average pooling layer before the classification head). Pass a large number of real images ($x_r$) and generated images ($x_g$) through the network up to this layer to obtain feature vectors for each image.

2. Distribution Modeling: Assume the extracted feature vectors for the real images and the generated images follow multivariate Gaussian distributions. Calculate the mean vector ($\mu_r$, $\mu_g$) and the covariance matrix ($\Sigma_r$, $\Sigma_g$) for the feature vectors of the real and generated sets, respectively.

3. Distance Calculation: Compute the Fréchet distance (also known as the Wasserstein-2 distance for Gaussian distributions) between the two modeled distributions ($N(\mu_r, \Sigma_r)$ and $N(\mu_g, \Sigma_g)$). The formula is:

   $$FID = ||\mu_r - \mu_g||^2_2 + \text{Tr}(\Sigma_r + \Sigma_g - 2(\Sigma_r \Sigma_g)^{1/2})$$

   Here, $||\cdot||^2_2$ denotes the squared Euclidean distance between the mean vectors, $\text{Tr}$ is the trace of a matrix (sum of diagonal elements), and $(\Sigma_r \Sigma_g)^{1/2}$ is the matrix square root of the product of the covariance matrices.

A lower FID score indicates that the statistics of the generated image features are more similar to the statistics of the real image features, implying that the generated distribution $p_g$ is closer to the real data distribution $p_{data}$. Lower FID generally corresponds to better image quality and diversity.

Feature vectors extracted from real and generated images using an Inception model are modeled as Gaussian distributions. FID measures the distance between these distributions, considering both their means ($\mu$) and covariances ($\Sigma$). Lower distance implies greater similarity.

FID is more sensitive to noise, sensitive to mode collapse (as it affects both mean and covariance), and correlates better with human judgment of image quality than IS. However, it requires a significant number of samples (typically 10,000 to 50,000) from both real and generated distributions to reliably estimate the means and covariance matrices. Its computation is also more intensive than IS.

Other Metrics and Considerations

While IS and FID are the most common, other metrics exist:

• Precision and Recall for Distributions: These metrics adapt concepts from information retrieval to GAN evaluation. Precision measures the fraction of generated samples that are considered realistic (fidelity), while Recall measures the fraction of real samples that the generator can produce (diversity).
• Perceptual Path Length (PPL): Used primarily for style-based generators (like StyleGAN), PPL measures the smoothness of the generator's latent space. Small changes in the latent input vector should ideally lead to small, perceptually smooth changes in the output image.

Practical Advice:

• Standardization: When reporting FID or IS, use standardized implementations and the same pre-trained Inception model (usually V3) to ensure fair comparisons across different studies or experiments.
• Sample Size: Be mindful of the sample size required for stable FID calculations. Using too few samples can lead to unreliable scores.
• Complementary Evaluation: Quantitative metrics provide valuable objective data, but they don't capture everything. Always supplement metric scores with qualitative visual inspection of generated samples to assess aspects like coherence, fine details, and potential artifacts that metrics might miss.

In summary, evaluating GANs requires moving past simple loss functions. Metrics like IS and particularly FID provide quantitative ways to assess the quality and diversity of generated images by comparing their distributions (often in a feature space) to those of real images. Understanding how these metrics work and their limitations is essential for effectively developing and comparing generative models.