Evaluating Quantum Generative Models

Once you've trained a Quantum Circuit Born Machine (QCBM) or a Quantum Generative Adversarial Network (QGAN), how do you determine if it's actually learned the target data distribution effectively? Evaluating generative models is a complex task even classically, and quantum approaches introduce unique considerations. This section covers metrics and methodologies for assessing the performance of quantum generative models.

Unlike supervised learning where accuracy provides a clear benchmark, evaluating generative models involves assessing both the quality of individual samples and the similarity of the overall distribution $p_{\text{model}}(x)$ generated by the model to the true data distribution $p_{\text{data}}(x)$. Directly calculating the likelihood $p_{\text{model}}(x)$ for models like QCBMs and QGANs is often intractable, similar to classical implicit generative models like GANs. Therefore, we rely heavily on sample-based evaluation techniques.

Comparing Probability Distributions

Several statistical distances or divergences can quantify the difference between $p_{\text{model}}$ and $p_{\text{data}}$. These typically require access to samples from both distributions.

Kullback-Leibler (KL) Divergence

A fundamental measure from information theory is the KL divergence: $D_{KL}(p_{\text{data}} || p_{\text{model}}) = \sum_x p_{\text{data}}(x) \log \frac{p_{\text{data}}(x)}{p_{\text{model}}(x)}$ or for continuous variables: $D_{KL}(p_{\text{data}} || p_{\text{model}}) = \int p_{\text{data}}(x) \log \frac{p_{\text{data}}(x)}{p_{\text{model}}(x)} dx$

$D_{KL} \ge 0$, with $D_{KL} = 0$ if and only if $p_{\text{data}} = p_{\text{model}}$. However, KL divergence has drawbacks:

• It's asymmetric: $D_{KL}(p_{\text{data}} || p_{\text{model}}) \neq D_{KL}(p_{\text{model}} || p_{\text{data}})$.
• It can be infinite if $p_{\text{model}}(x) = 0$ for some $x$ where $p_{\text{data}}(x) > 0$.
• Estimating it accurately from samples is difficult, especially in high dimensions, often requiring binning or density estimation techniques which introduce their own biases. Evaluating $p_{\text{model}}(x)$ itself might be hard.

While minimizing KL divergence is related to maximizing likelihood, its direct calculation or estimation for quantum generative models is often impractical.

Jensen-Shannon (JS) Divergence

The JS divergence is a symmetrized and bounded version of the KL divergence: $D_{JS}(p_{\text{data}} || p_{\text{model}}) = \frac{1}{2} D_{KL}(p_{\text{data}} || p_{\text{avg}}) + \frac{1}{2} D_{KL}(p_{\text{model}} || p_{\text{avg}})$ where $p_{\text{avg}} = \frac{1}{2} (p_{\text{data}} + p_{\text{model}})$.

JS divergence ranges between 0 and $\log 2$ (or 1 if using natural log), making it more stable than KL divergence. It's zero if and only if the distributions are identical. While symmetric and bounded, estimating JS divergence from samples still faces challenges in high dimensions, although it's often preferred over KL divergence in GAN literature, partly because it forms the basis of the original GAN objective function.

Maximum Mean Discrepancy (MMD)

MMD is a non-parametric metric based on the idea that two distributions are identical if and only if all their moments match. It measures the distance between the mean embeddings of the distributions in a Reproducing Kernel Hilbert Space (RKHS) $\mathcal{H}$ defined by a kernel function $k(x, x')$.

$MMD^2(p_{\text{data}}, p_{\text{model}}) = || \mathbb{E}_{x \sim p_{\text{data}}}[\phi(x)] - \mathbb{E}_{x' \sim p_{\text{model}}}[\phi(x')] ||^2_{\mathcal{H}}$ where $\phi(x) = k(x, \cdot)$ is the feature map associated with the kernel $k$.

Using the kernel trick, MMD can be estimated from samples $\{x_i\}_{i=1}^N \sim p_{\text{data}}$ and $\{x'_j\}_{j=1}^M \sim p_{\text{model}}$: $MMD^2_u = \frac{1}{N(N-1)} \sum_{i \neq j} k(x_i, x_j) + \frac{1}{M(M-1)} \sum_{i \neq j} k(x'_i, x'_j) - \frac{2}{NM} \sum_{i=1}^N \sum_{j=1}^M k(x_i, x'_j)$

Common kernel choices include the Gaussian (RBF) kernel. The performance of MMD depends significantly on the chosen kernel and its parameters (e.g., the bandwidth $\sigma$ for the Gaussian kernel). MMD is often computationally cheaper to estimate than KL or JS divergence and doesn't require explicit density estimation. It's frequently used for evaluating GANs and can even be used as a training objective (e.g., in MMD-GANs).

Assessing Sample Quality

Beyond distributional similarity, we often need to assess the quality or realism of individual samples.

• Qualitative Evaluation: For data types like images, visual inspection by humans remains a common, albeit subjective, method. Do the generated samples "look like" the real data?
• Downstream Task Performance: A more objective approach is to evaluate the utility of the generated data. Train a separate model (e.g., a classifier) on the generated samples and test its performance on a real test set. Compare this performance to a model trained solely on real data. If the model trained on synthetic data performs well, it suggests the generated samples capture relevant features.
• Domain-Specific Metrics: Depending on the data domain (e.g., finance, chemistry), specific metrics might exist to evaluate the validity or properties of generated samples (e.g., chemical validity of generated molecules).

Quantum-Specific Evaluation Challenges

Evaluating quantum generative models involves additional hurdles stemming from the nature of quantum computation:

1. Sampling Cost and Noise: Generating samples $x \sim p_{\text{model}}(x)$ requires executing the quantum circuit (QCBM generator or QGAN generator) and performing measurements. On Noisy Intermediate-Scale Quantum (NISQ) hardware, this process is susceptible to noise (decoherence, gate errors, readout errors), which distorts the resulting distribution $p_{\text{model}}$. Obtaining a large number of clean samples for accurate metric estimation can be time-consuming and resource-intensive. Error mitigation techniques can help but add overhead.
2. Metric Estimation: Estimating statistical divergences like KL, JS, or MMD requires a sufficient number of samples. The challenges of quantum sampling compound the usual difficulties of high-dimensional distribution comparison.
3. Benchmarking: Fairly comparing a quantum generative model to a classical one requires careful consideration of computational resources (qubits, circuit depth, shots vs. classical compute time, memory) and ensuring identical datasets and evaluation protocols.

Comparing empirical probability distributions obtained from real data samples and samples generated by a quantum model (e.g., QCBM or QGAN). Divergence metrics quantify the difference between these histograms.

Best Practices for Evaluation

Given these challenges, an evaluation strategy should include:

• Multiple Metrics: Use a combination of distribution similarity metrics (e.g., MMD, estimated JS divergence) and sample quality assessments (e.g., visual inspection, downstream task performance). No single number tells the whole story.
• Classical Baselines: Always compare against relevant classical generative models (e.g., GANs, VAEs) trained and evaluated on the same dataset and using the same metrics.
• Sample Size Awareness: Acknowledge the limitations imposed by available samples. Report the number of samples used for evaluation and, if possible, analyze metric stability with respect to sample size.
• Resource Reporting: Document the quantum resources (number of qubits, circuit depth, number of shots, error mitigation used) and classical computational resources involved.

Evaluating quantum generative models is an active area of research. As hardware improves and theoretical understanding deepens, evaluation techniques will continue to evolve, aiming for more reliable and efficient assessment of these powerful new models.