Flow-Based Models for TTS

Moving from autoregressive models like Tacotron and non-autoregressive approaches like FastSpeech, flow-based generative models offer another powerful framework for acoustic modeling in Text-to-Speech (TTS). Instead of generating outputs sequentially or relying solely on auxiliary losses for parallel generation, flow-based models learn an exact, invertible transformation from a simple base distribution (like a Gaussian) to the complex data distribution of mel-spectrograms, conditioned on the input text.

The Essence of Normalizing Flows

At their core, normalizing flows construct a complex distribution by applying a sequence of invertible and differentiable transformations $f$ to samples $z$ drawn from a simple base distribution $p_Z(z)$, typically a standard multivariate Gaussian. The goal is to model the target data distribution $p_X(x)$, where $x$ represents our data (e.g., mel-spectrograms).

The transformation $x = f(z)$ allows us to compute the exact likelihood of a data point $x$ using the change of variables formula from probability theory. If $z = f^{-1}(x)$, the probability density function of $x$ is given by:

$$p_X(x) = p_Z(f^{-1}(x)) \left| \det\left(\frac{\partial f^{-1}(x)}{\partial x}\right) \right|$$

Equivalently, using the inverse function theorem:

$$p_X(x) = p_Z(z) \left| \det\left(\frac{\partial z}{\partial x}\right) \right| = p_Z(z) \left| \det\left(\frac{\partial f(z)}{\partial z}\right) \right|^{-1}$$

Here, $\frac{\partial f^{-1}(x)}{\partial x}$ (or $\frac{\partial f(z)}{\partial z}$) is the Jacobian matrix of the transformation (or its inverse), and $\det(\cdot)$ denotes its determinant.

The critical insight is that if we design the transformation $f$ such that:

1. It is easily invertible (i.e., calculating $z = f^{-1}(x)$ is efficient).
2. The determinant of its Jacobian is easy to compute.

Then, we can efficiently calculate the exact likelihood $p_X(x)$ for any given data point $x$. Training is typically done by maximizing the log-likelihood of the training data:

$$\log p_X(x) = \log p_Z(z) - \log \left| \det\left(\frac{\partial f(z)}{\partial z}\right) \right|$$

where $z = f^{-1}(x)$.

Applying Flows to Acoustic Modeling

In the context of TTS, we want to model the conditional distribution of mel-spectrograms $x$ given some linguistic features or text embeddings $c$. So, the flow $f$ becomes conditional: $x = f(z; c)$. The base distribution $p_Z(z)$ remains a simple prior, like $\mathcal{N}(0, I)$.

• Training: During training, we take a ground truth mel-spectrogram $x$ and its corresponding condition $c$. We compute $z = f^{-1}(x; c)$ and maximize the conditional log-likelihood $\log p_X(x|c)$ using the change of variables formula. This directly optimizes the model to assign high probability to realistic mappings from text to speech features.
• Inference (Synthesis): To synthesize a mel-spectrogram for a new condition $c$, we first sample a latent variable $z$ from the prior $p_Z(z)$. Then, we compute the corresponding mel-spectrogram $x$ using the forward transformation: $x = f(z; c)$. Since the forward pass $f$ is typically designed to be efficient and parallel (often involving operations like convolutions and affine transformations), synthesis can be very fast, similar to non-autoregressive models.

Overview of a conditional flow-based model for TTS. Inference transforms noise into data, while training transforms data into noise and maximizes likelihood. Conditioning guides the transformation.

Prominent Flow-Based TTS Architectures

Several specific architectures have successfully applied normalizing flows to TTS acoustic modeling:

• Glow-TTS: Inspired by the Glow model for image generation, Glow-TTS adapts the flow architecture (affine coupling layers, invertible 1x1 convolutions) for mel-spectrogram generation. It uses an alignment mechanism (related to attention, but monotonic during inference) learned variationally during training to handle the variable-length alignment between text input and speech output. This alignment is a significant part of making flows work for sequence-to-sequence tasks like TTS.

• Flowtron: This model integrates elements from Tacotron (like location-sensitive attention for alignment) directly into a flow-based decoder architecture (specifically, using affine coupling layers similar to Glow). Flowtron explicitly maps variations in the latent code $z$ to variations in speech prosody and style, offering control over the synthesized output. By manipulating parts of $z$, one can influence pitch, rhythm, and other expressive aspects of the generated speech.

These models typically consist of:

1. A Text Encoder (e.g., Transformer or LSTM layers) to convert input text/phonemes into hidden representations.
2. An Alignment Mechanism (e.g., attention-based or duration predictor) to align text representations with the mel-spectrogram frame rate.
3. A Conditional Flow Decoder that transforms samples $z$ from the prior distribution into mel-spectrogram frames $x$, conditioned on the aligned text representations. This decoder is built from layers designed to be invertible with tractable Jacobians.

Advantages and Considerations

Flow-based models bring unique properties to TTS:

Advantages:

• Exact Log-Likelihood Training: Training is based on maximizing the exact log-likelihood of the data, providing a stable and well-understood optimization objective, unlike the adversarial training of GANs.
• Fast Parallel Synthesis: Like other non-autoregressive methods, inference is parallel and fast once the latent variable $z$ is sampled. The entire mel-spectrogram can often be generated in a single forward pass through the flow network.
• Latent Space Control: The explicit mapping from a structured latent space $Z$ to the data space $X$ offers potential for controllable synthesis. Manipulating $z$ can influence attributes like speaker identity, prosody, or emotion, particularly in models like Flowtron designed for this purpose.
• High-Quality Output: Flow-based models have demonstrated the ability to generate high-fidelity mel-spectrograms, leading to natural-sounding synthesized speech when combined with a good neural vocoder.

Considerations:

• Model Complexity and Size: Flow networks often need to be deep to model complex data distributions effectively, which can lead to large model sizes and significant computational requirements during training.
• Architectural Constraints: The need for invertibility and tractable Jacobian determinants restricts the design choices for the network layers (e.g., affine coupling, invertible convolutions). This might limit the representational power compared to less constrained architectures.
• Alignment Handling: Effectively aligning the input text sequence with the output mel-spectrogram sequence within the flow framework requires careful design, often involving separate attention mechanisms or duration predictors, adding complexity.

Compared to autoregressive models, flows offer much faster synthesis but might require more complex architectures for high quality. Compared to other non-autoregressive models (like FastSpeech), flows provide the benefit of exact likelihood training but can be computationally heavier. They generally offer more stable training than GANs while achieving competitive quality.

Flow-based models represent a compelling direction in generative modeling for speech synthesis, balancing parallel inference speed with principled likelihood-based training and offering avenues for expressive control. Understanding their mechanism provides valuable insight into the diverse techniques available for building advanced TTS systems.