Unpaired Image-to-Image Translation (CycleGAN)

Many real-world image generation tasks involve translating an image from a source domain $X$ to a target domain $Y$. For instance, converting photos into paintings, changing seasons in images, or transforming horses into zebras. When paired training data exists (e.g., pairs of architectural sketches and corresponding photos), models like pix2pix can learn this mapping effectively using supervised techniques. However, obtaining such paired datasets is often expensive, difficult, or simply impossible. How can we learn to translate between domains $X$ and $Y$ when we only have a collection of images from $X$ and a separate, unrelated collection of images from $Y$?

CycleGAN provides an elegant solution to this problem of unpaired image-to-image translation. It learns the mapping without requiring any direct correspondence between individual images in the two domains.

The Core Idea: Cycle Consistency

Imagine you want to translate photos of horses (domain $X$) into images resembling zebras (domain $Y$). CycleGAN employs two generator networks:

• $G$: Learns the mapping $G: X \to Y$ (e.g., horse to zebra).
• $F$: Learns the inverse mapping $F: Y \to X$ (e.g., zebra to horse).

It also uses two discriminator networks:

• $D_Y$: Aims to distinguish between real images from domain $Y$ (real zebras) and generated images $G(x)$ (fake zebras produced from horse photos $x$).
• $D_X$: Aims to distinguish between real images from domain $X$ (real horses) and generated images $F(y)$ (fake horses produced from zebra photos $y$).

The standard adversarial losses encourage $G$ to produce outputs $G(x)$ that look like they belong to domain $Y$, and $F$ to produce outputs $F(y)$ that look like they belong to domain $X$.

However, adversarial loss alone is insufficient. The generators could learn to map all inputs from one domain to a single, realistic-looking image in the other domain, satisfying the discriminators but failing to capture the desired input-output relationship. For example, $G$ might learn to generate a plausible zebra image for any input horse photo.

To address this, CycleGAN introduces the cycle consistency loss. The intuition is simple: if you translate an image from domain $X$ to domain $Y$ and then translate it back to domain $X$, you should recover something very close to the original image. The same logic applies when starting from domain $Y$.

Mathematically, this constraint is enforced using a loss function, typically L1 distance, as specified in the chapter introduction:

$$L_{cyc}(G, F) = \mathbb{E}_{x \sim p_{data}(x)}[||F(G(x)) - x||_1] + \mathbb{E}_{y \sim p_{data}(y)}[||G(F(y)) - y||_1]$$

This loss penalizes deviations between the original image ($x$ or $y$) and its reconstructed version after a forward and backward translation ($F(G(x))$ or $G(F(y))$). The L1 norm ($|| \cdot ||_1$) is often preferred over the L2 norm ($|| \cdot ||_2^2$) as it tends to produce less blurry results in image generation tasks.

The Full Objective Function

The complete objective function for CycleGAN combines the adversarial losses for both mapping directions with the cycle consistency loss:

$$L_{total}(G, F, D_X, D_Y) = L_{GAN}(G, D_Y, X, Y) + L_{GAN}(F, D_X, Y, X) + \lambda L_{cyc}(G, F)$$

Here:

• $L_{GAN}(G, D_Y, X, Y)$ is the adversarial loss for the $X \to Y$ mapping (Generator $G$ trying to fool Discriminator $D_Y$).
• $L_{GAN}(F, D_X, Y, X)$ is the adversarial loss for the $Y \to X$ mapping (Generator $F$ trying to fool Discriminator $D_X$).
• $L_{cyc}(G, F)$ is the cycle consistency loss defined previously.
• $\lambda$ is a hyperparameter that controls the relative importance of the adversarial losses versus the cycle consistency loss. A typical value used in the original paper is $\lambda = 10$.

The goal is to find generators $G^*$ and $F^*$ that minimize the cycle consistency loss while maximizing the adversarial loss against their respective discriminators, which are simultaneously trained to distinguish real from generated images:

$$G^*, F^* = \arg \min_{G, F} \max_{D_X, D_Y} L_{total}(G, F, D_X, D_Y)$$

Architecture and Training Details

While the cycle consistency loss is the main contribution, CycleGAN's implementation often incorporates architectures and techniques known to improve GAN stability and quality:

• Generators: Typically use convolutional architectures with residual blocks, inspired by models successful in style transfer. They often consist of downsampling layers, several residual blocks, and then upsampling layers.
• Discriminators: Frequently employ PatchGAN classifiers. Instead of outputting a single probability for the entire image being real or fake, PatchGAN outputs a grid of probabilities, where each value corresponds to the "realness" of overlapping image patches. This encourages sharper details across the image.
• Loss Function: The original paper used a least-squares GAN (LSGAN) loss instead of the standard minimax log loss for the adversarial terms ($L_{GAN}$), finding it led to more stable training.
• Image Buffer: To prevent the discriminator from adapting too quickly to the latest generated images, CycleGAN uses a buffer storing a history of previously generated images. The discriminator is trained using a mix of real images and images sampled from this buffer.

Diagram illustrating the CycleGAN framework. It involves two generators ($G$, $F$), two discriminators ($D_X$, $D_Y$), adversarial losses to ensure generated images match the target domain distributions, and cycle consistency losses to enforce structural similarity between input and reconstructed images.

Strengths and Limitations

CycleGAN's primary strength is its ability to perform image translation without paired data, opening up many applications previously infeasible. It has shown impressive results in tasks like style transfer (photo to Monet, Van Gogh, etc.), object transfiguration (horse to zebra, apple to orange), and domain adaptation (synthetic to real images).

However, it also has limitations:

• Geometric Changes: CycleGAN struggles with tasks requiring significant geometric transformations (e.g., translating between images of cats and dogs, which have different typical poses and shapes). The cycle consistency loss implicitly assumes that the underlying structure can be largely preserved during translation and reconstruction.
• Content Preservation: While effective at changing texture and style, it might sometimes fail to preserve the exact content details of the input image if the adversarial loss dominates or if the task is inherently ambiguous.
• Mode Collapse: Like other GANs, CycleGAN can suffer from mode collapse, although the cycle consistency loss often helps mitigate this compared to simpler adversarial setups.
• Resource Intensive: Training involves four networks (two generators, two discriminators), making it computationally more demanding than standard GANs or paired translation models.

Despite these limitations, CycleGAN represents a significant step forward in generative modeling, demonstrating how clever loss function design can overcome data limitations like the absence of paired examples, enabling a wide range of creative and practical image manipulation tasks.