Training Cycles: Epochs and Batches

When training an autoencoder, or indeed most machine learning models, the learning doesn't happen all at once. Instead, it's an iterative process where the model gradually improves by seeing the data multiple times. Two fundamental terms you'll encounter that describe how this iterative learning is structured are "epochs" and "batches." Understanding these will help you grasp how models like autoencoders are trained effectively.

Imagine you have a large textbook you need to study to learn a new subject. Reading through the entire textbook just once might not be enough to fully understand all the material. You'd likely need to go through it multiple times, perhaps focusing on different aspects each time.

In machine learning, an epoch is one complete pass through the entire training dataset. If your autoencoder goes through 10 epochs, it means it has seen every single sample in your training data 10 times. Each epoch gives the model another chance to learn the underlying patterns in the data and adjust its internal parameters (weights and biases) to better minimize the reconstruction error we discussed earlier. Typically, training a model involves many epochs because complex patterns require repeated exposure for the model to learn them.

Now, consider that large textbook again. Instead of trying to read the entire book in one go (which might be overwhelming and inefficient), you'd probably break it down into chapters or sections. You'd read a chapter, try to understand it, and then move to the next.

This is where batches come into play. Processing an entire large dataset all at once can be computationally very demanding, especially if you have millions of data samples. It might not even fit into your computer's memory. To handle this, the training dataset is divided into smaller, more manageable chunks called batches.

The batch size is the number of training samples included in one batch. For example, if you have a dataset of 1,000 images and you set a batch size of 100, then your dataset will be divided into $1000 / 100 = 10$ batches.

The model processes one batch at a time. After processing each batch, it calculates the loss and then updates its weights using backpropagation. This means the model gets updated multiple times within a single epoch. The number of batches required to complete one epoch is often called iterations or steps per epoch. In our example with 1,000 samples and a batch size of 100, there would be 10 iterations per epoch.

Using batches has a few advantages:

• Memory Efficiency: Smaller batches require less memory, making it possible to train on very large datasets.
• Faster Feedback and Updates: The model's weights are updated after every batch. If you have 10 batches in an epoch, the model gets 10 updates per epoch, rather than just one update if you processed the entire dataset at once. This can lead to faster convergence.
• Potential for Better Generalization: The updates from smaller batches can be a bit "noisier" (as each batch is only a small part of the data), but this characteristic can sometimes help the model avoid getting stuck in poor solutions during optimization and generalize better to new data.

So, the training process generally looks like this:

1. The entire training dataset is often shuffled at the beginning of each epoch (to ensure randomness in batches).
2. For each epoch: a. The dataset is divided into batches according to the chosen batch size. b. For each batch in the current epoch: i. The autoencoder processes the samples in the batch (forward propagation). ii. Calculate the reconstruction loss for this batch. iii. Adjust the autoencoder's weights based on this loss (backpropagation). c. Once all batches are processed, the epoch is complete.
3. This entire cycle of epochs is repeated until the model's performance is satisfactory or a predefined number of epochs is reached.

The following diagram illustrates how a dataset is used across epochs and batches:

This diagram shows the flow from a full dataset, through an epoch that processes data in batches, leading to iterative model refinement over multiple epochs.

Choosing the right number of epochs and the batch size are important decisions when training a model. These are often referred to as hyperparameters, which are settings you configure before training begins.

• Number of Epochs: If you train for too few epochs, the model might not learn enough from the data, a situation known as underfitting. Conversely, training for too many epochs can lead the model to learn the training data too well, including its noise, which makes it perform poorly on new, unseen data. This is called overfitting, a topic we'll touch upon briefly.
• Batch Size: A very small batch size can make training slow (many updates per epoch) and the learning process very noisy. A very large batch size might require too much memory and can sometimes lead to the model settling in less optimal solutions. Common batch sizes range from 32, 64, 128 up to a few thousands, depending on the dataset and hardware.

There are no universal best values; these often depend on the specific dataset and model architecture. Experimentation is usually required to find good values. As the model trains over several epochs, you'll typically observe the reconstruction loss decreasing, indicating that the autoencoder is getting better at its job.

This plot illustrates how the reconstruction loss typically decreases as the model trains over more epochs, showing the autoencoder's improvement.

By understanding epochs and batches, you now have a clearer picture of the mechanics behind training an autoencoder. This iterative refinement, processing data in manageable chunks over multiple passes, is fundamental to how these networks learn to compress and reconstruct information.