Understanding Overfitting and Underfitting

As we discussed in the chapter introduction, building a neural network that performs well on the data it was trained on is only the first step. Our primary objective is generalization, the model's ability to make accurate predictions on new, previously unseen data. Two common obstacles stand in the way of good generalization: underfitting and overfitting. Understanding these phenomena is essential for diagnosing training problems and improving your model's real-world performance.

Underfitting: When the Model Fails to Learn

Underfitting occurs when a model is too simple to capture the underlying structure of the data. It fails to learn the relevant patterns even in the training set, resulting in poor performance not just on new data, but also on the data it was trained on. Think of it as trying to draw a complex shape using only a straight ruler; you simply lack the right tool for the job.

Characteristics of Underfitting:

• High Bias: The model makes strong assumptions about the data (e.g., assuming a linear relationship when it's non-linear) that prevent it from fitting the data well.
• Poor Training Performance: The model exhibits high loss and low accuracy (or other relevant metrics) on the training dataset itself.
• Poor Validation/Test Performance: Consequently, performance on unseen data is also poor, often similar to the training performance.

Common Causes:

• Insufficient Model Complexity: The network might have too few layers or too few neurons per layer to represent the complexity of the task.
• Inadequate Training: The model hasn't been trained for enough epochs to learn the patterns.
• Feature Representation: The input features provided to the model might not contain enough information relevant to the target variable.
• Over-Regularization: While regularization helps prevent overfitting (as we'll see), applying it too aggressively can overly constrain the model, leading to underfitting.

If you observe high error rates on both your training and validation sets, your model is likely underfitting. The solution usually involves increasing model complexity, training longer, engineering better features, or reducing regularization.

Overfitting: When the Model Learns Too Much

Overfitting is the opposite problem. It occurs when a model learns the training data too well, capturing not only the underlying patterns but also the noise and random fluctuations specific to that particular dataset. The model essentially memorizes the training examples instead of learning a generalizable rule. This leads to excellent performance on the training set but poor performance on new, unseen data. Imagine a student who memorizes the answers to specific practice questions but hasn't grasped the underlying concepts needed to solve new problems.

Characteristics of Overfitting:

• High Variance: The model is highly sensitive to the specific training data. Small changes in the training set could lead to a significantly different model.
• Excellent Training Performance: The model achieves very low loss and high accuracy on the training dataset.
• Poor Validation/Test Performance: Performance on unseen data is significantly worse than on the training data. There's a noticeable gap between training metrics and validation metrics.

Common Causes:

• Excessive Model Complexity: The network has too many layers or neurons, giving it the capacity to memorize noise.
• Insufficient Training Data: With limited data, it's easier for a complex model to find spurious correlations and memorize individual examples.
• Training for Too Long: As training progresses, the model might initially learn general patterns but eventually start fitting the noise in the training data.

Overfitting is a very common issue in neural network training. You can spot it when your training loss continues to decrease while your validation loss starts to plateau or even increase. This divergence indicates that the model is no longer learning generalizable patterns but is instead specializing in the training set specifics.

The Bias-Variance Tradeoff

Underfitting and overfitting represent two extremes of the bias-variance tradeoff.

• Bias is the error introduced by approximating a real-world problem, which may be complex, by a too-simple model. High bias leads to underfitting.
• Variance is the amount by which the model's learned function would change if we trained it on a different training dataset. High variance means the model is too sensitive to the training data's specifics, leading to overfitting.

Ideally, we want a model with low bias and low variance. However, decreasing one often tends to increase the other. Very simple models have high bias and low variance (they are consistently wrong in the same way). Very complex models can have low bias but high variance (they fit the current data perfectly but generalize poorly). The goal of training and techniques like regularization is to find a sweet spot that balances this tradeoff, achieving good performance on unseen data.

Visualizing Model Fit During Training

A standard practice to monitor for underfitting and overfitting is to plot the model's loss (and/or accuracy) on both the training set and a separate validation set over the course of training epochs.

Comparison of training loss against different validation loss curves. Underfitting shows high loss for both. A good fit shows both converging at a low value. Overfitting shows validation loss increasing while training loss continues to decrease.

Observing these curves is fundamental:

1. Underfitting: Both training and validation loss are high and may plateau quickly without reaching a low value.
2. Good Fit: Both training and validation loss decrease and converge at a low value. The gap between them is small.
3. Overfitting: Training loss continues to decrease, while validation loss reaches a minimum and then starts to increase. The gap between training and validation loss widens significantly.

By monitoring these metrics, you can diagnose whether your model is underfitting, overfitting, or achieving a good balance. The following sections in this chapter will detail specific techniques like regularization and early stopping to combat overfitting and improve your model's ability to generalize.