Supervised Learning with Neural Networks

Now that we understand the basic building blocks of neural networks, artificial neurons arranged in layers, let's discuss how these networks actually learn from data. The most common approach for training neural networks, especially when starting out, is supervised learning.

In supervised learning, the goal is to train a model to map input data to known output labels. Think of it like a student learning with a teacher who provides the correct answers. We provide the neural network with a large dataset consisting of input examples and their corresponding correct outputs (labels or targets). The network's task is to learn the underlying patterns or relationships that connect the inputs to the outputs.

The Learning Objective

Mathematically, we want the network to learn an approximation of a function $f$ that maps inputs $X$ to outputs $Y$. Given a set of training examples $(x_i, y_i)$, where $x_i$ is an input sample and $y_i$ is its corresponding target label, the network, represented by a function $h$ with parameters (weights) $\theta$, tries to produce an output $\hat{y}_i = h(x_i; \theta)$ that is very close to the true label $y_i$.

The "learning" happens by adjusting the network's parameters $\theta$ (the weights and biases within its layers) iteratively.

The Core Process: Iterative Learning

Training a neural network using supervised learning generally involves repeatedly cycling through the following steps:

1. Forward Pass: An input sample $x_i$ is fed into the network. It passes through the layers, undergoing computations at each neuron (weighted sum followed by an activation function), until it produces an output prediction, $\hat{y}_i$.
2. Loss Calculation: The network's prediction $\hat{y}_i$ is compared to the true target label $y_i$ using a loss function (also called a cost function or objective function). The loss function quantifies how wrong the prediction is; a higher loss value indicates a larger error. The choice of loss function depends on the task (e.g., Mean Squared Error for regression, Cross-Entropy for classification).
3. Backward Pass (Backpropagation): The error calculated by the loss function is propagated backward through the network. This process determines how much each weight and bias in the network contributed to the overall error.
4. Weight Update: Based on the error contribution calculated during the backward pass, an optimization algorithm (like Gradient Descent or its variants Adam, RMSprop) adjusts the network's weights $\theta$. The goal is to slightly modify the weights in a direction that will reduce the loss for that input sample (and hopefully, generalize to reduce loss on similar samples).

An overview of the supervised learning process in neural networks. Input data and target labels are used to generate predictions, calculate error (loss), and subsequently adjust the network's internal weights through backpropagation and optimization.

This entire cycle (forward pass, loss calculation, backward pass, weight update) is repeated for many input samples from the training dataset. Processing the entire dataset once constitutes an epoch. Training typically involves many epochs, allowing the network to gradually refine its weights and improve its predictive accuracy. We will explore loss functions, backpropagation, and optimizers in much greater detail in Chapter 3.

Common Supervised Learning Tasks

Neural networks excel at various supervised learning tasks, primarily categorized as:

• Regression: Predicting a continuous numerical value.

  • Examples: Predicting house prices based on features like size and location, forecasting temperature, estimating the age of a person from their photo.
  • Network Output: Typically a single output neuron with a linear activation function (no activation).
  • Common Loss Function: Mean Squared Error (MSE) or Mean Absolute Error (MAE).

• Classification: Assigning an input to one of several predefined categories or classes.

  • Binary Classification: Two possible output classes.
    • Examples: Spam detection (spam/not spam), medical diagnosis (disease/no disease), sentiment analysis (positive/negative).
    • Network Output: Typically a single output neuron with a sigmoid activation function, outputting a probability between 0 and 1.
    • Common Loss Function: Binary Cross-Entropy.
  • Multi-class Classification: More than two mutually exclusive output classes.
    • Examples: Handwritten digit recognition (classes 0-9), image classification (cat/dog/bird), object detection within an image.
    • Network Output: Typically one output neuron per class, using a softmax activation function. Softmax ensures the outputs represent a probability distribution across the classes (all outputs are between 0 and 1, and they sum to 1).
    • Common Loss Function: Categorical Cross-Entropy (or Sparse Categorical Cross-Entropy, depending on how labels are formatted).

The Importance of Data

Supervised learning heavily relies on the availability of high-quality, labeled data. The network learns patterns from the data it sees. Therefore, the training dataset must be:

• Large enough: Deep neural networks often have millions of parameters (weights) and require substantial amounts of data to learn effectively and avoid simply memorizing the training examples (a problem called overfitting).
• Representative: The training data should accurately reflect the characteristics of the real-world data the model will encounter after deployment. If the training data is biased or doesn't cover the full range of expected inputs, the model's performance will suffer.

In the upcoming sections and chapters, we will introduce Keras, a powerful library that simplifies the process of defining network architectures, choosing loss functions and optimizers, and executing the training loop for these supervised learning tasks. You'll learn how Keras abstracts away much of the low-level complexity, allowing you to focus on designing and training effective deep learning models.