Structuring Networks: Layers and Connections

Now that we understand the components of an artificial neuron, including weights, biases, and activation functions, let's see how these units are organized to create a functional neural network. Individual neurons are powerful calculators, but it's their collective arrangement into layers and the connections between them that allow networks to learn intricate patterns from data.

The Layered Architecture

Neural networks are typically structured in layers. Each layer consists of one or more neurons. We generally distinguish between three types of layers:

1. Input Layer: This is the network's entry point. It doesn't perform any computation in the sense of applying weights and activation functions (or you can think of it as having an identity activation function). Instead, it simply holds the initial data fed into the network. The number of neurons in the input layer corresponds directly to the number of features in your input data. For example, if you're predicting house prices based on size (sq ft) and number of bedrooms, your input layer would have two neurons.

2. Hidden Layer(s): Situated between the input and output layers, hidden layers are where most of the computation occurs. Neurons in a hidden layer receive inputs from all neurons in the previous layer (either the input layer or another hidden layer), calculate their weighted sum plus bias ($z$), and apply an activation function ($a = f(z)$). The outputs ($a$) of one hidden layer then serve as inputs to the next layer. A network can have zero, one, or many hidden layers. Networks with multiple hidden layers are often referred to as "deep" neural networks. These intermediate layers enable the network to learn progressively more complex representations and combinations of the input features. The choice of the number of hidden layers and the number of neurons in each is a critical aspect of network design.

3. Output Layer: This is the final layer of the network, producing the ultimate result. The structure of the output layer depends heavily on the specific task the network is designed for:

   • Regression: For predicting a continuous value (like a house price), the output layer typically has a single neuron, often with a linear activation function (or no activation function).
   • Binary Classification: For classifying input into one of two categories (e.g., spam or not spam), the output layer usually has one neuron with a Sigmoid activation function, outputting a probability between 0 and 1.
   • Multi-class Classification: For classifying input into one of several categories (e.g., digit recognition 0-9), the output layer typically has one neuron per class, often using a Softmax activation function. Softmax ensures the outputs represent probabilities that sum to 1 across all classes.

Connections Between Layers

In the standard feedforward networks we're discussing initially, layers are typically fully connected (or dense). This means that every neuron in one layer sends its output signal to every neuron in the subsequent layer. Each of these connections has an associated weight, representing the strength of the connection.

Consider a simple network:

• Input Layer ($L_0$) with $n_0$ neurons (features).
• Hidden Layer ($L_1$) with $n_1$ neurons.
• Output Layer ($L_2$) with $n_2$ neurons.

In a fully connected setup:

• Each of the $n_1$ neurons in $L_1$ receives input from all $n_0$ neurons in $L_0$. This requires $n_0 \times n_1$ weights connecting $L_0$ to $L_1$. Each neuron in $L_1$ also has its own bias term.
• Each of the $n_2$ neurons in $L_2$ receives input from all $n_1$ neurons in $L_1$. This requires $n_1 \times n_2$ weights connecting $L_1$ to $L_2$. Each neuron in $L_2$ also has its own bias term.

The diagram below illustrates a simple fully connected feedforward network with one input layer (3 neurons), one hidden layer (4 neurons), and one output layer (2 neurons).

A simple feedforward neural network architecture. Input nodes (blue) pass data to the hidden layer nodes (green), which process it and pass results to the output nodes (red). Every node in one layer is connected to every node in the next layer.

This layered and connected structure defines the path that data follows through the network. When we feed data into the input layer, it propagates forward through the hidden layers, undergoing transformations at each step (linear combination via weights and biases, followed by non-linear activation), until it reaches the output layer, which produces the final prediction. This process is known as forward propagation, the topic of Chapter 3.