A Simple Feedforward Network Example

Let's put the pieces together and visualize a basic feedforward neural network. We've discussed individual neurons, their parameters (weights and biases), activation functions, and how they are organized into layers. Now, imagine we want to build a simple network to, for instance, predict a single output value based on two input features.

Consider a network with the following structure:

• An Input Layer with 2 neurons, corresponding to our two input features ($x_1, x_2$). This layer doesn't perform computations; it simply passes the input values forward.
• A Hidden Layer with 3 neurons ($h_1, h_2, h_3$). Each hidden neuron receives input from all neurons in the input layer.
• An Output Layer with 1 neuron ($o_1$). This neuron receives input from all neurons in the hidden layer and produces the final prediction ($\hat{y}$).

This is called a "feedforward" network because information flows strictly in one direction: from input to hidden to output layers, without looping back.

Here's how information propagates:

1. Input to Hidden Layer:

   • Each input feature ($x_1, x_2$) is connected to each hidden neuron ($h_1, h_2, h_3$).
   • For the first hidden neuron ($h_1$), the input signals are multiplied by corresponding weights (e.g., $w_{11}$ for $x_1 \to h_1$, $w_{21}$ for $x_2 \to h_1$).
   • A weighted sum ($z_{h1}$) is calculated: $z_{h1} = (x_1 \times w_{11}) + (x_2 \times w_{21}) + b_{h1}$, where $b_{h1}$ is the bias for neuron $h_1$.
   • An activation function $f$ (like ReLU or Sigmoid) is applied to this sum to get the neuron's output: $a_{h1} = f(z_{h1})$.
   • Similar calculations ($z = \sum(weight \times input) + bias$ followed by $a = f(z)$) are performed independently for the other hidden neurons ($h_2, h_3$), each using its own set of weights and bias.

2. Hidden to Output Layer:

   • The outputs of the hidden layer neurons ($a_{h1}, a_{h2}, a_{h3}$) become the inputs for the output layer.
   • Each hidden neuron output is connected to the output neuron ($o_1$).
   • For the output neuron ($o_1$), a weighted sum ($z_{o1}$) is calculated using the hidden layer activations and a new set of weights (e.g., $w_{h1, o1}$ for $a_{h1} \to o_1$, etc.) and its own bias ($b_{o1}$): $z_{o1} = (a_{h1} \times w_{h1, o1}) + (a_{h2} \times w_{h2, o1}) + (a_{h3} \times w_{h3, o1}) + b_{o1}$.
   • A final activation function $g$ (which might be different from the hidden layer's activation function, depending on the task) is applied to get the network's prediction: $\hat{y} = a_{o1} = g(z_{o1})$.

This entire process, from feeding the initial inputs $x_1, x_2$ to obtaining the final output $\hat{y}$, is called forward propagation, which we will examine in detail later.

The diagram below illustrates this simple network structure.

A simple feedforward neural network with 2 input neurons, 1 hidden layer of 3 neurons, and 1 output neuron. Arrows indicate the direction of information flow during forward propagation. Each connection represents a weight, and each neuron in the hidden and output layers has an associated bias (not explicitly drawn).

This example showcases the fundamental architecture. Real-world networks can have many more layers (making them "deep") and many more neurons per layer, but the core principles of weighted sums, activation functions, and layered connections remain the same. The specific number of layers and neurons, along with the choice of activation functions, are design decisions that depend on the problem you are trying to solve. In the upcoming chapters, we'll explore how to implement these calculations efficiently and how the network learns the optimal values for its weights and biases.