Classification with a Neural Network
Now that you know what a perceptron is, you are just one step away from knowing what a neural network is. A neural network is a collection of perceptrons organized in layers, where the outputs of one layer become the inputs to the next layer. Training works the same way as before (gradient descent), except now there are many more weights and biases to update.
From One Perceptron to a Neural Network
In the previous pages, we built a single perceptron for binary classification. It computed a weighted sum of the inputs, applied the sigmoid activation function, and output a prediction \(\hat{y}\) between 0 and 1. The perceptron then used log loss and gradient descent to learn the best weights.
That single perceptron works well, but it has a fundamental limitation. The decision boundary it produces is always a straight line. Real-world problems are rarely that simple. Language, images, and most natural data have complex, nonlinear boundaries between classes.
The solution is to combine multiple perceptrons into a network. By stacking perceptrons in layers, the network can learn decision boundaries of arbitrary complexity.
Why a Neural Network?
A single perceptron draws one straight line through the data. That works for simple problems, but language (and most real data) is far more complex. A single line cannot capture the patterns. By stacking multiple perceptrons, we allow the network to combine multiple lines into a complex, nonlinear decision boundary. The more perceptrons and layers, the more complex the boundaries the network can learn.
Two-Layer Neural Network
The simplest neural network beyond a single perceptron has three components.
- An input layer that receives the features (\(x_1\), \(x_2\)).
- A hidden layer with multiple perceptrons (we will use two).
- An output layer with one perceptron that produces the final prediction \(\hat{y}\).
The hidden layer is called “hidden” because its outputs are not directly observed in the training data. They are intermediate values that the network learns on its own.
How the Layers Connect
Each perceptron in the hidden layer takes the same inputs (\(x_1\), \(x_2\)) and produces its own output. These outputs then feed into the output perceptron, which combines them to make the final prediction.
Think of it this way: each hidden perceptron draws its own line through the data. The output perceptron then combines those lines into a more complex decision boundary that a single line could never achieve.
Math Inside the Network
Let us label the two hidden perceptrons as node 1 (red) and node 2 (green), and the output perceptron as the output node (purple).
Output Layer (Purple)
The output node takes \(a_1\) and \(a_2\) as its inputs. It has weights \(w_1\), \(w_2\) and bias \(b\). It computes:
\[z = a_1 w_1 + a_2 w_2 + b\]
\[\hat{y} = \sigma(z)\]
This final \(\hat{y}\) is the network’s prediction, a number between 0 and 1 representing the probability of class 1.
Full Picture
Putting it all together, the flow through the network is:
- The inputs \(x_1\) and \(x_2\) enter the network.
- Each hidden node computes a weighted sum plus bias, applies the sigmoid, and outputs a value (\(a_1\) and \(a_2\)).
- The output node takes \(a_1\) and \(a_2\), computes a weighted sum plus bias, applies the sigmoid, and produces the final prediction \(\hat{y}\).
The network has the following parameters to learn:
- Hidden layer: \(w_{11}\), \(w_{21}\), \(b_1\), \(w_{12}\), \(w_{22}\), \(b_2\) (6 parameters)
- Output layer: \(w_1\), \(w_2\), \(b\) (3 parameters)
That is 9 parameters total. A single perceptron only had 3. More parameters means the model can represent more complex patterns.
Loss Function
Since this is still a classification problem, we use the same log loss function as before:
\[L(y, \hat{y}) = -y\ln(\hat{y}) - (1 - y)\ln(1 - \hat{y})\]
where \(\hat{y}\) is the network’s prediction and \(y\) is the true label from the training data.
The goal remains the same: find the values of all 9 parameters that minimize this loss across the training data.
Training the Neural Network
Training a neural network uses gradient descent, exactly as we did with the single perceptron. The only difference is that now we have 9 parameters instead of 3, so we need 9 partial derivatives. We need to compute:
\[\frac{\partial L}{\partial w_{11}}, \quad \frac{\partial L}{\partial w_{21}}, \quad \frac{\partial L}{\partial b_1}, \quad \frac{\partial L}{\partial w_{12}}, \quad \frac{\partial L}{\partial w_{22}}, \quad \frac{\partial L}{\partial b_2}, \quad \frac{\partial L}{\partial w_1}, \quad \frac{\partial L}{\partial w_2}, \quad \frac{\partial L}{\partial b}\]
Each of these tells us how to adjust one parameter to reduce the loss. The gradient descent update rule for each parameter \(\theta\) is the same as before:
\[\theta \leftarrow \theta - \alpha \frac{\partial L}{\partial \theta}\]
where \(\alpha\) is the learning rate.
The technique for computing all of these derivatives efficiently is called backpropagation.
Backpropagation
Backpropagation is nothing more than the chain rule applied systematically, layer by layer, starting from the output and working backward through the network.
Scaling Up: Notation for Deeper Networks
Before we compute the derivatives, let us introduce a cleaner notation that scales to networks with many layers. We add a superscript to indicate which layer a weight, bias, or activation belongs to.
- Layer 1 (first hidden layer): weights \(w^{(1)}\), biases \(b^{(1)}\), linear combinations \(z^{(1)}\), activations \(a^{(1)}\)
- Layer 2 (second hidden layer or output): weights \(w^{(2)}\), biases \(b^{(2)}\), linear combinations \(z^{(2)}\), activations \(a^{(2)}\)
- Layer 3, 4, 5, … follow the same pattern.
This notation makes it straightforward to describe networks of any depth. A network with 10 hidden layers would simply have superscripts from 1 to 11 (10 hidden layers plus the output layer).
Chain Rule, Layer by Layer
To compute \(\frac{\partial L}{\partial w^{(k)}}\) for a weight in layer \(k\) of an \(n\)-layer network (where \(\hat{y} = a^{(n)}\)), we write out the chain of variables that connect \(w^{(k)}\) to the loss \(L\):
\[\frac{\partial L}{\partial w^{(k)}} = \frac{\partial L}{\partial a^{(n)}} \cdot \frac{\partial a^{(n)}}{\partial z^{(n)}} \cdot \frac{\partial z^{(n)}}{\partial a^{(n-1)}} \cdots \frac{\partial z^{(k)}}{\partial w^{(k)}}\]
Each factor in this product is either:
- The derivative of the sigmoid function (which we already know is \(\sigma(z)(1 - \sigma(z))\)), or
- A simple linear derivative (a weight value or an input value).
Both are trivial to compute individually. The power of backpropagation is that it handles the bookkeeping of multiplying them together in the right order.
Reusing Computed Derivatives
A key insight: when computing derivatives for earlier layers, we reuse the derivatives already computed for later layers. For example, \(\frac{\partial L}{\partial a^{(2)}}\) appears in the chain for every weight in layers 1 and 2. We compute it once, store it, and reuse it. This is what makes backpropagation efficient. Without this reuse, we would redundantly recompute the same sub-chains over and over.
The algorithm works backward (hence “back” propagation):
- Compute the derivative at the output layer.
- Use that to compute derivatives at the second-to-last layer.
- Continue backward until we reach the first layer.
At each step, every derivative is either a sigmoid derivative or a linear function, so the computation at each node is simple. The complexity comes only from the number of nodes and connections, not from the math at each step.
In Practice
If you are a machine learning practitioner, you will rarely compute these derivatives by hand. Libraries like PyTorch and TensorFlow handle backpropagation automatically. But understanding the mechanics at least once gives you intuition for why certain architectures train well (or poorly), why gradients can vanish in deep networks, and how to debug training issues.
Practice Questions
1. For a Regression using a Single Layer Perceptron, select all that apply:
The Loss Function used is \(L(y,\hat{y}) = -y\ln(\hat{y}) - (1-y)\ln(1-\hat{y})\).
The Loss Function used is \(L(y,\hat{y}) = \frac{1}{2}(y - \hat{y})^2\).
To minimize the Loss Function, we consider \(L(y,\hat{y})\) as a function of \(w_1\), \(w_2\) and \(b\).
To minimize the Loss Function, we consider \(L(y,\hat{y})\) as a function of \(x_1\) and \(x_2\).
b and c.
(a) is false. This is the log loss function, which is used for classification, not regression. Regression uses the squared error.
(b) is true. The mean squared error (or half of it) is the standard loss function for regression problems.
(c) is true. The weights \(w_1\), \(w_2\) and bias \(b\) are the parameters we are trying to learn. Minimizing the loss means finding the values of these parameters that produce the best predictions.
(d) is false. The inputs \(x_1\) and \(x_2\) are the data (features). They are fixed values from the training set. We do not optimize over them — we optimize over the weights and bias.
1. About Backpropagation, check all that apply:
It is a way to obtain the input values for a given output of a neural network.
It is a method to update the parameters of a neural network.
It is the same as gradient descent.
It is a method that starts in the output layer and finishes in the input layer.
b and d.
(a) is false. Backpropagation does not find inputs. It computes the gradients (partial derivatives) of the loss with respect to each parameter in the network.
(b) is true. Backpropagation computes the derivatives that are then used in the gradient descent update step to adjust all the weights and biases.
(c) is false. Backpropagation and gradient descent are related but different. Backpropagation is the algorithm for computing the gradients. Gradient descent is the algorithm for using those gradients to update the parameters. Backpropagation feeds into gradient descent, but they are not the same thing.
(d) is true. Backpropagation starts at the output layer (where the loss is computed) and works backward through the network, computing derivatives layer by layer until it reaches the input layer. This backward direction is what gives it its name.