{
const Plotly = await import("https://cdn.jsdelivr.net/npm/plotly.js-dist@2/+esm");
const res = 50;
const x = Array.from({length: res}, (_, i) => -2 + i * 4 / (res - 1));
const y = Array.from({length: res}, (_, i) => -2 + i * 4 / (res - 1));
const z = y.map(yi => x.map(xi => 2*xi*xi + 3*yi*yi - xi*yi));
const div = document.createElement("div");
div.style.width = "100%";
const surface = {
type: 'surface', x: x, y: y, z: z,
colorscale: 'RdBu', opacity: 0.9, showscale: false
};
const layout = {
scene: {
xaxis: {title: 'x'},
yaxis: {title: 'y'},
zaxis: {title: 'f(x, y)'},
camera: {eye: {x: 1.5, y: -1.8, z: 1.0}}
},
margin: {l: 0, r: 0, t: 30, b: 0},
title: {text: 'f(x, y) = 2x² + 3y² − xy', font: {size: 13, color: 'gray'}},
height: 450
};
Plotly.default.newPlot(div, [surface], layout, {responsive: true});
return div;
}Hessian Matrix
calculus
In the previous page, you learned the second derivative and how it helps with optimization in one variable. Now we extend this to functions of multiple…
In the previous page, you learned the second derivative and how it helps with optimization in one variable. Now we extend this to functions of multiple variables. For multiple variables, the second derivative is not a single number but a matrix of second derivatives called the Hessian. It plays the same role in multivariable optimization that \(f''(x)\) plays in single-variable optimization.
One Variable vs Two Variables
| One variable | Two variables | |
|---|---|---|
| Function | \(f(x)\) | \(f(x, y)\) |
| First derivative | \(f'(x)\) | \(\nabla f = \begin{bmatrix} f_x \\ f_y \end{bmatrix}\) (gradient, a vector) |
| Second derivative | \(f''(x)\) (a number) | \(H = \begin{bmatrix} f_{xx} & f_{xy} \\ f_{yx} & f_{yy} \end{bmatrix}\) (Hessian, a matrix) |
In one variable, the second derivative is a single number that tells us the curvature. In two variables, there are four second-order partial derivatives, and they form a \(2 \times 2\) matrix.
Worked Example
Let us compute all four second partial derivatives for:
\[f(x, y) = 2x^2 + 3y^2 - xy\]
First Partial Derivatives
\[f_x = \frac{\partial f}{\partial x} = 4x - y\]
\[f_y = \frac{\partial f}{\partial y} = 6y - x\]
Second Partial Derivatives
Now differentiate each first partial derivative with respect to both \(x\) and \(y\):
\[f_{xx} = \frac{\partial}{\partial x}(4x - y) = 4\]
\[f_{xy} = \frac{\partial}{\partial y}(4x - y) = -1\]
\[f_{yx} = \frac{\partial}{\partial x}(6y - x) = -1\]
\[f_{yy} = \frac{\partial}{\partial y}(6y - x) = 6\]
Notice that \(f_{xy} = f_{yx} = -1\). This is not a coincidence.
Hessian Matrix
Arrange these four values into a matrix:
\[H = \begin{bmatrix} f_{xx} & f_{xy} \\ f_{yx} & f_{yy} \end{bmatrix} = \begin{bmatrix} 4 & -1 \\ -1 & 6 \end{bmatrix}\]
This is the Hessian matrix of \(f\).
Four Types of Second Partial Derivatives
Given a function \(f(x, y)\) with first partial derivatives \(f_x\) and \(f_y\), we can differentiate each one with respect to each variable:
| Meaning | Leibniz notation | Lagrange notation |
|---|---|---|
| Rate of change of \(f_x(x,y)\) w.r.t \(x\) | \(\frac{\partial^2 f}{\partial x^2}\) | \(f_{xx}(x,y)\) |
| Rate of change of \(f_y(x,y)\) w.r.t \(y\) | \(\frac{\partial^2 f}{\partial y^2}\) | \(f_{yy}(x,y)\) |
| Rate of change of \(f_x(x,y)\) w.r.t \(y\) | \(\frac{\partial^2 f}{\partial x \partial y}\) | \(f_{xy}(x,y)\) |
| Rate of change of \(f_y(x,y)\) w.r.t \(x\) | \(\frac{\partial^2 f}{\partial y \partial x}\) | \(f_{yx}(x,y)\) |
The first two (\(f_{xx}\) and \(f_{yy}\)) work just like the one-variable second derivative: they measure curvature along a single axis.
The last two (\(f_{xy}\) and \(f_{yx}\)) are the mixed partials: they measure how the slope along one axis changes when you move along the other axis.
Symmetry of Mixed Partials
In our example, \(f_{xy} = f_{yx} = -1\). This happens almost always. Whenever both partial derivatives are differentiable (which covers all “nice” functions you will encounter in machine learning), the mixed partials are equal:
\[f_{xy} = f_{yx}\]
This means the Hessian matrix is always symmetric. The order of differentiation does not matter.
General Form
For any function \(f(x, y)\) with continuous second partial derivatives, we differentiate in a tree structure:
\[f(x,y) \xrightarrow{\partial / \partial x} f_x(x,y) \begin{cases} \xrightarrow{\partial / \partial x} f_{xx}(x,y) \\ \xrightarrow{\partial / \partial y} f_{xy}(x,y) \end{cases}\]
\[f(x,y) \xrightarrow{\partial / \partial y} f_y(x,y) \begin{cases} \xrightarrow{\partial / \partial x} f_{yx}(x,y) \\ \xrightarrow{\partial / \partial y} f_{yy}(x,y) \end{cases}\]
Collecting all four second partial derivatives into a matrix gives the Hessian matrix:
\[H = \begin{bmatrix} f_{xx}(x,y) & f_{xy}(x,y) \\ f_{yx}(x,y) & f_{yy}(x,y) \end{bmatrix}\]
This single matrix contains all information about the second derivatives of \(f\).
In Leibniz notation:
\[H = \begin{bmatrix} \frac{\partial^2 f}{\partial x^2} & \frac{\partial^2 f}{\partial x \partial y} \\ \frac{\partial^2 f}{\partial y \partial x} & \frac{\partial^2 f}{\partial y^2} \end{bmatrix}\]
For a function of \(n\) variables \(f(x_1, x_2, \ldots, x_n)\), the Hessian is an \(n \times n\) symmetric matrix where the entry in row \(i\), column \(j\) is \(\frac{\partial^2 f}{\partial x_i \partial x_j}\).
Why the Hessian Matters
The Hessian is the multivariable generalization of the second derivative. Just as \(f''(x)\) tells us whether a critical point is a minimum or maximum in one variable, the Hessian tells us the same thing in multiple variables. In the next page, we will see exactly how.
Practice Questions
1. If \(f(x, y) = x^2 + y^2\), then the Hessian Matrix is:
\(\begin{bmatrix} 2 & 0 \\ 0 & 2 \end{bmatrix}\)
\(\begin{bmatrix} 2x & 0 \\ 0 & 2y \end{bmatrix}\)
\(\begin{bmatrix} 0 & 0 \\ 0 & 0 \end{bmatrix}\)
TipAnswer
a. \(\begin{bmatrix} 2 & 0 \\ 0 & 2 \end{bmatrix}\)
Since \(f_x(x,y) = 2x\) and \(f_y(x,y) = 2y\), then:
\[\nabla f_x(x,y) = \begin{bmatrix} 2 \\ 0 \end{bmatrix}, \quad \nabla f_y(x,y) = \begin{bmatrix} 0 \\ 2 \end{bmatrix}\]
The Hessian is obtained by stacking the transposed gradients of each partial derivative:
\[H = \begin{bmatrix} f_{xx} & f_{xy} \\ f_{yx} & f_{yy} \end{bmatrix} = \begin{bmatrix} 2 & 0 \\ 0 & 2 \end{bmatrix}\]
Option (b) is wrong because it shows the gradient (first derivatives), not the second derivatives. Option (c) is wrong because the second derivatives of \(x^2\) and \(y^2\) are 2, not 0.
Hessian and Concavity
In one variable, the second derivative test was simple:
- \(f''(x) > 0\) at a critical point \(\Rightarrow\) local minimum
- \(f''(x) < 0\) at a critical point \(\Rightarrow\) local maximum
- \(f''(x) = 0\) \(\Rightarrow\) inconclusive
For functions of two or more variables, the Hessian replaces the second derivative. But a matrix is not simply “positive” or “negative.” Instead, we check its eigenvalues.
Example 1: Concave Up (Local Minimum)
Consider \(f(x, y) = 2x^2 + 3y^2 - xy\) (the function from above). Its Hessian is:
\[H = \begin{bmatrix} 4 & -1 \\ -1 & 6 \end{bmatrix}\]
To find the eigenvalues, solve \(\det(H - \lambda I) = 0\):
\[\det \begin{bmatrix} 4 - \lambda & -1 \\ -1 & 6 - \lambda \end{bmatrix} = (4-\lambda)(6-\lambda) - 1 = \lambda^2 - 10\lambda + 23 = 0\]
The roots are \(\lambda_1 \approx 6.41\) and \(\lambda_2 \approx 3.59\). Both are positive. This means the Hessian is positive definite, the function is concave up, and \((0, 0)\) is a local minimum.
Example 2: Concave Down (Local Maximum)
Consider \(f(x, y) = -2x^2 - 3y^2 - xy + 15\). Its Hessian is:
\[H = \begin{bmatrix} -4 & -1 \\ -1 & -6 \end{bmatrix}\]
The eigenvalues are \(\lambda_1 \approx -3.59\) and \(\lambda_2 \approx -6.41\). Both are negative. The Hessian is negative definite, the function is concave down, and \((0, 0)\) is a local maximum.
Example 3: Saddle Point
Consider \(f(x, y) = 2x^2 - 2y^2\). Its gradient is \(\nabla f = \begin{bmatrix} 4x \\ -4y \end{bmatrix}\), which equals zero at \((0, 0)\). The Hessian is:
\[H = \begin{bmatrix} 4 & 0 \\ 0 & -4 \end{bmatrix}\]
The eigenvalues are \(\lambda_1 = 4\) and \(\lambda_2 = -4\). One is positive, one is negative. The Hessian is neither positive definite nor negative definite. The point \((0, 0)\) is a saddle point: a minimum in the \(x\)-direction but a maximum in the \(y\)-direction.
Summary: Second Derivative Test (Multivariable)
| 1 variable \(f(x)\) | 2 variables \(f(x,y)\) | More variables \(f(x_1, x_2, \ldots, x_n)\) | |
|---|---|---|---|
| (Local) minima | Happy face \(f''(x) > 0\) |
Upper paraboloid \(\lambda_1 > 0\) and \(\lambda_2 > 0\) |
All \(\lambda_i > 0\) |
| (Local) maxima | Sad face \(f''(x) < 0\) |
Down paraboloid \(\lambda_1 < 0\) and \(\lambda_2 < 0\) |
All \(\lambda_i < 0\) |
| Need more information | \(f''(x) = 0\) | Saddle point \(\lambda_1 > 0\) and \(\lambda_2 < 0\) \(\lambda_1 < 0\) and \(\lambda_2 > 0\) Or some \(\lambda_i = 0\) |
Some \(\lambda_i > 0\) and some \(\lambda_j < 0\) OR At least one \(\lambda_i = 0\) |
The rule is simple: if all eigenvalues of the Hessian are strictly positive, you have a minimum. If all are strictly negative, a maximum. Anything else (some positive, some negative, or any zeros) is inconclusive with this test alone.