Back to the on-screen lesson ·
Stationarity as necessary, the second-order test as sufficient, and the saddle points in between.
Paper packet. Every task here also exists on screen, where it is checked automatically; answers written on paper are not assessed by Nydus. When you are back at a device, enter your answers there.
By the end of this lesson you will be able to find the stationary points of an unconstrained objective, classify each with the second-order test, and say why a zero gradient is necessary and not sufficient. You will also be able to explain what a saddle point is and why saddles dominate in many variables, to say what convexity adds — that solving the gradient equation is solving the problem — and to give both answers a minimisation has: where the minimum is, and what it is.
You found maxima and minima in calculus by setting a derivative to zero. You now also know what convexity adds to that: the classification is local, and only convexity upgrades it. This lesson is the same calculus, stated as the optimality conditions the rest of the course builds on — first order, then second, and the saddle points that live between them.
Gradient $\nabla f$: the vector of partial derivatives, one per variable.
Stationary point: a point where $\nabla f = 0$. A candidate, and the word is deliberately neutral about which kind.
Hessian $\nabla^2 f$: the matrix of second partial derivatives — the curvature, in every direction at once.
Positive definite: $d^\top \nabla^2 f\, d > 0$ for every non-zero direction $d$; the function curves upward whichever way you leave the point.
Indefinite: curving up in one direction and down in another.
Saddle point: a stationary point that is a minimum along one direction and a maximum along another. Stationary, and not optimal.
Necessary and sufficient: a necessary condition every minimum satisfies; a sufficient condition that guarantees one. Stationarity is the first and not the second.
For a differentiable $f$ with no constraints:
First-order necessary condition. If $x^\star$ is a local minimum then $\nabla f(x^\star) = 0$.
This is necessary, not sufficient. It produces candidates. A point with zero gradient is called stationary, and the word is deliberately neutral: it might be a minimum, a maximum, or a saddle.
Second-order sufficient condition. If $\nabla f(x^\star) = 0$ and the Hessian $\nabla^2 f(x^\star)$ is positive definite, then $x^\star$ is a strict local minimum.
In one variable the Hessian is $f''$ and positive definite means $f'' > 0$. In several variables, positive definite means $d^\top \nabla^2 f\, d > 0$ for every non-zero direction $d$ — the function curves upward whichever way you leave the point.
| Hessian at a stationary point | The point is |
|---|---|
| positive definite | a strict local minimum |
| negative definite | a strict local maximum |
| indefinite (both signs) | a saddle |
| singular | undecided by this test |
And then convexity. If $f$ is convex on the whole domain, stationarity is not merely necessary — it is sufficient, and the point is a global minimum. That is the payoff of lesson 9 stated in derivatives: for a convex function, solving $\nabla f = 0$ is solving the problem.
Another way: picture
Three surfaces, each flat at the origin. A bowl: every direction rises. A dome: every direction falls. A horse's saddle: sitting on it, the surface rises fore and aft and falls to left and right. All three have zero gradient at the middle, which is why the gradient alone cannot tell them apart.
Another way: steps
To minimise an unconstrained differentiable $f$:
Suppose $\nabla f(x^\star) \ne 0$, and set $d = -\nabla f(x^\star)$. Along that direction,
$$f(x^\star + t d) = f(x^\star) + t\, \nabla f(x^\star)^\top d + O(t^2) = f(x^\star) - t\, \|\nabla f(x^\star)\|^2 + O(t^2).$$
For small enough $t > 0$ the linear term dominates the remainder, so $f(x^\star + t d) < f(x^\star)$: there is a better point arbitrarily close by. So $x^\star$ was not a local minimum.
Two things worth keeping from this. First, the argument names the direction of steepest descent, $-\nabla f$, which is the whole idea of lesson 14. Second, it is local — it says nothing about points far away, which is precisely the limitation lesson 9 identified and convexity removes.
Stopping at stationarity. A zero gradient is a candidate. On a non-convex problem it may be a saddle, and high-dimensional problems have far more saddles than minima.
Reporting the location when asked for the value. Substitute back. It is one line and it is skipped constantly.
Using the second-order test as a convexity certificate. Positive definite at the point classifies the point. Convexity is positive semi-definite everywhere, and only that makes the answer global.
Forgetting the singular case. If the Hessian is singular the test is silent — $x^4$ and $-x^4$ and $x^3$ all have $f'' = 0$ at the origin and are a minimum, a maximum and neither. Silence is not a verdict.
In one variable saddles are rare — $x^3$ at the origin and not much else — so they get taught as an oddity. In many variables they are the opposite. A stationary point of a function of $n$ variables is a minimum only if the curvature is positive in all $n$ directions; a single negative direction makes it a saddle. For a Hessian with no particular structure that is like requiring $n$ coin flips to come up the same way, and saddles outnumber minima overwhelmingly as $n$ grows. This is why modern methods for large non-convex problems are designed around escaping saddles rather than around finding stationary points: finding a stationary point is easy and usually lands somewhere that is not an answer.
$f(x) = 3x^2 + 12x + 7$. Derivative $6x + 12$, zero at $x = -2$.
First order: the candidate.
Second derivative $6 > 0$, so a strict local minimum. And $6 > 0$ everywhere, so $f$ is convex and the minimum is global.
Second order classifies; convexity globalises.
Value: $3(4) + 12(-2) + 7 = 12 - 24 + 7 = -5$. The answer to "where" is $-2$ and the answer to "what" is $-5$.
Both answers, separately.
$f(x, y) = x^2 - y^2$. The gradient is $(2x, -2y)$, zero only at the origin.
One stationary point.
The Hessian is diagonal with entries $2$ and $-2$: indefinite. Along $x$ the function rises; along $y$ it falls.
Both signs present: a saddle.
So the origin is neither a minimum nor a maximum, and the function is unbounded below. A method that only checked $\nabla f = 0$ would report the origin with complete confidence — which is why every serious method in unit 3 has a second-order story, even when it does not compute a Hessian.
Necessary conditions are not answers.
$f'(x) = 3x^2 - 3$, zero at $x = 1$ and $x = -1$. Two candidates, so the function is certainly not convex.
More than one stationary point rules out convexity immediately.
$f''(x) = 6x$. At $x = 1$ that is $6 > 0$: a local minimum, value $1 - 3 = -2$. At $x = -1$ it is $-6 < 0$: a local maximum, value $2$.
Neither is global: $f \to -\infty$ as $x \to -\infty$, so the problem has no minimum at all. The local minimum at $x = 1$ is a perfectly correct local claim about a problem with no answer — which is the situation a method reports as convergence, and the reason the word "local" has to be said out loud.
Match each stationary-point condition to its classification.
| Local minimum | Local maximum | Saddle point | |
|---|---|---|---|
| Gradient zero; positive second derivative | |||
| Gradient zero; negative second derivative | |||
| Gradient zero; curvature changes by direction |
What is the derivative of $4x^2 + 5x + 4$?
Answer:
Describe the lowest point of $3x^2 + 12x + 7$: give where it is and what the value is there.
y = 3x^2 + 12x + 7
where the minimum is:
the value there:
At $x = 2$ the function $5x^2 + -20x + 26$ has second derivative $10$. What kind of point is it?
For $f(x,y) = 2(x^2 - y^2)$ the gradient is zero at the origin. Is the origin a minimum?
Lesson test: one question per skill, one attempt each, no hints. Your answers are checked when you submit.
For $f(x,y) = 8(x^2 - y^2)$ the gradient is zero at the origin. Is the origin a minimum?
You can find and classify stationary points, recognise a saddle, and say when stationarity settles the problem. Next: the same conditions when there are constraints in the way.
10. Your turn: classify the stationary points of $f(x) = x^3 - 3x$, step 3