04 Gradients and Optimization

A guided introduction to directional derivatives, gradients, tangent and normal geometry, unconstrained extrema, and Lagrange multipliers for constrained optimization.

Directional Change in a Chosen Direction

A scalar-valued function assigns a number to each point. Its partial derivatives describe change in coordinate directions, while a describes change along any chosen direction.

If $\mathbf{u}$ is a , the of $f$ at $\mathbf{a}$ is

D_{\mathbf{u}}f(\mathbf{a})=\lim_{h\to 0}\frac{f(\mathbf{a}+h\mathbf{u})-f(\mathbf{a})}{h}.

For a differentiable function, compute it using

D_{\mathbf{u}}f=\nabla f\cdot\mathbf{u}.

The direction must have length $1$ . If the problem gives a nonzero vector $\mathbf{v}$ , normalize it first:

\mathbf{u}=\frac{\mathbf{v}}{\|\mathbf{v}\|}.

For example, let $f(x,y)=x^2+xy$ , evaluate the rate of change at $(1,2)$ , and use the direction of $\mathbf{v}=\langle 3,4\rangle$ . The is $\nabla f(x,y)=\langle 2x+y,x\rangle$ , so $\nabla f(1,2)=\langle 4,1\rangle$ . Since $\|\mathbf{v}\|=5$ , the unit direction is $\mathbf{u}=\left\langle\frac{3}{5},\frac{4}{5}\right\rangle$ . Therefore,

D_{\mathbf{u}}f(1,2)=\langle 4,1\rangle\cdot\left\langle\frac{3}{5},\frac{4}{5}\right\rangle=\frac{16}{5}.

The function increases at a rate of $\frac{16}{5}$ units per unit distance in that direction.

Takeaway: Find the , normalize the given direction, and take their dot product.

The as a Rate and Direction

For a differentiable function of three variables,

\nabla f=\left\langle\frac{\partial f}{\partial x},\frac{\partial f}{\partial y},\frac{\partial f}{\partial z}\right\rangle.

For two variables, $\nabla f=\langle f_x,f_y\rangle$ . The dot-product formula connects the to every :

D_{\mathbf{u}}f=\nabla f\cdot\mathbf{u}.

Because $\mathbf{u}$ has length $1$ , the Cauchy–Schwarz inequality gives

D_{\mathbf{u}}f\leq\|\nabla f\|.

Thus:

The maximum rate of increase is $\|\nabla f\|$ , occurring in the direction of $\nabla f$ .
The maximum rate of decrease is $-\|\nabla f\|$ , occurring in the direction of $-\nabla f$ .
A zero occurs in directions perpendicular to $\nabla f$ .

If $\nabla f(\mathbf{a})=\mathbf{0}$ , the first-order information does not select a preferred direction. The point could still be a local maximum, local minimum, or saddle point, so further analysis is required.

Takeaway: The simultaneously encodes all first-order directional rates and identifies the direction of steepest ascent.

Level Sets, Normals, and Tangent Planes

A level curve of a function of two variables is defined by $f(x,y)=c$ . If $\mathbf{r}(t)=\langle x(t),y(t)\rangle$ traces that curve, then $f(\mathbf{r}(t))=c$ . Differentiating gives

\nabla f(\mathbf{r}(t))\cdot\mathbf{r}'(t)=0.

Therefore, the tangent vector $\mathbf{r}'(t)$ is perpendicular to the . A tangent direction $\mathbf{t}$ satisfies $\nabla f\cdot\mathbf{t}=0$ .

For a surface defined implicitly by $F(x,y,z)=c$ , the at $P=(x_0,y_0,z_0)$ is a normal vector, provided $\nabla F(P)\neq\mathbf{0}$ . The tangent plane is

\nabla F(P)\cdot\langle x-x_0,y-y_0,z-z_0\rangle=0.

In coordinate form,

F_x(P)(x-x_0)+F_y(P)(y-y_0)+F_z(P)(z-z_0)=0.

For a graph $z=f(x,y)$ , write it as $F(x,y,z)=f(x,y)-z=0$ . A normal vector is $\langle f_x,f_y,-1\rangle$ , and the tangent-plane equation at $(x_0,y_0,f(x_0,y_0))$ is

z-f(x_0,y_0)=f_x(x_0,y_0)(x-x_0)+f_y(x_0,y_0)(y-y_0).

This equation is also the first-order linear approximation to the function.

Takeaway: Tangent directions lie along a level set, while the supplies a perpendicular normal direction.

Finding and Classifying Unconstrained Extrema

To locate possible local extrema of a differentiable function of two variables, first solve $\nabla f=\mathbf{0}$ . Also include interior points where a relevant partial derivative does not exist. These candidates are critical points.

For each , compute

D=f_{xx}f_{yy}-(f_{xy})^2.

The second-derivative test gives:

If $D>0$ and $f_{xx}>0$ , the point is a local minimum.
If $D>0$ and $f_{xx}<0$ , the point is a local maximum.
If $D<0$ , the point is a saddle point.
If $D=0$ , the test is inconclusive.

For a restricted domain, local analysis is not enough to find global extrema. Check interior critical points, boundary candidates, and corners when the boundary has corners. Then compare all objective values.

A zero alone does not prove that a point is an extremum. Classification requires the second-derivative test or another appropriate argument.

Takeaway: Critical points provide candidates; classification and boundary analysis determine which candidates are actual extrema.

One-Constraint Optimization with Lagrange Multipliers

A constrained optimization problem asks for the largest or smallest value of an objective function while one or more equations restrict the allowed points. With one constraint, the problem has the form

\text{optimize }f(x,y,z)\quad\text{subject to}\quad g(x,y,z)=c.

At a constrained extremum, movement is limited to tangent directions on the constraint surface. The derivative of the objective must be zero in every allowed tangent direction. Since $\nabla g$ is normal to the constraint surface, the objective must be parallel to it. This gives

\nabla f=\lambda\nabla g, \qquad g(x,y,z)=c.

Use the following procedure:

Identify the objective function and the constraint.
Compute $\nabla f$ and $\nabla g$ .
Solve $\nabla f=\lambda\nabla g$ together with $g=c$ .
Evaluate the objective at every candidate point.
Compare the values to identify the maximum and minimum.

For example, optimize $f(x,y)=xy$ subject to $x^2+y^2=1$ . With $g(x,y)=x^2+y^2$ , the equations are

\langle y,x\rangle=\lambda\langle 2x,2y\rangle, \qquad x^2+y^2=1.

They imply $x^2=y^2$ . If $y=x$ , the product is $\frac{1}{2}$ ; if $y=-x$ , the product is $-\frac{1}{2}$ . Therefore,

\max f=\frac{1}{2}

at $\left(\frac{1}{\sqrt{2}},\frac{1}{\sqrt{2}}\right)$ and $\left(-\frac{1}{\sqrt{2}},-\frac{1}{\sqrt{2}}\right)$ , while

\min f=-\frac{1}{2}

at $\left(\frac{1}{\sqrt{2}},-\frac{1}{\sqrt{2}}\right)$ and $\left(-\frac{1}{\sqrt{2}},\frac{1}{\sqrt{2}}\right)$ .

Takeaway: Lagrange equations locate constrained candidates, but evaluating and comparing the objective values is essential.

Multiple Constraints and Sensitivity

When two constraint equations restrict a function of three variables,

g(x,y,z)=c, \qquad h(x,y,z)=d,

the allowed set is generally the intersection of two surfaces. A tangent direction is perpendicular to both constraint normals, so the objective must lie in their span:

\nabla f=\lambda\nabla g+\mu\nabla h.

Solve the complete system

\begin{cases} \nabla f=\lambda\nabla g+\mu\nabla h,\\ g=c,\\ h=d. \end{cases}

The multipliers $\lambda$ and $\mu$ are unknowns alongside the coordinates. After finding all candidates, evaluate the objective and compare the resulting values. As with one constraint, the usual method assumes suitable smoothness and that the relevant constraint gradients do not fail to provide the required normal directions.

A multiplier can also describe sensitivity. If the constrained optimum has value $M(c)$ when the constraint level is $c$ , then under appropriate differentiability conditions,

\frac{dM}{dc}=\lambda.

Thus, a small increase in the constraint level changes the optimized value by approximately $\lambda$ times that increase. This interpretation is called the of the constraint.

Takeaway: With multiple constraints, combine the constraint gradients and interpret multipliers as approximate marginal changes in the optimal value.

A Reliable Problem-Solving Checklist

Several checks prevent common errors in and optimization problems:

Normalize a direction before using it in a .
Include the constraint equation when solving Lagrange equations.
Find and evaluate every candidate rather than reporting the first one found.
Examine boundaries and corners when the domain is restricted.
Do not assume every Lagrange candidate is a maximum; compare objective values or use additional analysis.
Check the regularity condition, such as $\nabla g\neq\mathbf{0}$ , when applying the standard one-constraint theorem.
Distinguish tangent vectors, which lie along a constraint, from constraint gradients, which are normal to it.

The central geometric connection is consistent throughout: gradients describe first-order change, gradients are normal to level sets, tangent directions are orthogonal to those normals, and constrained extrema occur when the objective is compatible with the available constraint normals.