Gradient vector

This article describes an analogue for functions of multiple variables of the following term/fact/notion for functions of one variable: derivative

Definition at a point

Generic definition

Suppose $f$ is a function of many variables. We can view $f$ as a function of a vector variable. The gradient vector at a particular point in the domain is a vector whose direction captures the direction (in the domain) along which changes to $f$ are concentrated, and whose magnitude is the directional derivative in that direction.

If the gradient vector of $f$ exists at a point, then we say that $f$ is differentiable at that point.

Formal epsilon-delta definition

Suppose $f$ is a function of a vector variable $\bar{x}$ . Suppose $\bar{c}$ is a point in the interior of the domain of $f$ , i.e., $f$ is defined in an open ball centered at $\bar{c}$ . The gradient vector of $f$ at $\bar{c}$ , denoted $(\nabla f) (\bar{c})$ , is a vector $\bar{v}$ satisfying the following:

For every $ϵ > 0$
there exists $δ > 0$ such that
for every $\bar{x}$ satisfying $0 < | \bar{x} - \bar{c} | < δ$ (in other words, $\bar{x}$ is in an open ball of radius $δ$ centered at $\bar{c}$ , but not qual to $\bar{c}$ )
we have $| f (\bar{x}) - f (\bar{c}) - \bar{v} \cdot (\bar{x} - \bar{c}) | < ϵ | \bar{x} - \bar{c} |$

Note on why the epsilon-delta definition is necessary

Intuitively, we want to define the gradient vector analogously to the derivative of a function of one variable, i.e., as the limit of the difference quotient:

$lim_{\bar{x} \to \bar{c}} \frac{f (\bar{x}) - f (\bar{c})}{\bar{x} - \bar{c}}$

Unfortunately, the above notation does not make direct sense because it is not permissible to divide a scalar by a vector. To rectify this, we revisit what the $ϵ - δ$ definition of the derivative says. It turns out that that $ϵ - δ$ definition can more readily be generalized to functions of vector variables. The key insight is to use the dot product of vectors.

Definition as a function

Generic definition

Suppose $f$ is a function of many variables. We can view $f$ as a function of a vector variable. The gradient vector of $f$ is a vector-valued function (with vector outputs in the same dimension as vector inputs) defined as follows: it sends every point to the gradient vector of the function at the point. Note that the domain of the function is precisely the subset of the domain of $f$ where the gradient vector is defined.

If the gradient vector of $f$ exists at all points of the domain of $f$ , we say that $f$ is differentiable everywhere on its domain.

Relation with directional derivatives

Statement of relation

For further information, refer: Relation between gradient vector and directional derivatives

Version type	Statement
at a point, in vector notation (multiple variables)	Suppose $f$ is a function of a vector variable $\bar{x} = ⟨ x_{1}, x_{2}, \dots, x_{n} ⟩$ . Suppose $\bar{u}$ is a unit vector and $\bar{a}$ is a point in the domain of $f$ . Suppose that the gradient vector of $f$ at $\bar{a}$ exists. We denote this gradient vector by $\nabla f (\bar{a})$ . Then, we have the following relationship: $D_{\bar{u}} (f) (\bar{a}) = \bar{u} \cdot (\nabla f (\bar{a}))$ The right side here is the dot product of vectors.
generic point, in vector notation (multiple variables)	Suppose $f$ is a function of a vector variable $\bar{x} = ⟨ x_{1}, x_{2}, \dots, x_{n} ⟩$ . Suppose $\bar{u}$ is a unit vector. We then have: $D_{\bar{u}} (f) (\bar{x}) = \bar{u} \cdot (\nabla f (\bar{x}))$ The right side here is a dot product of vectors. The equality holds whenever the right side makes sense.
generic point, point-free notation (multiple variables)	Suppose $f$ is a function of a vector variable $\bar{x} = ⟨ x_{1}, x_{2}, \dots, x_{n} ⟩$ . Suppose $\bar{u}$ is a unit vector. We then have: $D_{\bar{u}} (f) = \bar{u} \cdot (\nabla f)$ The right side here is a dot product of vector-valued functions (the constant function $\bar{u}$ and the gradient vector of $f$ ). The equality holds whenever the right side makes sense.

Relation with partial derivatives

For further information, refer: Relation between gradient vector and partial derivatives

Version type	Statement
at a point, in multivariable notation	Suppose $f$ is a real-valued function of $n$ variables $x_{1}, x_{2}, \dots, x_{n}$ . Suppose $(a_{1}, a_{2}, \dots, a_{n})$ is a point in the domain of $f$ such that the gradient vector of $f$ at $(a_{1}, a_{2}, \dots, a_{n})$ , denoted $(\nabla f) (a_{1}, a_{2}, \dots, a_{n})$ , exists. Then, the partial derivatives of $f$ with respect to all variables exist, and the coordinates of the gradient vector are the partial derivatives. In other words: $(\nabla f) (a_{1}, a_{2}, \dots, a_{n}) = ⟨ f_{x_{1}} (a_{1}, a_{2}, \dots, a_{n}), f_{x_{2}} (a_{1}, a_{2}, \dots, a_{n}), \dots f_{x_{n}} (a_{1}, a_{2}, \dots, a_{n}) ⟩$
generic point, in multivariable notation	Suppose $f$ is a real-valued function of $n$ variables $x_{1}, x_{2}, \dots, x_{n}$ . Then, we have $(\nabla f) (x_{1}, x_{2}, \dots, x_{n}) = ⟨ f_{x_{1}} (x_{1}, x_{2}, \dots, x_{n}), f_{x_{2}} (x_{1}, x_{2}, \dots, x_{n}), \dots f_{x_{n}} (x_{1}, x_{2}, \dots, x_{n}) ⟩$ . Equality holds wherever the left side makes sense.
generic point, point-free notation	Suppose $f$ is a function of $n$ variables $x_{1}, x_{2}, \dots, x_{n}$ . Then, we have $\nabla f = ⟨ f_{x_{1}}, f_{x_{2}}, \dots f_{x_{n}} ⟩$ . Equality holds wherever the left side makes sense.

Graphical interpretation

For a function of two variables

Suppose $f$ is a function of two variables $x, y$ and suppose $(x_{0}, y_{0})$ is a point in the domain. We say that $f$ is differentiable at a point $(x_{0}, y_{0})$ if the gradient vector exists at the point. This is equivalent to the graph of the function having a well defined tangent plane at $(x_{0}, y_{0}, f (x_{0}, y_{0}))$ . Further, the equation of this tangent plane is given by:

$z - f (x_{0}, y_{0}) = f_{x} (x_{0}, y_{0}) (x - x_{0}) + f_{y} (x_{0}, y_{0}) (y - y_{0})$

Another way of putting this is:

$z - f (x_{0}, y_{0}) = (\nabla f) (x_{0}, y_{0}) \cdot (⟨ x, y ⟩ - ⟨ x_{0}, y_{0} ⟩)$

Note that it is possible that the partial derivatives both exist but the function is not differentiable. In this case, the surface does not have a well defined tangent plane at the point. Even though we can define a plane by the equation above, this is not the tangent plane, because the tangent plane does not exist.

For a function of multiple variables

Suppose $f$ is a function of multiple variables $x_{1}, x_{2}, \dots, x_{n}$ and suppose $(a_{1}, a_{2}, \dots, a_{n})$ is a point in the domain of $f$ . We say that $f$ is differentiable at $(a_{1}, a_{2}, \dots, a_{n})$ if the gradient vector $(\nabla f) (a_{1}, a_{2}, \dots, a_{n})$ exists. This is equivalent to the graph of the function having a well defined tangent hyperplane at the point $(a_{1}, a_{2}, \dots, a_{n}, f (a_{1}, a_{2}, \dot{,} a_{n}))$ . The equation of the tangent hyperplane is given by:

$x_{n + 1} - f (a_{1}, a_{2}, \dots, a_{n}) = (\nabla f) (a_{1}, a_{2}, \dots, a_{n}) \cdot (⟨ x_{1}, x_{2}, \dots, x_{n} ⟩ - ⟨ a_{1}, a_{2}, \dots, a_{n} ⟩)$