Quadratic function of multiple variables
Contents
Definition
Consider variables . A quadratic function of the variables
is a function of the form:
In vector form, if we denote by the column vector with coordinates
, then we can write the function as:
where is a
matrix with entries
and
is the column vector with entries
.
Note that the matrix is non-unique: if
then we could replace
by
. Therefore, we could choose to replace
by the matrix
and have the advantage of working with a symmetric matrix.
Key data
For the discussion here, assume that has been made a symmetric matrix.
Item | Value | Consistency with the case ![]() ![]() ![]() ![]() ![]() |
---|---|---|
default domain | the whole of ![]() |
the whole of ![]() |
range | If the matrix ![]() ![]() If the matrix ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() |
The case of "not positive semidefinite or negative semidefinite" does not arise for ![]() The positive definite case corresponds to ![]() The negative definite case corresponds to ![]() |
local minimum value and points of attainment | If the matrix ![]() ![]() ![]() If ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() If ![]() ![]() ![]() |
The positive definite case corresponds to ![]() ![]() ![]() The negative definite case corresponds to ![]() |
local maximum value and points of attainment | If the matrix ![]() ![]() ![]() If ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() If ![]() ![]() ![]() |
The negative definite case corresponds to ![]() ![]() ![]() The positive definite case corresponds to ![]() |
gradient vector function (analogous to the derivative) | ![]() |
the derivative is ![]() |
Hessian matrix (analogous to the second derivative) | ![]() |
the second derivative is the constant function ![]() |
Differentiation
Partial derivatives and gradient vector
Case of general matrix
The partial derivative with respect to the variable , and therefore also the
coordinate of the gradient vector, is given by:
In terms of the matrix and vector notation, the gradient vector, expressed as a column vector, is:
Case of symmetric matrix
In the case that is a symmetric matrix, the above expressions simplify as follows.
Since for all
, the expression for the partial derivative becomes:
The expression for the gradient vector becomes:
Case 
A sanity check for the above expressions is that in the case , where
, we get the same answers as for the quadratic function
.
This is indeed the case. The only partial derivative here is the ordinary derivative, and this also is the gradient vector, and has expression:
This agrees with both the expression for and the expression for
.
Second-order partial derivatives and Hessian matrix
Case of general matrix
Recall that we had obtained (we replace the dummy variable by
to facilitate differentiation with respect to
in the next step):
Differentiating both sides with respect to (note that
may be equal to
or different from
) we find that the only term with a nonzero derivative is the term where
. In this case, the derivative is the coefficient of
. Therefore, we obtain:
Thus, the Hessian matrix of the quadratic function is given as:
Note that this is independent of the choice of . This fact is true only because of the nature of the function: for more general functional forms, the Hessian matrix varies with the choice of input vector.
We can also see this in matrix form directly. The gradient function is:
This is a linear transformation, and the Jacobian matrix of this linear transformation computes the Hessian that we want. We can use the well-known fact that the Jacobian matrix of a linear transformation coincides with the matrix describing the linear part of the transformation, and therefore the Hessian is:
Case of symmetric matrix
We can either plug into the formulas for the general case or perform similar calculations to get the formulas in the case that is a symmetric matrix:
Case 
A sanity check for the above expressions is that in the case , where
, we get the same answers as for the quadratic function
.
This is indeed the case. The only second-order partial derivative is . This agrees both with the formula for the second-order partial derivative and with the formula for the Hessian matrix.
Higher derivatives
All higher order partial derivatives (pure or mixed) are zero. This can be seen directly from the fact that the second-order partial derivatives are all constants, so differentiating them further (with respect to any variable) gives zero.
Therefore, the higher derivative tensors (the higher-order analogues of the gradient vector and Hessian matrix) are also identically zero.
Points and intervals of interest
For the discussion here, assume that is symmetric. If it is not, replace
by the matrix
.
Critical points
Case that the matrix
is invertible
To find the critical points, we need to set the gradient vector equal to zero. This gives the vector equation:
In other words:
Under the assumption that is invertible (or equivalently, that all its eigenvalues are nonzero), we can left-multiply both sides by
and obtain:
We thus have a unique critical point as described above.
Case that the matrix
is non-invertible
We still need to solve the same linear system:
However, since is no longer invertible (i.e., it has zero as an eigenvalue, or equivalently, it has a nonzero kernel), two cases arise:
- No solution exists. This happens if the vector
is not in the image of the linear transformation defined by
. Conceptually, what is happening is that the linear part of
is not constrained by the quadratic part, and therefore the function is unbounded.
- A solution exists, but it is not unique. This happens if the vector
is in the image of the linear transformation defined by
. In fact, the solution set is an affine space of dimension equal to the nullity of
. Thus, we get an affine space's worth of critical points.
Determination of local extremum behavior at critical points
Case that the matrix
is invertible
Recall that the Hessian matrix of is
. Therefore, by the second derivative test for a function of multiple variables, we obtain the following:
- If
is a symmetric positive-definite matrix, i.e., all its eigenvalues are positive, then the unique critical point
is a point of local minimum and is the unique point of absolute minimum (an alternate derivation of this fact is later in the page).
- If
is negative-definite matrix, i.e., all its eigenvalues are negative, then the unique critical point
is a point of local maximum and is the unique point of absolute maximum.
- If
has both positive and negative eigenvalues, then the unique critical point
is neither a point of local minimum nor a point of local maximum. In fact, it gives a saddle point for the function. There is no point of local extremum and the range of the function is all of
.
Case that the matrix
is non-invertible
In this case, the Hessian matrix, , is also non-invertible and in particular has zero as an eigenvalue.
In the case that there are no critical points, there is nothing to say.
In the case that there are critical points, we note that, through any critical point, there is a direction along which the second derivative is zero. In fact, what's happening geometrically is that the set of critical points form an affine subspace and the quadratic function is constant on that affine subspace. How the behavior of that affine subspace compares with the values of the function elsewhere depends on the nonzero eigenvalues.
- In the case that
is a symmetric positive-semidefinite matrix, i.e., all its nonzero eigenvalues are positive, the function attains a local minimum and also its absolute minimum at all its critical points. Note that, since the function is constant at this minimum value on the affine space, none of these is a point of strict local minimum.
- In the case that
is a symmetric negative-semidefinite matrix, i.e., all its nonzero eigenvalues are negative, the function attains a local maximum and also its absolute maximum value at all its critical points. Note that, since the function is constant at this minimum value on the affine space, none of these is a point of strict local minimum.
Geometry of the function
The geometry of the quadratic function is largely determined by the spectrum of the matrix (or equivalently, the spectrum of the matrix
. As before, we shall assume that
is a symmetric matrix. If not, replace
in the discussion below by
.
Eigenvalues of the Hessian
Since is a symmetric real matrix, it can be written in the form:
where is an orthogonal matrix and
is a diagonal matrix with real entries. The diagonal entries of
are the eigenvalues of
. Another way of thinking of the above is that with a change of basis that preserves the Euclidean norm, we can convert the Hessian to a diagonal transformation.
The following cases are of interest:
Case on ![]() |
Corresponding term for matrix ![]() ![]() |
---|---|
all positive | symmetric positive-definite matrix |
all nonnegative | symmetric positive-semidefinite matrix |
all negative | symmetric negative-definite matrix |
all nonpositive | symmetric negative-semidefinite matrix |
all nonzero | symmetric invertible matrix |
some positive, some negative (maybe some zero) | indefinite matrix |
Alternate analysis of extreme values
For the discussion of cases, assume that is a symmetric matrix. If
is not symmetric, replace it by the symmetric matrix
.
Positive definite case
First, we consider the case where is a symmetric positive definite matrix. In other words, we can write
in the form:
where is a
invertible matrix.
We can "complete the square" for this function:
In other words:
This is minimized when the expression whose norm we are measuring is zero, so that it is minimized when we have:
Simplifying, we obtain that we minimum occurs at:
Moreover, the value of the minimum is: