Gradient descent with exact line search for a quadratic function of multiple variables
This page describes gradient descent with exact line search for a quadratic function of multiple variables. Since the function is quadratic, its restriction to any line is quadratic, and therefore the line search on any line can be implemented using Newton's method. Therefore, the analysis on this page also applies to using gradient descent using Newton's method for a quadratic function of multiple variables.
The function we are interested is a function of the form:
where is a symmetric positive-definite matrix with entries and is the column vector with entries . For part of this page, we will generalize somewhat to the case that is a symmetric positive-semidefinite matrix.
Note that we impose the condition of symmetry because we could replace the matrix with the symmetric matrix without changing the functional form. The positive-definite or positive-semidefinite condition is to guarantee that the function has the right sort of convexity. In the positive-definite case, we are guaranteed that there is a unique point of local minimum that is also the point of absolute minimum. The positive-semidefinite case is somewhat more complicated: we could either have no minimum, or we could have an affine space worth of points at which the function has a local and absolute minimum.
The algorithm starts with a guess and updates according to the following rule. We use superscripts to denote the iterates, to avoid confusion with the use of subscripts for coordinates.
where is the second derivative of along the direction of the gradient vector. Explicitly, if we set , we have:
where the matrix in the denominator arises as the Hessian of the function . Since the function is quadratic, the Hessian is globally constant.
We will carry out our analysis of eigenvalues in terms of the Hessian matrix . If we were instead dealing with the eigenvalues of , we would have to double them to get the eigenvalues of . Since the bounds on the learning rate are described as reciprocals of the eigenvalues, we would therefore obtain additional factors of 2 in the denominators, or equivalently, we would remove the factor of 2 from the numerators.
Note that in the case , we have .
Detailed analysis after transforming the problem to a setting where the coordinates are simple
Suppose the eigenvalues of are:
Via a coordinate change of the domain using an orthogonal matrix, we can transform into a diagonal matrix with positive real entries. These entries are halves of the eigenvalues of . Via a translation, we can get rid of the linear term, and finally, we can translate the function value by a constant (this does not affect the point of local minimum, though it affects the minimum value). We can therefore obtain the following simplified functional form:
The unique local and absolute minimum is at the zero vector.
Note that even though we know that our matrix can be transformed this way, we do not in general know how to bring it in this form -- if we did, we could directly solve the problem without using gradient descent (this is an alternate solution method). However, even though we may not know the explicit diagonalized form of the function, the fact that it does have such a form gives us information about how the gradient descent process converges.
Starting with a point:
we obtain that the gradient vector at the point is:
The multiplier is:
This simplifies to:
Therefore, the update in each coordinate is as follows:
This can also be simplified as:
The one-dimensional case
In the case that , gradient descent with exact line search converges in one iteration for a quadratic function. In fact, it is equivalent to Newton's method for optimization of a function of one variable, which we know converges in one iteration for a quadratic function.
We can verify this using the above expression: if , then after one iteration, we get to 0, the point of local and absolute minimum.
The two-dimensional case
The two-dimensional case is qualitatively distinct from the general case in the following respect: if we consider the vectors obtained after an even number of iterations, these are collinear and heading towards the optimum at a constant convergence rate. Using the simplified setup here, we work out the details, including the convergence rate.
In our setup, the function has the function:
We have the rule from above:
Because there are only two coordinates, the expressions can be simplified somewhat:
Suppose we start with the zeroth iterate . For convenience, we omit superscripts for the zeroth iterate, i.e., we set , . Applying the above gives us:
We also have similar expressions for :
We now want to work out the expression , that appear in the denominator for calculating . Let's do that. We simply plug in the expressions of and from the first two of the above four equations.
Note that . Adding up, we get:
Plugging in the results from Equations 1, 2, and 7 into Equations 3 and 4, we obtain:
In particular, note that , so every two iterations, we move directly in the line toward the minimum.
Moreover, the common value equals:
- If , convergence is immediate. This makes sense -- we are converging in only one coordinate, and the problem reduces to the one-dimensional case.
- If , convergence is immediate, because we move coordinated amounts in both directions.
As in the earlier section, assume , so and . The value is the condition number. The above expression works out to:
The linear convergence rate on the domain side is the square root of this quantity, because we get this multiple every two iterations, i.e., the linear convergence rate is:
To obtain the linear convergence rate on the function side, we square back, and obtain:
Note that the convergence rate is sensitive to the choice of starting point. Let's make cases based on starting point:
- : In this case, convergence occurs in one step.
- : In this case, convergence occurs in one step.
- Both and are nonzero. In this case, the convergence rate is positive. Let's compute bounds on this.
Suppose . The convergence rate simplifies to:
After doing some calculus to minimize this function, we find that the minimum occurs when , or equivalently, when . In other words, the convergence is slowest when the values in the coordinates are inversely proportional to the eigenvalues. In this case, the convergence rate on the domain side works out to the following:
The convergence rate on the function value side works out to:
In this worst-case scenario, the iterates alternate between the lines and .
Note that, even in this worst-case scenario, the convergence rate matches the best possible convergence rate we can get for gradient descent (attained when the learning rate is ) as discussed on the page gradient descent with constant learning rate for a quadratic function of multiple variables.