# Approximate Newton's method with constant approximate inverse Hessian

## Contents

## Definition

**Approximate Newton's method with constant approximate inverse Hessian** refers to a type of approximate Newton's method for optimization of a function of multiple variables where the matrix used as the stand-in for the inverse Hessian is constant across iterations, independent both of the current value of the iterate and of the number of iterations.

## Iterative step

Note that we use parenthesized superscripts to denote iterates and iteration-specific variables, and subscripts to denote coordinates. Parenthesized superscripts do *not* denote exponents.

Suppose we are applying gradient descent to minimize a function . We will denote the input to as a vector . Given a guess , gradient descent computes the next guess as follows:

Here, is intended as a stand-in (an approximation or a bound of some sort) on the inverse matrix of the Hessian matrix of . Unlike the usual Newton's method for optimization of a function of multiple variables, where would be replaced by the inverse Hessian *at the point* , the matrix needs to be independent of .

## Particular cases

### The one-dimensional version is just gradient descent with constant learning rate

In one dimension, this reduces to gradient descent with constant learning rate, where the constant approximate inverse Hessian is the matrix with entry equal to the constant learning rate. Therefore, the analysis of that problem applies here. In particular, see:

- Gradient descent with constant learning rate
- Gradient descent with constant learning rate for a quadratic function of one variable
- Gradient descent with constant learning rate for a convex function of one variable

### The general case

Function type | Discussion of approximate Newton's method with constant approximate inverse Hessian |
---|---|

quadratic function of multiple variables | approximate Newton's method with constant approximate inverse Hessian for a quadratic function of multiple variables |

convex function of multiple variables | approximate Newton's method with constant approximate inverse Hessian for a convex function of multiple variables |