Non-linear least squares

Definition

Non-linear least squares (NLLS) is a generalized problem type related to the problem of linear least squares. It occurs frequently in the context of optimization problems.

Consider the following setup: we have a model function $y=f(x,\overrightarrow{\beta })$ (here, $x$ may be a scalar or vector variable, but $y$ must be scalar; for simplicity, we will notationally treat $x$ as a scalar). The vector $\overrightarrow{\beta }$ is an unknown parameter vector with $n$ coordinates ${\beta }_1,{\beta }_2,\dots ,{\beta }_n$. We are given a set of $m$ data points $(x_1,y_1),(x_2,y_2),\dots ,(x_m,y_m)$ with $m\ge n$.

For $1\le i\le m$, we define the residual $r_i$ as follows:

$r_i=y_i-f(x_i,\overrightarrow{\beta })$

Our goal is to find a choice of the parameter vector $\overrightarrow{\beta }$ for which the sum is minimized:

$\sum _{i=1}^m{r}_i^2$

In other words, we want to minimize the sum:

$\sum _{i=1}^m(y_i-f(x_i,\overrightarrow{\beta }){)}^2$

How linear least squares is a special case

The case of linear least squares is the case where the function $f(x,\overrightarrow{\beta })$ is linear as a function of the vector $\overrightarrow{\beta }$ for each value of $x$. It need not be linear in $x$.