Non-linear least squares

Definition

Non-linear least squares (NLLS) is a generalized problem type related to the problem of linear least squares. It occurs frequently in the context of optimization problems.

Consider the following setup: we have a model function $y=f(x,{\vec {\beta }})$ (here, $x$ may be a scalar or vector variable, but $y$ must be scalar; for simplicity, we will notationally treat $x$ as a scalar). The vector ${\vec {\beta }}$ is an unknown parameter vector with $n$ coordinates $\beta _{1},\beta _{2},\dots ,\beta _{n}$. We are given a set of $m$ data points $(x_{1},y_{1}),(x_{2},y_{2}),\dots ,(x_{m},y_{m})$ with $m\geq n$.

For $1\leq i\leq m$, we define the residual $r_{i}$ as follows:

$r_{i}=y_{i}-f(x_{i},{\vec {\beta }})$

Our goal is to find a choice of the parameter vector ${\vec {\beta }}$ for which the sum is minimized:

$\sum _{i=1}^{m}r_{i}^{2}$

In other words, we want to minimize the sum:

$\sum _{i=1}^{m}(y_{i}-f(x_{i},{\vec {\beta }}))^{2}$

How linear least squares is a special case

The case of linear least squares is the case where the function $f(x,{\vec {\beta }})$ is linear as a function of the vector ${\vec {\beta }}$ for each value of $x$. It need not be linear in $x$.