Gradient descent with decaying learning rate - Revision history

Vipul at 15:11, 1 September 2014

2014-09-01T15:11:47Z

← Older revision		Revision as of 15:11, 1 September 2014
Line 3:		Line 3:
	'''Gradient descent with decaying learning rate''' is a form of [[gradient descent]] where the learning rate varies as a function of the number of iterations, but is not otherwise dependent on the value of the vector at the stage. The update rule is as follows:		'''Gradient descent with decaying learning rate''' is a form of [[gradient descent]] where the learning rate varies as a function of the number of iterations, but is not otherwise dependent on the value of the vector at the stage. The update rule is as follows:

	<math>\vec{x}^{(k+1)} = \vec{x}^{(k)} - \alpha_k f\left(\vec{x^{(k)}}\right)</math>		<math>\vec{x}^{(k+1)} = \vec{x}^{(k)} - \alpha_k f\left(\vec{x}^{(k)}\right)</math>

	where <math>\alpha_k</math> depends only on <math>k</math> and not on the choice of <math>x^{(k)}</math>.		where <math>\alpha_k</math> depends only on <math>k</math> and not on the choice of <math>x^{(k)}</math>.

Vipul at 15:11, 1 September 2014

2014-09-01T15:11:32Z

← Older revision		Revision as of 15:11, 1 September 2014
Line 3:		Line 3:
	'''Gradient descent with decaying learning rate''' is a form of [[gradient descent]] where the learning rate varies as a function of the number of iterations, but is not otherwise dependent on the value of the vector at the stage. The update rule is as follows:		'''Gradient descent with decaying learning rate''' is a form of [[gradient descent]] where the learning rate varies as a function of the number of iterations, but is not otherwise dependent on the value of the vector at the stage. The update rule is as follows:

	<math>\vec{x}^{(k+1)} = \vec{x}^{(k)} - \alpha_k f\left(\vec{x^{(k)}\right)</math>		<math>\vec{x}^{(k+1)} = \vec{x}^{(k)} - \alpha_k f\left(\vec{x^{(k)}}\right)</math>

	where <math>\alpha_k</math> depends only on <math>k</math> and not on the choice of <math>x^{(k)}</math>.		where <math>\alpha_k</math> depends only on <math>k</math> and not on the choice of <math>x^{(k)}</math>.

Vipul: Created page with "==Definition== '''Gradient descent with decaying learning rate''' is a form of gradient descent where the learning rate varies as a function of the number of iterations,..."

2014-09-01T15:10:51Z

Created page with "==Definition== '''Gradient descent with decaying learning rate''' is a form of gradient descent where the learning rate varies as a function of the number of iterations,..."

New page

==Definition==

'''Gradient descent with decaying learning rate''' is a form of [[gradient descent]] where the learning rate varies as a function of the number of iterations, but is not otherwise dependent on the value of the vector at the stage. The update rule is as follows:

<math>\vec{x}^{(k+1)} = \vec{x}^{(k)} - \alpha_k f\left(\vec{x^{(k)}\right)</math>

where <math>\alpha_k</math> depends only on <math>k</math> and not on the choice of <math>x^{(k)}</math>.

==Cases==

{| class="sortable" border="1"
! Type of decay !! Example expression for <math>\alpha_k</math> !! More information
|-
| linear decay || <math>\alpha_k = \frac{\alpha_0}{k + 1}</math> || [[Gradient descent with linearly decaying learning rate]]
|-
| quadratic decay || <math>\alpha_k = \frac{\alpha_0}{(k + 1)^2}</math> || [[Gradient descent with quadratically decaying learning rate]]
|-
| exponential decay || <math>\alpha_k = \alpha_0 e^{-\beta k}</math> where <math>\beta > 0</math> || [[Gradient descent with exponentially decaying learning rate]]
|}