Gradient descent using Newton's method - Revision history

Vipul at 22:31, 8 May 2016

2016-05-08T22:31:53Z

← Older revision		Revision as of 22:31, 8 May 2016
Line 13:		Line 13:
	where <math>\nabla f(\vec{x}_k)</math> is the [[gradient vector]] of <math>f</math> at the point <math>\vec{x}^{(k)}</math> and <math>\alpha_k</math> is the second derivative of <math>f</math> along the gradient vector. Explicitly, if <math>\vec{v}^{(k)} = \nabla f\left(\vec{x}^{(k)}\right)</math>, we have:		where <math>\nabla f(\vec{x}_k)</math> is the [[gradient vector]] of <math>f</math> at the point <math>\vec{x}^{(k)}</math> and <math>\alpha_k</math> is the second derivative of <math>f</math> along the gradient vector. Explicitly, if <math>\vec{v}^{(k)} = \nabla f\left(\vec{x}^{(k)}\right)</math>, we have:

	<math>\alpha_k = \frac{\|\vec{v}^{(k)}\|^2}{\vec{v}^{(k)}^T\left(H(f)\left(\vec{x}^{(k)}\right)\right)\vec{v}^{(k)}}</math>		<math>\alpha_k = \frac{\|\vec{v}^{(k)}\|^2}{\left(\vec{v}^{(k)}\right)^T\left(H(f)\left(\vec{x}^{(k)}\right)\right)\vec{v}^{(k)}}</math>

	Here, <math>H(f)(\vec{x}^{(k)})</math> is the [[Hessian matrix]] of <math>f</math> at the point <math>\vec{x}^{(k)}</math>.		Here, <math>H(f)(\vec{x}^{(k)})</math> is the [[Hessian matrix]] of <math>f</math> at the point <math>\vec{x}^{(k)}</math>.

	The expression in the denominator can be justified based on the fact that [[Hessian matrix defines bilinear form that outputs second-order directional derivatives]].		The expression in the denominator can be justified based on the fact that [[Hessian matrix defines bilinear form that outputs second-order directional derivatives]].

Vipul at 22:31, 8 May 2016

2016-05-08T22:31:13Z

← Older revision		Revision as of 22:31, 8 May 2016
Line 9:		Line 9:
	Explicitly, the learning algorithm is:		Explicitly, the learning algorithm is:

	<math>\vec{x}_{k+1} = \vec{x}_k - \alpha_k \nabla f(\vec{x}_k)</math>		<math>\vec{x}^{(k+1)} = \vec{x}_k - \alpha_k \nabla f\left(\vec{x}^{(k)}\right)</math>

	where <math>\nabla f(\vec{x}_k)</math> is the [[gradient vector]] of <math>f</math> at the point <math>\vec{x}_k</math> and <math>\alpha_k</math> is the second derivative of <math>f</math> along the gradient vector. Explicitly, if <math>\vec{v}_k = \nabla f(\vec{x}_k)</math>, we have:		where <math>\nabla f(\vec{x}_k)</math> is the [[gradient vector]] of <math>f</math> at the point <math>\vec{x}^{(k)}</math> and <math>\alpha_k</math> is the second derivative of <math>f</math> along the gradient vector. Explicitly, if <math>\vec{v}^{(k)} = \nabla f\left(\vec{x}^{(k)}\right)</math>, we have:

	<math>\alpha_k = \frac{\|\vec{v}_k\|^2}{\vec{v}_k^T(H(f)(\vec{x}_k))\vec{v}_k}</math>		<math>\alpha_k = \frac{\|\vec{v}^{(k)}\|^2}{\vec{v}^{(k)}^T\left(H(f)\left(\vec{x}^{(k)}\right)\right)\vec{v}^{(k)}}</math>

	Here, <math>H(f)(\vec{x}_k)</math> is the [[Hessian matrix]] of <math>f</math> at the point <math>\vec{x}_k</math>.		Here, <math>H(f)(\vec{x}^{(k)})</math> is the [[Hessian matrix]] of <math>f</math> at the point <math>\vec{x}^{(k)}</math>.

	The expression in the denominator can be justified based on the fact that [[Hessian matrix defines bilinear form that outputs second-order directional derivatives]].		The expression in the denominator can be justified based on the fact that [[Hessian matrix defines bilinear form that outputs second-order directional derivatives]].

Vipul at 22:26, 8 May 2016

2016-05-08T22:26:23Z

← Older revision		Revision as of 22:26, 8 May 2016
Line 11:		Line 11:
	<math>\vec{x}_{k+1} = \vec{x}_k - \alpha_k \nabla f(\vec{x}_k)</math>		<math>\vec{x}_{k+1} = \vec{x}_k - \alpha_k \nabla f(\vec{x}_k)</math>

	where <math>\nabla f(\vec{x}_k)</math> is the [[gradient vector]] of <math>f</math> at the point <math>\vec{x}_k)</math> and <math>\alpha_k</math> is the second derivative of <math>f</math> along the gradient vector. Explicitly, if <math>\vec{v}_k = \nabla f(\vec{x}_k)</math>, we have:		where <math>\nabla f(\vec{x}_k)</math> is the [[gradient vector]] of <math>f</math> at the point <math>\vec{x}_k</math> and <math>\alpha_k</math> is the second derivative of <math>f</math> along the gradient vector. Explicitly, if <math>\vec{v}_k = \nabla f(\vec{x}_k)</math>, we have:

	<math>\alpha_k = \frac{\|\vec{v}_k\|^2}{\vec{v}_k^T(H(f)(\vec{x}_k))\vec{v}_k}</math>		<math>\alpha_k = \frac{\|\vec{v}_k\|^2}{\vec{v}_k^T(H(f)(\vec{x}_k))\vec{v}_k}</math>

Vipul: /* Definition */

2016-05-08T22:26:03Z

Definition

← Older revision		Revision as of 22:26, 8 May 2016
Line 11:		Line 11:
	<math>\vec{x}_{k+1} = \vec{x}_k - \alpha_k \nabla f(\vec{x}_k)</math>		<math>\vec{x}_{k+1} = \vec{x}_k - \alpha_k \nabla f(\vec{x}_k)</math>

	where <math>\nabla f(\vec{x}_k)</math> is the [[gradient vector]] of <math>f</math> at the point <math>\vec{x}_k)</math> and <math>\alpha_k</math> is the second derivative of <math>f</math> along the gradient ~~vecto~~. Explicitly, if <math>\vec{v}_k = \nabla f(\vec{x}_k)</math>, we have:		where <math>\nabla f(\vec{x}_k)</math> is the [[gradient vector]] of <math>f</math> at the point <math>\vec{x}_k)</math> and <math>\alpha_k</math> is the second derivative of <math>f</math> along the gradient vector. Explicitly, if <math>\vec{v}_k = \nabla f(\vec{x}_k)</math>, we have:

	<math>\alpha_k = \frac{\|\vec{v}_k\|^2}{\vec{v}_k^T(H(f)(\vec{x}_k))\vec{v}_k}</math>		<math>\alpha_k = \frac{\|\vec{v}_k\|^2}{\vec{v}_k^T(H(f)(\vec{x}_k))\vec{v}_k}</math>

Vipul: /* Learning algorithm */

2016-05-08T22:23:02Z

Learning algorithm

← Older revision		Revision as of 22:23, 8 May 2016
Line 11:		Line 11:
	<math>\vec{x}_{k+1} = \vec{x}_k - \alpha_k \nabla f(\vec{x}_k)</math>		<math>\vec{x}_{k+1} = \vec{x}_k - \alpha_k \nabla f(\vec{x}_k)</math>

	where <math>\nabla f(\vec{x}_k)</math> is the [[gradient vector]] of <math>f</math> at the point <math>\vec{x}_k)</math> and <math>\alpha_k</math> is the second derivative of <math>f</math> along the gradient ~~vector~~. Explicitly, if <math>\vec{v}_k = \nabla f(\vec{x}_k)</math>, we have:		where <math>\nabla f(\vec{x}_k)</math> is the [[gradient vector]] of <math>f</math> at the point <math>\vec{x}_k)</math> and <math>\alpha_k</math> is the second derivative of <math>f</math> along the gradient vecto. Explicitly, if <math>\vec{v}_k = \nabla f(\vec{x}_k)</math>, we have:

	<math>\alpha_k = \frac{\|\vec{v}_k\|^2}{\vec{v}_k^T(H(f)(\vec{x}_k))\vec{v}_k}</math>		<math>\alpha_k = \frac{\|\vec{v}_k\|^2}{\vec{v}_k^T(H(f)(\vec{x}_k))\vec{v}_k}</math>

	Here, <math>H(f)(\vec{x}_k)</math> is the [[Hessian matrix]] of <math>f</math> at the point <math>\vec{x}_k</math>.		Here, <math>H(f)(\vec{x}_k)</math> is the [[Hessian matrix]] of <math>f</math> at the point <math>\vec{x}_k</math>.

			The expression in the denominator can be justified based on the fact that [[Hessian matrix defines bilinear form that outputs second-order directional derivatives]].

Vipul at 16:43, 31 May 2014

2014-05-31T16:43:27Z

← Older revision		Revision as of 16:43, 31 May 2014
Line 11:		Line 11:
	<math>\vec{x}_{k+1} = \vec{x}_k - \alpha_k \nabla f(\vec{x}_k)</math>		<math>\vec{x}_{k+1} = \vec{x}_k - \alpha_k \nabla f(\vec{x}_k)</math>

	where <math>\alpha_k</math> is the second derivative of <math>f</math> along the gradient vector. ~~Epxlicitly~~, if <math>\vec{v}_k = \nabla f(\vec{x}_k)</math>, we have:		where <math>\nabla f(\vec{x}_k)</math> is the [[gradient vector]] of <math>f</math> at the point <math>\vec{x}_k)</math> and <math>\alpha_k</math> is the second derivative of <math>f</math> along the gradient vector. Explicitly, if <math>\vec{v}_k = \nabla f(\vec{x}_k)</math>, we have:

	<math>\alpha_k = \frac{\|\vec{v}_k\|^2}{\vec{v}_k^T(H(f)(\vec{x}_k))\vec{v}_k}</math>		<math>\alpha_k = \frac{\|\vec{v}_k\|^2}{\vec{v}_k^T(H(f)(\vec{x}_k))\vec{v}_k}</math>

			Here, <math>H(f)(\vec{x}_k)</math> is the [[Hessian matrix]] of <math>f</math> at the point <math>\vec{x}_k</math>.

Vipul at 16:42, 31 May 2014

2014-05-31T16:42:00Z

← Older revision		Revision as of 16:42, 31 May 2014
Line 4:		Line 4:

	By default, we are referring to gradient descent using ''one'' iteration of Newton's method, i.e., we stop Newton's method after one iteration. However, we can consider gradient descent using Newton's method where we use multiple iterations of Newton's method to determine the step size for gradient descent.		By default, we are referring to gradient descent using ''one'' iteration of Newton's method, i.e., we stop Newton's method after one iteration. However, we can consider gradient descent using Newton's method where we use multiple iterations of Newton's method to determine the step size for gradient descent.

			===Learning algorithm===

			Explicitly, the learning algorithm is:

			<math>\vec{x}_{k+1} = \vec{x}_k - \alpha_k \nabla f(\vec{x}_k)</math>

			where <math>\alpha_k</math> is the second derivative of <math>f</math> along the gradient vector. Epxlicitly, if <math>\vec{v}_k = \nabla f(\vec{x}_k)</math>, we have:

			<math>\alpha_k = \frac{\|\vec{v}_k\|^2}{\vec{v}_k^T(H(f)(\vec{x}_k))\vec{v}_k}</math>

Vipul at 02:30, 31 May 2014

2014-05-31T02:30:03Z

← Older revision		Revision as of 02:30, 31 May 2014
Line 2:		Line 2:

	'''Gradient descent using Newton's method''' is a variant of [[gradient descent]] where the step size along the gradient descent is determined using [[Newton's method for optimization of a function of one variable\|Newton's method]]. In other words, we move the same way that we would move if we were applying Newton's method to the function restricted to the line of the gradient vector through the point.		'''Gradient descent using Newton's method''' is a variant of [[gradient descent]] where the step size along the gradient descent is determined using [[Newton's method for optimization of a function of one variable\|Newton's method]]. In other words, we move the same way that we would move if we were applying Newton's method to the function restricted to the line of the gradient vector through the point.

			By default, we are referring to gradient descent using ''one'' iteration of Newton's method, i.e., we stop Newton's method after one iteration. However, we can consider gradient descent using Newton's method where we use multiple iterations of Newton's method to determine the step size for gradient descent.

Vipul: Created page with "==Definition== '''Gradient descent using Newton's method''' is a variant of gradient descent where the step size along the gradient descent is determined using Newton's..."

2014-05-31T02:28:49Z

Created page with "==Definition== '''Gradient descent using Newton's method''' is a variant of gradient descent where the step size along the gradient descent is determined using Newton's..."

New page

==Definition==

'''Gradient descent using Newton's method''' is a variant of [[gradient descent]] where the step size along the gradient descent is determined using [[Newton's method for optimization of a function of one variable|Newton's method]]. In other words, we move the same way that we would move if we were applying Newton's method to the function restricted to the line of the gradient vector through the point.