Logistic log-loss function of one variable - Revision history

Vipul: /* Optimization methods */

2014-09-28T02:02:29Z

Optimization methods

← Older revision		Revision as of 02:02, 28 September 2014
Line 114:		Line 114:

	{\| class="sortable" border="1"		{\| class="sortable" border="1"
	! Method !! Domain of convergence !! Order of convergence !! Convergence rate (general case) !! Convergence rate (special cases)		! Method !! Domain of convergence !! Order of convergence (general case) !! Convergence rate (general case) !! Convergence rate (special cases)
	\|-		\|-
	\| [[Gradient descent with constant learning rate for a logistic log-loss function of one variable]], with learning rate <math>\alpha</math> \|\| If <math>\alpha < \frac{1}{f''(x^)} = \frac{1}{p(1 - p)}</math> with <math>x^</math> the point of minimum, convergence occurs from any starting point, but can be quite slow. Since we don't know <math>x^</math>, setting <math>\alpha = 4</math> is a safe choice. \|\| Linear ~~(unless <math>\alpha = \frac{1}{f''(x)}</math>, in which case~~ convergence ~~is quadratic.~~ \|\| <math>\|1 - \alpha f''(x^*)\| = \|1 - \alpha p(1- p)\|</math><br>If we set <math>\alpha = 4</math>, we get <math>\|1 - 4p(1 - p)\|</math>. \|\| Case <math>\alpha = \frac{1}{p(1 - p)}</math> and <math>p \ne \frac{1}{2}</math>: quadratic convergence with convergence rate <math>\left\| \frac{1}{2} - p \right\|</math>.<br>Case that <math>p = \frac{1}{2}, \alpha = 4</math>: cubic convergence with convergence rate <math>\frac{1}{6} \left\| \frac{g'''(0)}{g'(0)} \right\| = \frac{1}{12}</math>		\| [[Gradient descent with constant learning rate for a logistic log-loss function of one variable]], with learning rate <math>\alpha</math> \|\| If <math>\alpha < \frac{1}{f''(x^)} = \frac{1}{p(1 - p)}</math> with <math>x^</math> the point of minimum, convergence occurs from any starting point, but can be quite slow. Since we don't know <math>x^</math>, setting <math>\alpha = 4</math> is a safe choice. \|\| Linear convergence\|\| <math>\|1 - \alpha f''(x^)\| = \|1 - \alpha p(1- p)\|</math><br>If we set <math>\alpha = 4</math>, we get <math>\|1 - 4p(1 - p)\|</math>. \|\| Case <math>\alpha = \frac{1}{p(1 - p)}</math> and <math>p \ne \frac{1}{2}</math>: quadratic convergence with convergence rate <math>\left\| \frac{1}{2} - p \right\|</math>.<br>Case that <math>p = \frac{1}{2}, \alpha = 4</math>: cubic convergence with convergence rate <math>\frac{1}{6} \left\| \frac{g'''(0)}{g'(0)} \right\| = \frac{1}{12}</math>
	\|-		\|-
	\| [[Newton's method for optimization of a logistic log-loss function of one variable]] \|\| ''Not'' all reals. However, the domain includes the interval between 0 and <math>x^*</math>, and convergence is monotone on the interval. \|\| Quadratic convergence. \|\| <math>\left\|\frac{1}{2} - p \right\|</math> \|\| Case that <math>p = \frac{1}{2}</math>: cubic convergence with convergence rate <math>\frac{1}{3} \left\| \frac{g'''(0)}{g'(0)} \right\| = \frac{1}{6}</math>		\| [[Newton's method for optimization of a logistic log-loss function of one variable]] \|\| ''Not'' all reals. However, the domain includes the interval between 0 and <math>x^*</math>, and convergence is monotone on the interval. \|\| Quadratic convergence. \|\| <math>\left\|\frac{1}{2} - p \right\|</math> \|\| Case that <math>p = \frac{1}{2}</math>: cubic convergence with convergence rate <math>\frac{1}{3} \left\| \frac{g'''(0)}{g'(0)} \right\| = \frac{1}{6}</math>
	\|}		\|}

Vipul: /* Optimization methods */

2014-09-28T01:43:22Z

Optimization methods

← Older revision		Revision as of 01:43, 28 September 2014
Line 116:		Line 116:
	! Method !! Domain of convergence !! Order of convergence !! Convergence rate (general case) !! Convergence rate (special cases)		! Method !! Domain of convergence !! Order of convergence !! Convergence rate (general case) !! Convergence rate (special cases)
	\|-		\|-
	\| [[Gradient descent with constant learning rate for a logistic log-loss function of one variable]], with learning rate <math>\alpha</math> \|\| If <math>\alpha < \frac{1}{f''(x^)} = \frac{1}{p(1 - p)}</math> with <math>x^</math> the point of minimum, convergence occurs from any starting point, but can be quite slow. Since we don't know <math>x^</math>, setting <math>\alpha = 4</math> is a safe choice. \|\| Linear (unless <math>\alpha = \frac{1}{f''(x)}</math>, in which case convergence is quadratic. \|\| <math>\|1 - \alpha f''(x^)\| = \|1 - \alpha p(1- p)</math> if <~~math>\alpha \ne \frac{1}{f''(x^)}</math~~>. If we set <math>\alpha = 4</math>, we get <math>\|1 - 4p(1 - p)\|</math>. \|\| Case <math>\alpha = \frac{1}{p(1 - p)}</math> and <math>p \ne \frac{1}{2}</math>: quadratic convergence with convergence rate <math>\left\| \frac{1}{2} - p \right\|</math>.<br>Case that <math>p = \frac{1}{2}, \alpha = 4</math>: cubic convergence with convergence rate <math>\frac{1}{6} \left\| \frac{g'''(0)}{g'(0)} \right\| = \frac{1}{12}</math>		\| [[Gradient descent with constant learning rate for a logistic log-loss function of one variable]], with learning rate <math>\alpha</math> \|\| If <math>\alpha < \frac{1}{f''(x^)} = \frac{1}{p(1 - p)}</math> with <math>x^</math> the point of minimum, convergence occurs from any starting point, but can be quite slow. Since we don't know <math>x^</math>, setting <math>\alpha = 4</math> is a safe choice. \|\| Linear (unless <math>\alpha = \frac{1}{f''(x)}</math>, in which case convergence is quadratic. \|\| <math>\|1 - \alpha f''(x^*)\| = \|1 - \alpha p(1- p)\|</math><br>If we set <math>\alpha = 4</math>, we get <math>\|1 - 4p(1 - p)\|</math>. \|\| Case <math>\alpha = \frac{1}{p(1 - p)}</math> and <math>p \ne \frac{1}{2}</math>: quadratic convergence with convergence rate <math>\left\| \frac{1}{2} - p \right\|</math>.<br>Case that <math>p = \frac{1}{2}, \alpha = 4</math>: cubic convergence with convergence rate <math>\frac{1}{6} \left\| \frac{g'''(0)}{g'(0)} \right\| = \frac{1}{12}</math>
	\|-		\|-
	\| [[Newton's method for optimization of a logistic log-loss function of one variable]] \|\| ''Not'' all reals. However, the domain includes the interval between 0 and <math>x^*</math>, and convergence is monotone on the interval. \|\| Quadratic convergence. \|\| <math>\left\|\frac{1}{2} - p \right\|</math> \|\| Case that <math>p = \frac{1}{2}</math>: cubic convergence with convergence rate <math>\frac{1}{3} \left\| \frac{g'''(0)}{g'(0)} \right\| = \frac{1}{6}</math>		\| [[Newton's method for optimization of a logistic log-loss function of one variable]] \|\| ''Not'' all reals. However, the domain includes the interval between 0 and <math>x^*</math>, and convergence is monotone on the interval. \|\| Quadratic convergence. \|\| <math>\left\|\frac{1}{2} - p \right\|</math> \|\| Case that <math>p = \frac{1}{2}</math>: cubic convergence with convergence rate <math>\frac{1}{3} \left\| \frac{g'''(0)}{g'(0)} \right\| = \frac{1}{6}</math>
	\|}		\|}

Vipul: /* Optimization methods */

2014-09-28T01:41:40Z

Optimization methods

← Older revision		Revision as of 01:41, 28 September 2014
Line 116:		Line 116:
	! Method !! Domain of convergence !! Order of convergence !! Convergence rate (general case) !! Convergence rate (special cases)		! Method !! Domain of convergence !! Order of convergence !! Convergence rate (general case) !! Convergence rate (special cases)
	\|-		\|-
	\| [[Gradient descent with constant learning rate for a logistic log-loss function of one variable]], with learning rate <math>\alpha</math> \|\| If <math>\alpha < \frac{1}{f''(x^)} = \frac{1}{p(1 - p)}</math> with <math>x^</math> the point of minimum, convergence occurs from any starting point, but can be quite slow. Since we don't know <math>x^</math>, setting <math>\alpha = 4</math> is a safe choice. \|\| Linear (unless <math>\alpha = \frac{1}{f''(x)}</math>, in which case convergence is quadratic. \|\| <math>\|1 - \alpha f''(x^)\| = \|1 - \alpha p(1- p)</math> if <math>\alpha \ne \frac{1}{f''(x^)}</math>. If we set <math>\alpha = 4</math>, we get <math>\|1 - 4p(1 - p)\|</math>. \|\| Case <math>\alpha = \frac{1}{p(1 - p)}</math> and <math>p \ne \frac{1}{2}</math>: quadratic convergence with convergence rate <math>\left\| \frac{1}{2} - p \right\|</math>.<br>Case that <math>p = \frac{1}{2}, \alpha = 4</math>: <math>\frac{1}{6} \left\| \frac{g'''(0)}{g'(0)} \right\| = \frac{1}{12}</math>		\| [[Gradient descent with constant learning rate for a logistic log-loss function of one variable]], with learning rate <math>\alpha</math> \|\| If <math>\alpha < \frac{1}{f''(x^)} = \frac{1}{p(1 - p)}</math> with <math>x^</math> the point of minimum, convergence occurs from any starting point, but can be quite slow. Since we don't know <math>x^</math>, setting <math>\alpha = 4</math> is a safe choice. \|\| Linear (unless <math>\alpha = \frac{1}{f''(x)}</math>, in which case convergence is quadratic. \|\| <math>\|1 - \alpha f''(x^)\| = \|1 - \alpha p(1- p)</math> if <math>\alpha \ne \frac{1}{f''(x^)}</math>. If we set <math>\alpha = 4</math>, we get <math>\|1 - 4p(1 - p)\|</math>. \|\| Case <math>\alpha = \frac{1}{p(1 - p)}</math> and <math>p \ne \frac{1}{2}</math>: quadratic convergence with convergence rate <math>\left\| \frac{1}{2} - p \right\|</math>.<br>Case that <math>p = \frac{1}{2}, \alpha = 4</math>: cubic convergence with convergence rate <math>\frac{1}{6} \left\| \frac{g'''(0)}{g'(0)} \right\| = \frac{1}{12}</math>
	\|-		\|-
	\| [[Newton's method for optimization of a logistic log-loss function of one variable]] \|\| ''Not'' all reals. However, the domain includes the interval between 0 and <math>x^*</math>, and convergence is monotone on the interval. \|\| Quadratic convergence. \|\| <math>\left\|\frac{1}{2} - p \right\|</math> \|\| Case that <math>p = \frac{1}{2}</math>: cubic convergence with convergence rate <math>\frac{1}{3} \left\| \frac{g'''(0)}{g'(0)} \right\| = \frac{1}{6}</math>		\| [[Newton's method for optimization of a logistic log-loss function of one variable]] \|\| ''Not'' all reals. However, the domain includes the interval between 0 and <math>x^*</math>, and convergence is monotone on the interval. \|\| Quadratic convergence. \|\| <math>\left\|\frac{1}{2} - p \right\|</math> \|\| Case that <math>p = \frac{1}{2}</math>: cubic convergence with convergence rate <math>\frac{1}{3} \left\| \frac{g'''(0)}{g'(0)} \right\| = \frac{1}{6}</math>
	\|}		\|}

Vipul: /* Optimization methods */

2014-09-28T01:41:16Z

Optimization methods

← Older revision		Revision as of 01:41, 28 September 2014
Line 116:		Line 116:
	! Method !! Domain of convergence !! Order of convergence !! Convergence rate (general case) !! Convergence rate (special cases)		! Method !! Domain of convergence !! Order of convergence !! Convergence rate (general case) !! Convergence rate (special cases)
	\|-		\|-
	\| [[Gradient descent with constant learning rate for a logistic log-loss function of one variable]], with learning rate <math>\alpha</math> \|\| If <math>\alpha < \frac{1}{f''(x^)} = \frac{1}{p(1 - p)}</math> with <math>x^</math> the point of minimum, convergence occurs from any starting point, but can be quite slow. Since we don't know <math>x^</math>, setting <math>\alpha = 4</math> is a safe choice. \|\| Linear (unless <math>\alpha = \frac{1}{f''(x)}</math>, in which case convergence is quadratic. \|\| <math>\|1 - \alpha f''(x^)\|</math> if <math>\alpha \ne \frac{1}{f''(x^)}</math>. If we set <math>\alpha = 4</math>, we get <math>\|1 - 4p(1 - p)\|</math>. \|\| Case <math>\alpha = \frac{1}{p(1 - p)}</math> and <math>p \ne \frac{1}{2}</math>: quadratic convergence with convergence rate <math>\|1 - 2p\|</math>.<br>Case that <math>p = \frac{1}{2}, \alpha = 4</math>: <math>\frac{1}{6} \left\| \frac{g'''(0)}{g'(0)} \right\| = \frac{1}{12}</math>		\| [[Gradient descent with constant learning rate for a logistic log-loss function of one variable]], with learning rate <math>\alpha</math> \|\| If <math>\alpha < \frac{1}{f''(x^)} = \frac{1}{p(1 - p)}</math> with <math>x^</math> the point of minimum, convergence occurs from any starting point, but can be quite slow. Since we don't know <math>x^</math>, setting <math>\alpha = 4</math> is a safe choice. \|\| Linear (unless <math>\alpha = \frac{1}{f''(x)}</math>, in which case convergence is quadratic. \|\| <math>\|1 - \alpha f''(x^)\| = \|1 - \alpha p(1- p)</math> if <math>\alpha \ne \frac{1}{f''(x^)}</math>. If we set <math>\alpha = 4</math>, we get <math>\|1 - 4p(1 - p)\|</math>. \|\| Case <math>\alpha = \frac{1}{p(1 - p)}</math> and <math>p \ne \frac{1}{2}</math>: quadratic convergence with convergence rate <math>\left\| \frac{1}{2} - p \right\|</math>.<br>Case that <math>p = \frac{1}{2}, \alpha = 4</math>: <math>\frac{1}{6} \left\| \frac{g'''(0)}{g'(0)} \right\| = \frac{1}{12}</math>
	\|-		\|-
	\| [[Newton's method for optimization of a logistic log-loss function of one variable]] \|\| ''Not'' all reals. However, the domain includes the interval between 0 and <math>x^*</math>, and convergence is monotone on the interval. \|\| Quadratic convergence. \|\| <math>\left\|\frac{1}{2} - p \right\|</math> \|\| Case that <math>p = \frac{1}{2}</math>: cubic convergence with convergence rate <math>\frac{1}{3} \left\| \frac{g'''(0)}{g'(0)} \right\| = \frac{1}{6}</math>		\| [[Newton's method for optimization of a logistic log-loss function of one variable]] \|\| ''Not'' all reals. However, the domain includes the interval between 0 and <math>x^*</math>, and convergence is monotone on the interval. \|\| Quadratic convergence. \|\| <math>\left\|\frac{1}{2} - p \right\|</math> \|\| Case that <math>p = \frac{1}{2}</math>: cubic convergence with convergence rate <math>\frac{1}{3} \left\| \frac{g'''(0)}{g'(0)} \right\| = \frac{1}{6}</math>
	\|}		\|}

Vipul: /* Optimization methods */

2014-09-28T01:39:48Z

Optimization methods

← Older revision		Revision as of 01:39, 28 September 2014
Line 116:		Line 116:
	! Method !! Domain of convergence !! Order of convergence !! Convergence rate (general case) !! Convergence rate (special cases)		! Method !! Domain of convergence !! Order of convergence !! Convergence rate (general case) !! Convergence rate (special cases)
	\|-		\|-
	\| [[Gradient descent with constant learning rate for a logistic log-loss function of one variable]], with learning rate <math>\alpha</math> \|\| If <math>\alpha < \frac{1}{f''(x^)} = \frac{1}{p(1 - p)}</math> with <math>x^</math> the point of minimum, convergence occurs from any starting point, but can be quite slow. Since we don't know <math>x^</math>, setting <math>\alpha = 4</math> is a safe choice. \|\| Linear (unless <math>\alpha = \frac{1}{f''(x)}</math>, in which case convergence is quadratic. \|\| <math>\|1 - \alpha f''(x^)\|</math> if <math>\alpha \ne \frac{1}{f''(x^)}</math>. If we set <math>\alpha = 4</math>, we get <math>\|1 - 4p(1 - p)\|</math>. \|\| Case <math>\alpha = \frac{1}{p(1 - p)}<math> and <math>p \ne \frac{1}{2}</math>: quadratic convergence with convergence rate <math>\|1 - 2p\|</math>.<br>Case that <math>p = \frac{1}{2}, \alpha = 4</math>: <math>\frac{1}{6} \left\| \frac{g'''(0)}{g'(0)} \right\| = \frac{1}{12}</math>		\| [[Gradient descent with constant learning rate for a logistic log-loss function of one variable]], with learning rate <math>\alpha</math> \|\| If <math>\alpha < \frac{1}{f''(x^)} = \frac{1}{p(1 - p)}</math> with <math>x^</math> the point of minimum, convergence occurs from any starting point, but can be quite slow. Since we don't know <math>x^</math>, setting <math>\alpha = 4</math> is a safe choice. \|\| Linear (unless <math>\alpha = \frac{1}{f''(x)}</math>, in which case convergence is quadratic. \|\| <math>\|1 - \alpha f''(x^)\|</math> if <math>\alpha \ne \frac{1}{f''(x^)}</math>. If we set <math>\alpha = 4</math>, we get <math>\|1 - 4p(1 - p)\|</math>. \|\| Case <math>\alpha = \frac{1}{p(1 - p)}</math> and <math>p \ne \frac{1}{2}</math>: quadratic convergence with convergence rate <math>\|1 - 2p\|</math>.<br>Case that <math>p = \frac{1}{2}, \alpha = 4</math>: <math>\frac{1}{6} \left\| \frac{g'''(0)}{g'(0)} \right\| = \frac{1}{12}</math>
	\|-		\|-
	\| [[Newton's method for optimization of a logistic log-loss function of one variable]] \|\| ''Not'' all reals. However, the domain includes the interval between 0 and <math>x^*</math>, and convergence is monotone on the interval. \|\| Quadratic convergence. \|\| <math>\left\|\frac{1}{2} - p \right\|</math> \|\| Case that <math>p = \frac{1}{2}</math>: cubic convergence with convergence rate <math>\frac{1}{3} \left\| \frac{g'''(0)}{g'(0)} \right\| = \frac{1}{6}</math>		\| [[Newton's method for optimization of a logistic log-loss function of one variable]] \|\| ''Not'' all reals. However, the domain includes the interval between 0 and <math>x^*</math>, and convergence is monotone on the interval. \|\| Quadratic convergence. \|\| <math>\left\|\frac{1}{2} - p \right\|</math> \|\| Case that <math>p = \frac{1}{2}</math>: cubic convergence with convergence rate <math>\frac{1}{3} \left\| \frac{g'''(0)}{g'(0)} \right\| = \frac{1}{6}</math>
	\|}		\|}

Vipul: /* Optimization methods */

2014-09-28T01:36:18Z

Optimization methods

← Older revision		Revision as of 01:36, 28 September 2014
Line 118:		Line 118:
	\| [[Gradient descent with constant learning rate for a logistic log-loss function of one variable]], with learning rate <math>\alpha</math> \|\| If <math>\alpha < \frac{1}{f''(x^)} = \frac{1}{p(1 - p)}</math> with <math>x^</math> the point of minimum, convergence occurs from any starting point, but can be quite slow. Since we don't know <math>x^</math>, setting <math>\alpha = 4</math> is a safe choice. \|\| Linear (unless <math>\alpha = \frac{1}{f''(x)}</math>, in which case convergence is quadratic. \|\| <math>\|1 - \alpha f''(x^)\|</math> if <math>\alpha \ne \frac{1}{f''(x^)}</math>. If we set <math>\alpha = 4</math>, we get <math>\|1 - 4p(1 - p)\|</math>. \|\| Case <math>\alpha = \frac{1}{p(1 - p)}<math> and <math>p \ne \frac{1}{2}</math>: quadratic convergence with convergence rate <math>\|1 - 2p\|</math>.<br>Case that <math>p = \frac{1}{2}, \alpha = 4</math>: <math>\frac{1}{6} \left\| \frac{g'''(0)}{g'(0)} \right\| = \frac{1}{12}</math>		\| [[Gradient descent with constant learning rate for a logistic log-loss function of one variable]], with learning rate <math>\alpha</math> \|\| If <math>\alpha < \frac{1}{f''(x^)} = \frac{1}{p(1 - p)}</math> with <math>x^</math> the point of minimum, convergence occurs from any starting point, but can be quite slow. Since we don't know <math>x^</math>, setting <math>\alpha = 4</math> is a safe choice. \|\| Linear (unless <math>\alpha = \frac{1}{f''(x)}</math>, in which case convergence is quadratic. \|\| <math>\|1 - \alpha f''(x^)\|</math> if <math>\alpha \ne \frac{1}{f''(x^)}</math>. If we set <math>\alpha = 4</math>, we get <math>\|1 - 4p(1 - p)\|</math>. \|\| Case <math>\alpha = \frac{1}{p(1 - p)}<math> and <math>p \ne \frac{1}{2}</math>: quadratic convergence with convergence rate <math>\|1 - 2p\|</math>.<br>Case that <math>p = \frac{1}{2}, \alpha = 4</math>: <math>\frac{1}{6} \left\| \frac{g'''(0)}{g'(0)} \right\| = \frac{1}{12}</math>
	\|-		\|-
	\| [[Newton's method for optimization of a logistic log-loss function of one variable]] \|\| ''Not'' all reals. However, the domain includes the interval between 0 and <math>x^*</math>, and convergence is monotone on the interval. \|\| Quadratic convergence. \|\| <math>\left\|\frac{1}{2} - p \right\|</math> \|\| Case that <math>p = \frac{1}{2]</math>: cubic convergence with convergence rate <math>\frac{1}{3} \left\| \frac{g'''(0)}{g'(0)} \right\| = \frac{1}{6}</math>		\| [[Newton's method for optimization of a logistic log-loss function of one variable]] \|\| ''Not'' all reals. However, the domain includes the interval between 0 and <math>x^*</math>, and convergence is monotone on the interval. \|\| Quadratic convergence. \|\| <math>\left\|\frac{1}{2} - p \right\|</math> \|\| Case that <math>p = \frac{1}{2}</math>: cubic convergence with convergence rate <math>\frac{1}{3} \left\| \frac{g'''(0)}{g'(0)} \right\| = \frac{1}{6}</math>
	\|}		\|}

Vipul: /* Optimization methods */

2014-09-28T01:36:04Z

Optimization methods

← Older revision		Revision as of 01:36, 28 September 2014
Line 114:		Line 114:

	{\| class="sortable" border="1"		{\| class="sortable" border="1"
	! Method !! Domain of convergence !! Order of convergence !! Convergence rate		! Method !! Domain of convergence !! Order of convergence !! Convergence rate (general case) !! Convergence rate (special cases)
	\|-		\|-
	\| [[Gradient descent with constant learning rate for a logistic log-loss function of one variable]], with learning rate <math>\alpha</math> \|\| If <math>\alpha < \frac{1}{f''(x^)} = \frac{1}{p(1 - p)}</math> with <math>x^</math> the point of minimum, convergence occurs from any starting point, but can be quite slow. Since we don't know <math>x^</math>, setting <math>\alpha = 4</math> is a safe choice. \|\| Linear (unless <math>\alpha = \frac{1}{f''(x)}</math>, in which case convergence is quadratic. \|\| <math>\|1 - \alpha f''(x^)\|</math> if <math>\alpha \ne \frac{1}{f''(x^)}</math>. If we set <math>\alpha = 4</math>, we get <math>\|1 - 4p(1 - p)\|</math>.		\| [[Gradient descent with constant learning rate for a logistic log-loss function of one variable]], with learning rate <math>\alpha</math> \|\| If <math>\alpha < \frac{1}{f''(x^)} = \frac{1}{p(1 - p)}</math> with <math>x^</math> the point of minimum, convergence occurs from any starting point, but can be quite slow. Since we don't know <math>x^</math>, setting <math>\alpha = 4</math> is a safe choice. \|\| Linear (unless <math>\alpha = \frac{1}{f''(x)}</math>, in which case convergence is quadratic. \|\| <math>\|1 - \alpha f''(x^)\|</math> if <math>\alpha \ne \frac{1}{f''(x^)}</math>. If we set <math>\alpha = 4</math>, we get <math>\|1 - 4p(1 - p)\|</math>. \|\| Case <math>\alpha = \frac{1}{p(1 - p)}<math> and <math>p \ne \frac{1}{2}</math>: quadratic convergence with convergence rate <math>\|1 - 2p\|</math>.<br>Case that <math>p = \frac{1}{2}, \alpha = 4</math>: <math>\frac{1}{6} \left\| \frac{g'''(0)}{g'(0)} \right\| = \frac{1}{12}</math>
	\|-		\|-
	\| [[Newton's method for optimization of a logistic log-loss function of one variable]] \|\| ''Not'' all reals. However, the domain includes the interval between 0 and <math>x^*</math>, and convergence is monotone on the interval. \|\| Quadratic convergence. \|\| <math>\left\|\frac{1}{2} - p \right\|</math>		\| [[Newton's method for optimization of a logistic log-loss function of one variable]] \|\| ''Not'' all reals. However, the domain includes the interval between 0 and <math>x^*</math>, and convergence is monotone on the interval. \|\| Quadratic convergence. \|\| <math>\left\|\frac{1}{2} - p \right\|</math> \|\| Case that <math>p = \frac{1}{2]</math>: cubic convergence with convergence rate <math>\frac{1}{3} \left\| \frac{g'''(0)}{g'(0)} \right\| = \frac{1}{6}</math>
	\|}		\|}

Vipul: /* Optimization methods */

2014-09-28T01:09:10Z

Optimization methods

← Older revision		Revision as of 01:09, 28 September 2014
Line 111:		Line 111:
	==Optimization methods==		==Optimization methods==

	* [[Gradient descent with constant learning rate for a logistic log-loss function of one variable]]		We will denote the point of minimum, <math>g^{-1}(p)</math>, as <math>x^*</math>.
	* [[Newton's method for optimization of a logistic log-loss function of one variable]]
			{\| class="sortable" border="1"
			! Method !! Domain of convergence !! Order of convergence !! Convergence rate
			\|-
			\| [[Gradient descent with constant learning rate for a logistic log-loss function of one variable]], with learning rate <math>\alpha</math> \|\| If <math>\alpha < \frac{1}{f''(x^)} = \frac{1}{p(1 - p)}</math> with <math>x^</math> the point of minimum, convergence occurs from any starting point, but can be quite slow. Since we don't know <math>x^</math>, setting <math>\alpha = 4</math> is a safe choice. \|\| Linear (unless <math>\alpha = \frac{1}{f''(x)}</math>, in which case convergence is quadratic. \|\| <math>\|1 - \alpha f''(x^)\|</math> if <math>\alpha \ne \frac{1}{f''(x^)}</math>. If we set <math>\alpha = 4</math>, we get <math>\|1 - 4p(1 - p)\|</math>.
			\|-
			\| [[Newton's method for optimization of a logistic log-loss function of one variable]] \|\| ''Not'' all reals. However, the domain includes the interval between 0 and <math>x^*</math>, and convergence is monotone on the interval. \|\| Quadratic convergence. \|\| <math>\left\|\frac{1}{2} - p \right\|</math>
			\|}

Vipul at 05:04, 22 September 2014

2014-09-22T05:04:22Z

← Older revision		Revision as of 05:04, 22 September 2014
Line 108:		Line 108:

	This is always positive, so the function is a convex function and its graph is concave up everywhere. In particular, there are no points of inflection.		This is always positive, so the function is a convex function and its graph is concave up everywhere. In particular, there are no points of inflection.

			==Optimization methods==

			* [[Gradient descent with constant learning rate for a logistic log-loss function of one variable]]
			* [[Newton's method for optimization of a logistic log-loss function of one variable]]

Vipul: /* Key data */

2014-09-22T04:50:37Z

Key data

← Older revision		Revision as of 04:50, 22 September 2014
Line 36:		Line 36:
	\| [[second derivative]] \|\| <math>g(x)(1 - g(x)) = g(x)g(-x)</math>		\| [[second derivative]] \|\| <math>g(x)(1 - g(x)) = g(x)g(-x)</math>
	\|-		\|-
	\| [[third derivative]] \|\| <math>g(x)g(-x)(g(x) - g(-x)) = g(x)(1 - g(x))(1 - 2g(x))</math>		\| [[third derivative]] \|\| <math>g(x)g(-x)(g(-x) - g(x)) = g(x)(1 - g(x))(1 - 2g(x))</math>
	\|}		\|}