Add notes on parabolic interpolation

_code/pset_08/py/lagrange_poly.py

@@ -26,5 +26,6 @@ print(sp.latex(sol))
 q = (x_2 - x_3) * (y_2 - y_1)
 r = (x_2 - x_1) * (y_2 - y_3)
 alt = x_2 - ((x_2 - x_3) * q - (x_2 - x_1) * r) / 2 / (q - r)
+print(sp.latex(alt))
 
 print(sp.simplify(sol - alt))

_notes/optimization.md

@@ -2,6 +2,49 @@
 title: Optimization Algorithms
 ---
 
+## Parabolic Interpolation
+
+A task that comes up frequently in optimization problems is guessing a function minimum based on a
+parabola fit between three points. So I'll derive this technique here.
+
+First, we need a parabola that goes through three points, say $$(x_1, y_1)$$, $$(x_1, y_1)$$, and
+$$(x_3, y_3)$$. In the interest of generality, I'll construct it using Lagrange polynomials.
+
+$$
+p(x) = \frac{y_1 (x - x_2) (x - x_3)}{(x_1 - x_2) (x_1 - x_3)}
+     + \frac{y_2 (x - x_1) (x - x_3)}{(x_2 - x_1) (x_2 - x_3)}
+     + \frac{y_3 (x - x_1) (x - x_2)}{(x_3 - x_1) (x_3 - x_2)}
+$$
+
+Differentiating, we find that the derivative is equal to the following.
+
+$$
+\begin{aligned}
+p'(x) &=
+\frac{y_{1} \left(x - x_{2}\right)}{\left(x_{1} - x_{2}\right) \left(x_{1} - x_{3}\right)}
++ \frac{y_{1} \left(x - x_{3}\right)}{\left(x_{1} - x_{2}\right) \left(x_{1} - x_{3}\right)} \\
+&+ \frac{y_{2} \left(x - x_{1}\right)}{\left(- x_{1} + x_{2}\right) \left(x_{2} - x_{3}\right)}
++ \frac{y_{2} \left(x - x_{3}\right)}{\left(- x_{1} + x_{2}\right) \left(x_{2} - x_{3}\right)} \\
+&+ \frac{y_{3} \left(x - x_{1}\right)}{\left(- x_{1} + x_{3}\right) \left(- x_{2} + x_{3}\right)}
++ \frac{y_{3} \left(x - x_{2}\right)}{\left(- x_{1} + x_{3}\right) \left(- x_{2} + x_{3}\right)}
+\end{aligned}
+$$
+
+Setting this to zero, we find that the solution is surprisingly simple.
+
+$$
+x_\text{min} = \frac{1}{2} \cdot \frac{x_1^2 (y_2 - y_3) + x_2^2 (y_3 - y_1) + x_3^2 (y_{1} - y_2)}
+{x_1 (y_2 -  y_3) + x_2 (y_3 - y_1) + x_3 (y_1 - y_2)}
+$$
+
+This can also be factored so that it only involves differences of coordinates and function values.
+
+$$
+x_\text{min} =
+x_{2} - \frac{1}{2} \cdot
+\frac{(x_2 - x_3)^2 (y_2 - y_1) - (x_2 - x_1)^2 (y_2 - y_3)}{(x_2 - x_3) (y_2 - y_1) - (x_2 - x_1) (y_2 - y_3)}
+$$
+
 ## Nelder-Mead
 
 TODO: Demonstrate that the Numerical Recipes formulation is equivalent to the standard formulation.