- This is an implementation of the widely used and very popular BFGS method. Here we are using an inexact line search accepting steps that satisfy the weak Wolfe conditions. The line search is thoroughly described here.
- Although BFGS is usually only used for smooth optimization, it turns out that it works quite well for small- and medium-scale non-smooth optimization as well. See this paper for details.
- The BFGS method requires O(n^2) work in each iteration making it infeasible to use on problems with the number of variables exceeding a few thousands. This, of course, depends on your machine and your patience.
- The algorithm was invented in 1970 by C. G. Broyden, R. Fletcher, D. Goldfarb and D. Shanno.
- The algorithm is described in Nocedal and Wright's book: Numerical Optimization. See also the references.
- Download the m-file for the BFGS method here.
- Do not forget the necessary subroutines. Download them here.