Why do quasi newton methods such as DFP and BFGS have poor performance on ill-conditionned problem, even if quadratic

Question

I've been reading in the litterature that quasi newton methods such as DFP and BFGS have poor performance on ill-conditionned problems, but I don't understand the reason of it. I have been trying to use these methods on a quadratic problem that is ill-conditionned and it doesn't converge in p+1 iterations (it is one of quasi newton methods properties for quadratic problems), but a little more. Why is that ? Thank you for your help.

Answer 1

Ill-conditioning is a general problem for the optimization algorithms. Since quasi-Newton methods are based on the Newton method, let´s consider its properties first. Basically there are two main aspects with ill-conditioning:

1. it leads to the numerical instability (e.g., roundoff errors) that are accumulated by the algorithm
2. it slows down the convergence rate due to the stretched shape of the resulting contours of the Hessian

Standard Newton method also involves the operation of the inverse of the Hessian which in case of large condition number results in the corresponding small eigenvalues blowing up leading to the numerical instability.

Quasi-Newton methods have the same issues. However, due to the fact that they iteratively approximate the inverse Hessian, they are more robust in handling roundoff errors and also may be a bit faster in convergence but it doesn´t eliminate the problem completely hence they have poor performance.