I am trying to understand the convergence analysis/derivation of the momentum algorithm, or the stochastic heavy ball algorithm, using the regret bound analysis from different research papers.
- https://ieeexplore.ieee.org/document/7330562 - Page3
- https://www.mdpi.com/2504-3110/6/12/709 - Page6
- http://arxiv.org/abs/1707.01647 - Page4
In the derivation, there is the following simplification, which I do not understand at all
The term $\left|\boldsymbol{\theta}{0} + \boldsymbol{p}{0} - \boldsymbol{\theta}^* \right|^2 - \left|\boldsymbol{\theta}{T+1} + \boldsymbol{p}{T+1} - \boldsymbol{\theta}^\right|^2$ is simplified to $\leq \left| \boldsymbol{\theta}_{0} - \boldsymbol{\theta}^ \right|^2$ directly. I am sure that there is certainly some assumption or some intermediate steps involved and the reader is supposed to know that. Could you please help me understand.
My understanding:
