On Preconditioning Polyak Step-Size and its Variants
The family of Stochastic Gradient Methods with Polyak Step-size offers an update rule that alleviates the need of fine-tuning the learning rate of an optimizer. Recent work  has been proposed to introduce a slack variable, which makes these methods applicable outside of the interpolation regime. In this paper, we combine preconditioning and slack in an updated optimization algorithm to show its performance on badly scaled and/or ill-conditioned datasets. We use Hutchinson’s method to obtain an estimate of a Hessian which is used as the preconditioner.
F. Abdukhakimov, "On Preconditioning Polyak Step-Size and its Variants", M.S. Thesis, Machine Learning, MBZUAI, Abu Dhabi, UAE, 2023.