Effective MM-Algorithms for Statistics and Machine Learning
As an optimization method, majorization and minorization (MM) algorithms have been applied with success in a variety of models arising in the area of statistics and data science. A key property of majorization algorithms is guaranteed descent, that is, the function value decreases in each step. In practical cases, the function is decreased until it has converged to a local minimum. If the function is convex and coercive, a global minimum is guaranteed. The auxiliary function, the so-called majorizing function, is often quadratic so that an update can be obtained in one step. Here, we present a selection of useful applications of MM algorithms. We discuss its use multidimensional scaling, and in binary and multiclass classification such as logistic regression, multinomial regression, and support vector machines. In the case of regularized generalized canonical correlation analysis, its MM algorithm coincides with several partial least squares (PLS) algorithms thereby providing a previously unknown goal function for the PLS algorithms. We show how MM can also be effective in large scale optimization problems, such as the SoftImpute approach for dealing with missings in principal components analysis, and the MM algorithm for convex clustering.