Nonparametric Statistics with Applications to Science and Engineering with R, 2nd Edition

16EM Algorithm

Insanity is doing the same thing over and over again and expecting different results.

Albert Einstein

The expectation–maximization (EM) algorithm is broadly applicable statistical technique for maximizing complex likelihoods while handling problems with incomplete data. Within each iteration of the algorithm, two steps are performed: (i) the ‐step consisting of projecting an appropriate functional containing the augmented data on the space of the original, incomplete data and (ii) the ‐step consisting of maximizing the functional.

The name EM algorithm was coined by Dempster, Laird, and Rubin (1977) in their fundamental paper, referred to here as the DLR paper. However, as is usually the case, if one comes to a smart idea, one may be sure that other smart guys in the history had already thought about it. Long before, McKendrick (1926) and Healy and Westmacott (1956) proposed iterative methods that are examples of the EM algorithm. In fact, before the DLR paper appeared in 1997, dozens of papers proposing various iterative solvers were essentially applying the EM Algorithm in some form.

However, the DLR paper was the first to formally recognize these separate algorithms as having the same fundamental underpinnings, so perhaps their 1977 paper prevented further reinventions ...