July 2016
Beginner to intermediate
462 pages
9h 14m
English
Content preview from Python Data Analysis CookbookBecome an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
Start your free trial
Calculating the mean, variance, skewness, and kurtosis on the fly
Mean, variance, skewness, and kurtosis are important quantities in statistics. Some of the calculations involve sums of squares, which for large values may lead to overflow. To avoid loss of precision, we have to realize that variance is invariant under shift by a certain constant number.
When we have enough space in memory, we can directly calculate the moments, taking into account numerical issues if necessary. However, we may want to not keep the data in memory because there is a lot of it, or because it is more convenient to calculate the moments on the fly.
An online and numerically stable algorithm to calculate the variance has been provided by Terriberry (Terriberry, Timothy ...