Calculate Standard deviation incrementally one batch at a time
In this article, I would like to display a way to calculate standard deviation in an incremental way. Unlike existing articles, that shows how to update the calculation when adding one more element. I will display a method to calculate standard deviation (std.dev) from existing batch of data and additional batch of data. Motivation - suppose you have an ETL system that processes batches of data every day like Apache Spark and you need to display statistical data for all the data like std.dev. If you calculate it on the entire set of data the calculation will become increasingly "harder" since the data set will always grow in size and the calculation will require more and more time. The definition of standard deviation is: So once additional data is added the entire formula should be re-calculated. And the "hard" parts (where we do a full scan of the data) is in the average and the squared difference from the mean should be recalculated. How...