This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Convert statistics sum of squares to a single pass algorithm
Type: performance Stage: resolved
Components: Library (Lib) Versions: Python 3.11
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: rhettinger, steven.daprano
Priority: normal Keywords: patch

Created on 2022-01-04 17:04 by rhettinger, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 30403 merged rhettinger, 2022-01-04 17:06
Messages (2)
msg409692 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2022-01-04 17:04
The existing code makes two passes, one to compute the mean and another to compute the sum of squared differences from the mean.  A consequence of making two passes is that iterator inputs must be converted to a list before processing.  This throws away the memory saving advantages of iterators. 

The ostensible reason for the two pass code is that the single pass variant is numerically unstable when implemented with floating point accumulators.  However, this code uses fractions throughout, so the accumulation is exact.

Changing to a single pass saves memory, doubles the speed, and simplifies the upstream code in variance(), pvariance(), stdev(), and pstdev().
msg409777 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2022-01-05 15:39
New changeset 43aac29cbbb8a963a22c334b5b795d1e43417d6b by Raymond Hettinger in branch 'main':
bpo-46257: Convert statistics._ss() to a single pass algorithm (GH-30403)
https://github.com/python/cpython/commit/43aac29cbbb8a963a22c334b5b795d1e43417d6b
History
Date User Action Args
2022-04-11 14:59:54adminsetgithub: 90415
2022-01-05 15:39:37rhettingersetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2022-01-05 15:39:19rhettingersetmessages: + msg409777
2022-01-04 17:06:26rhettingersetkeywords: + patch
stage: patch review
pull_requests: + pull_request28611
2022-01-04 17:04:05rhettingercreate