classification
Title: Improve stdlib module initialization error handling.
Type: behavior Stage: patch review
Components: Library (Lib) Versions: Python 3.9, Python 3.8, Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: brandtbucher Nosy List: asvetlov, brandtbucher, miss-islington, serhiy.storchaka, shihai1991, twouters, vstinner
Priority: normal Keywords: patch

Created on 2019-11-16 18:54 by brandtbucher, last changed 2020-02-08 04:02 by shihai1991.

Pull Requests
URL Status Linked Edit
PR 17195 merged brandtbucher, 2019-11-16 21:57
PR 17196 merged miss-islington, 2019-11-16 22:27
PR 17197 merged miss-islington, 2019-11-16 22:27
PR 17198 merged brandtbucher, 2019-11-16 23:13
PR 17199 merged miss-islington, 2019-11-16 23:57
PR 17200 merged miss-islington, 2019-11-16 23:58
PR 17206 merged brandtbucher, 2019-11-17 19:21
PR 17215 merged brandtbucher, 2019-11-18 06:38
PR 17216 merged brandtbucher, 2019-11-18 06:49
PR 17226 merged miss-islington, 2019-11-18 14:52
PR 17227 merged miss-islington, 2019-11-18 14:53
PR 17235 merged brandtbucher, 2019-11-18 16:49
PR 17236 merged brandtbucher, 2019-11-18 16:55
PR 17250 merged brandtbucher, 2019-11-19 07:02
PR 17260 merged brandtbucher, 2019-11-19 16:33
PR 17263 merged miss-islington, 2019-11-19 23:13
PR 17264 merged miss-islington, 2019-11-19 23:14
PR 17271 closed miss-islington, 2019-11-20 00:59
PR 17272 closed miss-islington, 2019-11-20 00:59
PR 17276 merged brandtbucher, 2019-11-20 08:00
PR 17280 merged miss-islington, 2019-11-20 09:56
PR 17281 merged miss-islington, 2019-11-20 09:57
PR 17282 merged miss-islington, 2019-11-20 10:00
PR 17283 merged miss-islington, 2019-11-20 10:00
PR 17298 open brandtbucher, 2019-11-20 18:44
PR 18365 shihai1991, 2020-02-05 18:56
Messages (27)
msg356764 - (view) Author: Brandt Bucher (brandtbucher) * (Python triager) Date: 2019-11-16 18:54
Many of the C stdlib modules can benefit from improved error handling during initialization. I've now had two PRs where the authors had reference leaks on error conditions, but defended their decisions by pointing to examples of similar idioms all over the stdlib.

The problems fall into two related categories, mostly:

- Not DECREF'ing the new module object on failure.
- Not handling errors raised by the PyModule_Add* family of functions... specifically, the weird steal-on-success semantics of PyModule_AddObject. I've already improved the docs for this, so we should see the issue less, but our own code should still be fixed.

I intend to turn this into a longer term project. I'll be working my way through these modules bit-by-bit over time, using this issue to track all of them (since there are a few dozen cases). I'd rather not make one huge one that spams all of the code owners and is impossible to review.

If anybody want to make themselves available to review/merge these as I go along, that would be great! Many of the ones I'll start with are just adding trivial DECREFs to the more popular modules (_asyncio, _contextvars, _functools, _random, _warnings, etc...).
msg356765 - (view) Author: Brandt Bucher (brandtbucher) * (Python triager) Date: 2019-11-16 19:37
How do others feel about the creation of a new private API? It would keep these diffs smaller and ease refactoring... and it would probably be good to have around anyways:

/* Like PyModule_AddObject, but steals o on success AND failure. */

int
_PyModule_StealObject(PyObject *m, const char *name, PyObject *o)
{
    if (PyModule_AddObject(m, name, o) < 0) {
        Py_XDECREF(o);
        return -1;
    }
    return 0;
}
msg356773 - (view) Author: miss-islington (miss-islington) Date: 2019-11-16 22:26
New changeset c3f6bdc332d23588102eba749a5929dd5bb67c9d by Miss Islington (bot) (Brandt Bucher) in branch 'master':
bpo-38823: Clean up refleaks in _asyncio initialization. (GH-17195)
https://github.com/python/cpython/commit/c3f6bdc332d23588102eba749a5929dd5bb67c9d
msg356774 - (view) Author: miss-islington (miss-islington) Date: 2019-11-16 22:44
New changeset 48f4f75baeee8ade1fbfab1b0aa6a21a3b13a2f2 by Miss Islington (bot) in branch '3.8':
bpo-38823: Clean up refleaks in _asyncio initialization. (GH-17195)
https://github.com/python/cpython/commit/48f4f75baeee8ade1fbfab1b0aa6a21a3b13a2f2
msg356775 - (view) Author: miss-islington (miss-islington) Date: 2019-11-16 22:45
New changeset 825e91be0407d6fc7fa034286b4e90634f181fab by Miss Islington (bot) in branch '3.7':
bpo-38823: Clean up refleaks in _asyncio initialization. (GH-17195)
https://github.com/python/cpython/commit/825e91be0407d6fc7fa034286b4e90634f181fab
msg356781 - (view) Author: miss-islington (miss-islington) Date: 2019-11-16 23:57
New changeset 143a97f64128070386b12a0ee589bdaad5e51f40 by Miss Islington (bot) (Brandt Bucher) in branch 'master':
bpo-38823: Clean up refleaks in _contextvars initialization. (GH-17198)
https://github.com/python/cpython/commit/143a97f64128070386b12a0ee589bdaad5e51f40
msg356782 - (view) Author: miss-islington (miss-islington) Date: 2019-11-17 00:14
New changeset 8a334af13368573cc645488481b1173d65eeeb9a by Miss Islington (bot) in branch '3.7':
bpo-38823: Clean up refleaks in _contextvars initialization. (GH-17198)
https://github.com/python/cpython/commit/8a334af13368573cc645488481b1173d65eeeb9a
msg356783 - (view) Author: miss-islington (miss-islington) Date: 2019-11-17 00:16
New changeset 1fe79a43400d092a074c6f9ae5eb290ea3e4f281 by Miss Islington (bot) in branch '3.8':
bpo-38823: Clean up refleaks in _contextvars initialization. (GH-17198)
https://github.com/python/cpython/commit/1fe79a43400d092a074c6f9ae5eb290ea3e4f281
msg356809 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2019-11-17 13:44
Is anything left to be done for the issue?
msg356820 - (view) Author: Brandt Bucher (brandtbucher) * (Python triager) Date: 2019-11-17 17:19
Yes, there are still a few dozen modules I plan to update!
msg356872 - (view) Author: miss-islington (miss-islington) Date: 2019-11-18 14:52
New changeset 289cf0fbf78c4f38c38ac71ac8b772be7ec2672f by Miss Islington (bot) (Brandt Bucher) in branch 'master':
bpo-38823: Clean up refleaks in _tkinter initialization. (GH-17206)
https://github.com/python/cpython/commit/289cf0fbf78c4f38c38ac71ac8b772be7ec2672f
msg356873 - (view) Author: miss-islington (miss-islington) Date: 2019-11-18 15:09
New changeset 9e4d0312101cc2bc44a9549974d4a25f80e1dc12 by Miss Islington (bot) in branch '3.7':
bpo-38823: Clean up refleaks in _tkinter initialization. (GH-17206)
https://github.com/python/cpython/commit/9e4d0312101cc2bc44a9549974d4a25f80e1dc12
msg356874 - (view) Author: miss-islington (miss-islington) Date: 2019-11-18 15:10
New changeset 42a4359390b78b3360e100fc9c075495e48354b2 by Miss Islington (bot) in branch '3.8':
bpo-38823: Clean up refleaks in _tkinter initialization. (GH-17206)
https://github.com/python/cpython/commit/42a4359390b78b3360e100fc9c075495e48354b2
msg356984 - (view) Author: Thomas Wouters (twouters) * (Python committer) Date: 2019-11-19 19:16
New changeset 54b32c987146123f2237f0e21b1d02b1c1ebdf6f by T. Wouters (Brandt Bucher) in branch 'master':
bpo-38823: Clean up refleak in fcntl module initialization. (GH-17236)
https://github.com/python/cpython/commit/54b32c987146123f2237f0e21b1d02b1c1ebdf6f
msg357002 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-11-19 23:13
New changeset ac2235432c607ce2c0faf6dff5d9b2534d2f6652 by Victor Stinner (Brandt Bucher) in branch 'master':
bpo-38823: Fix refleaks in faulthandler init error path on Windows (GH-17250)
https://github.com/python/cpython/commit/ac2235432c607ce2c0faf6dff5d9b2534d2f6652
msg357003 - (view) Author: miss-islington (miss-islington) Date: 2019-11-19 23:30
New changeset 5bd2af9c79796aa03b06d1e35ab6df7e28364e24 by Miss Islington (bot) in branch '3.7':
bpo-38823: Fix refleaks in faulthandler init error path on Windows (GH-17250)
https://github.com/python/cpython/commit/5bd2af9c79796aa03b06d1e35ab6df7e28364e24
msg357004 - (view) Author: miss-islington (miss-islington) Date: 2019-11-19 23:31
New changeset a5ed2fe0eedefa1649aa93ee74a0bafc8e628a10 by Miss Islington (bot) in branch '3.8':
bpo-38823: Fix refleaks in faulthandler init error path on Windows (GH-17250)
https://github.com/python/cpython/commit/a5ed2fe0eedefa1649aa93ee74a0bafc8e628a10
msg357014 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-11-20 00:59
New changeset 33b671e72450bf4b5a946ce0dde6b7fe21150108 by Victor Stinner (Brandt Bucher) in branch 'master':
bpo-38823: Fix refleak in marshal init error path (GH-17260)
https://github.com/python/cpython/commit/33b671e72450bf4b5a946ce0dde6b7fe21150108
msg357019 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-11-20 01:42
The Azure Pipelines are sick tonight (unable to publish test results). PR #17272 and PR #17271 are blocked by this CI.
msg357029 - (view) Author: Brandt Bucher (brandtbucher) * (Python triager) Date: 2019-11-20 05:51
Thanks Victor. These obviously aren’t urgent, so feel free to return to them whenever’s convenient.

I also pinged you on #17235 (_tracemalloc) too.
msg357044 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-11-20 10:00
New changeset d51a363a4379385fdfe9c09a56324631465ede29 by Victor Stinner (Brandt Bucher) in branch 'master':
bpo-38823: Fix refleak in _tracemalloc init error handling (GH-17235)
https://github.com/python/cpython/commit/d51a363a4379385fdfe9c09a56324631465ede29
msg357046 - (view) Author: miss-islington (miss-islington) Date: 2019-11-20 10:15
New changeset 63f09e7628bd72f1bb2106226655b1f775757806 by Miss Islington (bot) in branch '3.7':
bpo-38823: Fix refleak in marshal init error path (GH-17260)
https://github.com/python/cpython/commit/63f09e7628bd72f1bb2106226655b1f775757806
msg357047 - (view) Author: miss-islington (miss-islington) Date: 2019-11-20 10:16
New changeset 2ea4c37c1ecf05a8495211d55ed6888439b1b9cf by Miss Islington (bot) in branch '3.8':
bpo-38823: Fix refleak in marshal init error path (GH-17260)
https://github.com/python/cpython/commit/2ea4c37c1ecf05a8495211d55ed6888439b1b9cf
msg357050 - (view) Author: miss-islington (miss-islington) Date: 2019-11-20 10:27
New changeset daf7a082b20e59a0518cda1500add42c36ab058f by Miss Islington (bot) in branch '3.8':
bpo-38823: Fix refleak in _tracemalloc init error handling (GH-17235)
https://github.com/python/cpython/commit/daf7a082b20e59a0518cda1500add42c36ab058f
msg357051 - (view) Author: miss-islington (miss-islington) Date: 2019-11-20 10:27
New changeset 1d7245c3e0cfe4508855c5025b25d8894155ecc5 by Miss Islington (bot) in branch '3.7':
bpo-38823: Fix refleak in _tracemalloc init error handling (GH-17235)
https://github.com/python/cpython/commit/1d7245c3e0cfe4508855c5025b25d8894155ecc5
msg357114 - (view) Author: miss-islington (miss-islington) Date: 2019-11-21 00:17
New changeset e5d1f734db135e284af8e8868e7ccc85355952b9 by Miss Islington (bot) (Brandt Bucher) in branch 'master':
bpo-38823: Clean up _xxtestfuzz initialization. (GH-17216)
https://github.com/python/cpython/commit/e5d1f734db135e284af8e8868e7ccc85355952b9
msg361486 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-06 14:46
New changeset d2f96672642cc51b92f61b951ca1b11d615c12d1 by Brandt Bucher in branch 'master':
bpo-38823: Fix refleaks in _ast initialization error path (GH-17276)
https://github.com/python/cpython/commit/d2f96672642cc51b92f61b951ca1b11d615c12d1
History
Date User Action Args
2020-02-08 04:02:21shihai1991setnosy: + shihai1991
2020-02-06 14:46:01vstinnersetmessages: + msg361486
2020-02-05 18:56:46shihai1991setpull_requests: + pull_request17745
2019-11-21 00:17:06miss-islingtonsetmessages: + msg357114
2019-11-20 18:44:44brandtbuchersetpull_requests: + pull_request16790
2019-11-20 10:27:07miss-islingtonsetmessages: + msg357051
2019-11-20 10:27:07miss-islingtonsetmessages: + msg357050
2019-11-20 10:16:10miss-islingtonsetmessages: + msg357047
2019-11-20 10:15:25miss-islingtonsetmessages: + msg357046
2019-11-20 10:00:57miss-islingtonsetpull_requests: + pull_request16776
2019-11-20 10:00:50miss-islingtonsetpull_requests: + pull_request16775
2019-11-20 10:00:36vstinnersetmessages: + msg357044
2019-11-20 09:57:16miss-islingtonsetpull_requests: + pull_request16774
2019-11-20 09:56:37miss-islingtonsetpull_requests: + pull_request16773
2019-11-20 08:00:07brandtbuchersetpull_requests: + pull_request16769
2019-11-20 05:51:53brandtbuchersetmessages: + msg357029
2019-11-20 01:42:14vstinnersetmessages: + msg357019
2019-11-20 00:59:53miss-islingtonsetpull_requests: + pull_request16765
2019-11-20 00:59:46miss-islingtonsetpull_requests: + pull_request16764
2019-11-20 00:59:40vstinnersetmessages: + msg357014
2019-11-19 23:31:12miss-islingtonsetmessages: + msg357004
2019-11-19 23:30:13miss-islingtonsetmessages: + msg357003
2019-11-19 23:14:06miss-islingtonsetpull_requests: + pull_request16756
2019-11-19 23:13:59miss-islingtonsetpull_requests: + pull_request16755
2019-11-19 23:13:09vstinnersetnosy: + vstinner
messages: + msg357002
2019-11-19 19:16:59twouterssetnosy: + twouters
messages: + msg356984
2019-11-19 16:33:22brandtbuchersetpull_requests: + pull_request16753
2019-11-19 07:02:18brandtbuchersetpull_requests: + pull_request16747
2019-11-18 16:55:36brandtbuchersetpull_requests: + pull_request16736
2019-11-18 16:49:55brandtbuchersetpull_requests: + pull_request16735
2019-11-18 15:10:34miss-islingtonsetmessages: + msg356874
2019-11-18 15:09:52miss-islingtonsetmessages: + msg356873
2019-11-18 14:53:00miss-islingtonsetpull_requests: + pull_request16729
2019-11-18 14:52:53miss-islingtonsetpull_requests: + pull_request16728
2019-11-18 14:52:42miss-islingtonsetmessages: + msg356872
2019-11-18 06:49:46brandtbuchersetpull_requests: + pull_request16719
2019-11-18 06:38:53brandtbuchersetpull_requests: + pull_request16718
2019-11-17 19:21:10brandtbuchersetpull_requests: + pull_request16711
2019-11-17 17:19:34brandtbuchersetmessages: + msg356820
2019-11-17 13:44:08asvetlovsetnosy: + asvetlov

messages: + msg356809
versions: + Python 3.7, Python 3.8
2019-11-17 00:16:36miss-islingtonsetmessages: + msg356783
2019-11-17 00:14:52miss-islingtonsetmessages: + msg356782
2019-11-16 23:58:29miss-islingtonsetpull_requests: + pull_request16704
2019-11-16 23:57:46miss-islingtonsetpull_requests: + pull_request16703
2019-11-16 23:57:36miss-islingtonsetmessages: + msg356781
2019-11-16 23:13:47brandtbuchersetpull_requests: + pull_request16702
2019-11-16 22:45:28miss-islingtonsetmessages: + msg356775
2019-11-16 22:44:30miss-islingtonsetmessages: + msg356774
2019-11-16 22:27:15miss-islingtonsetpull_requests: + pull_request16701
2019-11-16 22:27:09miss-islingtonsetpull_requests: + pull_request16700
2019-11-16 22:26:58miss-islingtonsetnosy: + miss-islington
messages: + msg356773
2019-11-16 21:57:59brandtbuchersetkeywords: + patch
stage: patch review
pull_requests: + pull_request16699
2019-11-16 20:23:38brandtbuchersetnosy: + serhiy.storchaka
2019-11-16 19:37:20brandtbuchersetmessages: + msg356765
2019-11-16 18:54:25brandtbuchercreate