classification
Title: Make atexit state per interpreter
Type: Stage: resolved
Components: Library (Lib) Versions: Python 3.10
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: corona10, shihai1991, vstinner
Priority: normal Keywords: patch

Created on 2020-12-14 12:00 by vstinner, last changed 2020-12-15 16:17 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 23763 merged vstinner, 2020-12-14 12:02
PR 23770 merged vstinner, 2020-12-14 21:18
PR 23771 merged vstinner, 2020-12-14 21:43
PR 23699 closed corona10, 2020-12-15 14:09
PR 23683 closed shihai1991, 2020-12-15 14:09
PR 23777 merged vstinner, 2020-12-15 14:11
PR 23779 merged vstinner, 2020-12-15 15:11
Messages (7)
msg382982 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-12-14 12:00
In Python 2.7, atexit was implemented in Python and registered itself using sys.exitfunc public attribute:
https://docs.python.org/2.7/library/sys.html#sys.exitfunc
https://docs.python.org/2.7/library/atexit.html#module-atexit

In Python 3.0, the atexit module was rewritten in C. A new private _Py_PyAtExit() function was added to set a new private global "pyexitfunc" variable: variable used by call_py_exitfuncs() called by Py_Finalize().

In Python 3.7, the global "pyexitfunc" variable was moved int _PyRuntimeState (commit 2ebc5ce42a8a9e047e790aefbf9a94811569b2b6), and then into PyInterpreterState (commit 776407fe893fd42972c7e3f71423d9d86741d07c).

In Python 3.7, the atexit module was upgrade to the multiphase initialization API (PEP 489): PyInit_atexit() uses PyModuleDef_Init().

Since Python 2.7, the atexit module has a limitation: if a second instance is created, the new instance overrides the old one, and old registered callbacks are newer called.

One option is to disallow creating a second instance: see bpo-40600 and PR 23699 for that.

Another option is to move the atexit state (callbacks) into PyInterpreterState. Two atexit module instances would modify the same list of callbacks. In this issue, I propose to investigate this option.
msg383005 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-12-14 21:40
New changeset 83d52044ae4def1e8611a4b1b9263b850ca5c458 by Victor Stinner in branch 'master':
bpo-42639: Cleanup atexitmodule.c (GH-23770)
https://github.com/python/cpython/commit/83d52044ae4def1e8611a4b1b9263b850ca5c458
msg383008 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-12-14 22:08
New changeset 357704c9f2375f29ed5b3a93adac086fa714538d by Victor Stinner in branch 'master':
bpo-42639: atexit now logs callbacks exceptions (GH-23771)
https://github.com/python/cpython/commit/357704c9f2375f29ed5b3a93adac086fa714538d
msg383055 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-12-15 13:34
New changeset b8fa135908d294b350cdad04e2f512327a538dee by Victor Stinner in branch 'master':
bpo-42639: Move atexit state to PyInterpreterState (GH-23763)
https://github.com/python/cpython/commit/b8fa135908d294b350cdad04e2f512327a538dee
msg383064 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-12-15 15:08
New changeset f7049b5fb680c774e4950d10be62859a749f4e79 by Victor Stinner in branch 'master':
bpo-42639: Add script_helper.run_test_script() (GH-23777)
https://github.com/python/cpython/commit/f7049b5fb680c774e4950d10be62859a749f4e79
msg383071 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-12-15 16:12
New changeset 3ca2b8fd75043927f0bb03b8dac72d32beae255d by Victor Stinner in branch 'master':
bpo-42639: atexit._run_exitfuncs() uses sys.unraisablehook (GH-23779)
https://github.com/python/cpython/commit/3ca2b8fd75043927f0bb03b8dac72d32beae255d
msg383073 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-12-15 16:17
Ok, the initial issue is now fixed. It's possible to have more than one atexit module instance, registered callbacks in one or the other module are all called as expected. I also wrote an unit test for that.

I also took the opportunity of this issue to modernize the C code base, fix a bug in test_atexit (no longer clear atexit callbacks, run tests in subprocesses), and enhance atexit._run_exitfuncs(): log all callback exceptions using sys.unraisablehook so it's possible to catch them using the hook.
History
Date User Action Args
2020-12-27 23:43:32vstinnerlinkissue40288 superseder
2020-12-15 16:17:11vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg383073

stage: patch review -> resolved
2020-12-15 16:12:10vstinnersetmessages: + msg383071
2020-12-15 15:11:37vstinnersetpull_requests: + pull_request22637
2020-12-15 15:08:24vstinnersetmessages: + msg383064
2020-12-15 14:11:31vstinnersetpull_requests: + pull_request22635
2020-12-15 14:09:30shihai1991setpull_requests: + pull_request22634
2020-12-15 14:09:22corona10setnosy: + corona10
pull_requests: + pull_request22633
2020-12-15 13:34:50vstinnersetmessages: + msg383055
2020-12-15 04:43:44shihai1991setnosy: + shihai1991
2020-12-14 22:08:19vstinnersetmessages: + msg383008
2020-12-14 21:43:33vstinnersetpull_requests: + pull_request22627
2020-12-14 21:40:47vstinnersetmessages: + msg383005
2020-12-14 21:18:56vstinnersetpull_requests: + pull_request22626
2020-12-14 12:02:08vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request22620
2020-12-14 12:00:33vstinnercreate