classification
Title: new_interpreter() should reuse more Py_InitializeFromConfig() code
Type: Stage: resolved
Components: Interpreter Core Versions: Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: eric.snow, vstinner
Priority: normal Keywords: patch

Created on 2019-11-19 23:11 by vstinner, last changed 2020-01-15 15:29 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 17265 merged vstinner, 2019-11-19 23:14
PR 17273 merged vstinner, 2019-11-20 01:37
PR 17286 merged vstinner, 2019-11-20 10:44
PR 17293 merged vstinner, 2019-11-20 14:59
PR 17297 merged vstinner, 2019-11-20 17:01
PR 17301 merged vstinner, 2019-11-20 21:49
PR 17315 merged vstinner, 2019-11-21 08:16
PR 17330 merged vstinner, 2019-11-21 22:38
PR 17346 merged vstinner, 2019-11-22 14:55
PR 17347 merged vstinner, 2019-11-22 15:48
PR 17350 merged vstinner, 2019-11-22 17:29
PR 17351 merged vstinner, 2019-11-22 17:53
PR 17353 merged vstinner, 2019-11-22 18:29
PR 17481 merged vstinner, 2019-12-06 01:06
PR 17483 merged vstinner, 2019-12-06 01:47
PR 17512 merged vstinner, 2019-12-08 20:24
Messages (23)
msg357001 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-11-19 23:11
Currently, new_interpreter() is a subset of Py_InitializeFromConfig(): the code was duplicated. I would prefer that both functions stay in sync and so that new_interpreter() reuses more Py_InitializeFromConfig() code.
msg357005 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-11-19 23:38
New changeset ef5aa9af7c7e493402ac62009e4400aed7c3d54e by Victor Stinner in branch 'master':
bpo-38858: Reorganize pycore_init_types() (GH-17265)
https://github.com/python/cpython/commit/ef5aa9af7c7e493402ac62009e4400aed7c3d54e
msg357043 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-11-20 09:38
New changeset 7eee5beaf87be898a679278c480e8dd0df76d351 by Victor Stinner in branch 'master':
bpo-38858: Factorize Py_EndInterpreter() code (GH-17273)
https://github.com/python/cpython/commit/7eee5beaf87be898a679278c480e8dd0df76d351
msg357052 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-11-20 10:46
> New changeset ef5aa9af7c7e493402ac62009e4400aed7c3d54e by Victor Stinner in branch 'master':
> bpo-38858: Reorganize pycore_init_types() (GH-17265)

This change introduced a reference leak:

https://buildbot.python.org/all/#builders/80/builds/771

test_atexit leaked [792, 792, 792] references, sum=2376
test_capi leaked [528, 528, 528] references, sum=1584
test__xxsubinterpreters leaked [17954, 17952, 17954] references, sum=53860
test_threading leaked [792, 792, 792] references, sum=2376

It should be fixed by PR 17286.
msg357055 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-11-20 11:08
New changeset e7e699e4df73420ddccaa0057cd07ebb3b590b9b by Victor Stinner in branch 'master':
bpo-38858: Fix reference leak in pycore_init_types() (GH-17286)
https://github.com/python/cpython/commit/e7e699e4df73420ddccaa0057cd07ebb3b590b9b
msg357083 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-11-20 16:34
New changeset fff7bbfdb6b7c143b73b6b4b6b40e828c101110c by Victor Stinner in branch 'master':
bpo-38858: Add _Py_IsMainInterpreter(tstate) (GH-17293)
https://github.com/python/cpython/commit/fff7bbfdb6b7c143b73b6b4b6b40e828c101110c
msg357092 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-11-20 17:39
New changeset b93f31fcd9f10b213c614d4944baf9ca2df2016c by Victor Stinner in branch 'master':
bpo-38858: Fix Py_Finalize() when called from a subinterpreter (GH-17297)
https://github.com/python/cpython/commit/b93f31fcd9f10b213c614d4944baf9ca2df2016c
msg357139 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-11-21 07:52
New changeset 5dcc06f6e0d7b5d6589085692b86c63e35e2325e by Victor Stinner in branch 'master':
bpo-38858: Allocate small integers on the heap (GH-17301)
https://github.com/python/cpython/commit/5dcc06f6e0d7b5d6589085692b86c63e35e2325e
msg357140 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-11-21 07:52
> bpo-38858: Allocate small integers on the heap (GH-17301)

I associated this change to this issue because I had troubles when I tried to call _PyLong_Fini() in subinterpreters. I'm now trying to have different small integers per interpreter.
msg357268 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-11-22 11:27
New changeset 3d4833488a173c16446c3f94f58f05e2d13c5dee by Victor Stinner in branch 'master':
bpo-38858: Call _PyUnicode_Fini() in Py_EndInterpreter() (GH-17330)
https://github.com/python/cpython/commit/3d4833488a173c16446c3f94f58f05e2d13c5dee
msg357278 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-11-22 13:27
I introduced a workaround to a reference leak in bpo-36854 because _PyImport_Cleanup() doesn't clear modules dictionary in some cases:
https://bugs.python.org/issue36854#msg357160

Maybe _PyImport_Cleanup() should be enhanced to ensure that the dictionary of all modules is cleared when the function exit.
msg357296 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-11-22 15:19
New changeset e0c9ab8e26d1648b870b80c296b2490a5e9553e5 by Victor Stinner in branch 'master':
bpo-38858: Add init_set_builtins_open() subfunction (GH-17346)
https://github.com/python/cpython/commit/e0c9ab8e26d1648b870b80c296b2490a5e9553e5
msg357298 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-11-22 16:52
New changeset b00513636c9891deba4cae50217e25e8faf6c6ac by Victor Stinner in branch 'master':
bpo-38858: Add init_interp_main() subfunction (GH-17347)
https://github.com/python/cpython/commit/b00513636c9891deba4cae50217e25e8faf6c6ac
msg357306 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-11-22 17:52
New changeset 82c83bd907409c287a5bd0d0f4598f2c0538f34d by Victor Stinner in branch 'master':
bpo-38858: _PyImport_FixupExtensionObject() handles subinterpreters (GH-17350)
https://github.com/python/cpython/commit/82c83bd907409c287a5bd0d0f4598f2c0538f34d
msg357311 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-11-22 18:24
New changeset 2582d46fbcf7bdf86b9cf4016850b8d155267ac6 by Victor Stinner in branch 'master':
bpo-38858: new_interpreter() reuses pycore_init_builtins() (GH-17351)
https://github.com/python/cpython/commit/2582d46fbcf7bdf86b9cf4016850b8d155267ac6
msg357321 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-11-22 20:54
New changeset 2ec1a1b307cc893adae4662a32e1d2e94b6908e3 by Victor Stinner in branch 'master':
bpo-38858: new_interpreter() uses pycore_init_import_warnings() (GH-17353)
https://github.com/python/cpython/commit/2ec1a1b307cc893adae4662a32e1d2e94b6908e3
msg357897 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-12-06 01:43
New changeset 81fe5bd3d78f9bb955f8255404d99df27a31c36a by Victor Stinner in branch 'master':
bpo-38858: new_interpreter() reuses _PySys_Create() (GH-17481)
https://github.com/python/cpython/commit/81fe5bd3d78f9bb955f8255404d99df27a31c36a
msg357899 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-12-06 02:37
New changeset d863ade0c7fa4826e8b71aa467809c83a711f019 by Victor Stinner in branch 'master':
bpo-38858: Add pycore_interp_init() code to factorize code (GH-17483)
https://github.com/python/cpython/commit/d863ade0c7fa4826e8b71aa467809c83a711f019
msg357920 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-12-06 16:01
Py_NewInterpreter() (new_interpreter() in practice) and Py_InitializeEx() now share almost all their code.

Py_EndInterpreter() shares almost all its code with Py_Finalizer(). It's not perfect, but it's way better than previously.

Py_NewInterpreter() now isolates more things from the main interpreter. For example, builtins and sys modules no longer copy the module dictionary of the main interpreter, but create their own dictionary from scratch.

See each commit for the details.

Py_Finalizer() could share more code with Py_EndInterpreter(), but each Py_Finalizer() change is really tricky and require to pay a lot attention. The Python finalization is really fragile. I started to take notes on this fragile code:
https://pythondev.readthedocs.io/finalization.html

I consider that the initial issue is fixed, so I close the issue.
msg358028 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-12-08 20:56
New changeset 080ee5a88406fb68aaab741145cd5d2a7c5f2ad6 by Victor Stinner in branch 'master':
bpo-38858: Fix ref leak in pycore_interp_init() (GH-17512)
https://github.com/python/cpython/commit/080ee5a88406fb68aaab741145cd5d2a7c5f2ad6
msg358356 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2019-12-13 21:29
Thanks for working on this.  It really does have far-reaching benefits, not just for the subinterpreter stuff I'm interested in. :)
msg358543 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-12-17 12:02
New changeset 630c8df5cf126594f8c1c4579c1888ca80a29d59 by Victor Stinner in branch 'master':
bpo-38858: Small integer per interpreter (GH-17315)
https://github.com/python/cpython/commit/630c8df5cf126594f8c1c4579c1888ca80a29d59
msg360057 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-01-15 15:29
FYI this change broke librepo which calls PyLong_FromLong() without holding the GIL. In Python 3.8, "it works". In Python 3.9, it does crash: get_small_int() gets a NULL tstate and then dereference a NULL pointer.

librepo bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1788918

IMHO it's a bug in librepo: the GIL must be held to use Python C API.
History
Date User Action Args
2020-01-15 15:29:47vstinnersetmessages: + msg360057
2019-12-17 12:02:27vstinnersetmessages: + msg358543
2019-12-13 21:29:11eric.snowsetmessages: + msg358356
2019-12-08 20:56:04vstinnersetmessages: + msg358028
2019-12-08 20:24:03vstinnersetpull_requests: + pull_request16988
2019-12-06 16:01:54vstinnersetstatus: open -> closed

nosy: + eric.snow
messages: + msg357920

resolution: fixed
stage: patch review -> resolved
2019-12-06 02:37:11vstinnersetmessages: + msg357899
2019-12-06 01:47:10vstinnersetpull_requests: + pull_request16962
2019-12-06 01:43:37vstinnersetmessages: + msg357897
2019-12-06 01:06:55vstinnersetpull_requests: + pull_request16960
2019-11-22 20:54:40vstinnersetmessages: + msg357321
2019-11-22 18:29:58vstinnersetpull_requests: + pull_request16838
2019-11-22 18:24:59vstinnersetmessages: + msg357311
2019-11-22 17:53:26vstinnersetpull_requests: + pull_request16835
2019-11-22 17:52:31vstinnersetmessages: + msg357306
2019-11-22 17:29:00vstinnersetpull_requests: + pull_request16834
2019-11-22 16:52:51vstinnersetmessages: + msg357298
2019-11-22 15:48:28vstinnersetpull_requests: + pull_request16831
2019-11-22 15:19:18vstinnersetmessages: + msg357296
2019-11-22 14:55:25vstinnersetpull_requests: + pull_request16830
2019-11-22 13:27:01vstinnersetmessages: + msg357278
2019-11-22 11:27:56vstinnersetmessages: + msg357268
2019-11-21 22:38:25vstinnersetpull_requests: + pull_request16817
2019-11-21 08:16:20vstinnersetpull_requests: + pull_request16802
2019-11-21 07:52:54vstinnersetmessages: + msg357140
2019-11-21 07:52:02vstinnersetmessages: + msg357139
2019-11-20 21:49:21vstinnersetpull_requests: + pull_request16793
2019-11-20 17:39:17vstinnersetmessages: + msg357092
2019-11-20 17:01:58vstinnersetpull_requests: + pull_request16789
2019-11-20 16:34:43vstinnersetmessages: + msg357083
2019-11-20 14:59:27vstinnersetpull_requests: + pull_request16786
2019-11-20 11:08:41vstinnersetmessages: + msg357055
2019-11-20 10:46:28vstinnersetmessages: + msg357052
2019-11-20 10:44:48vstinnersetpull_requests: + pull_request16779
2019-11-20 09:38:39vstinnersetmessages: + msg357043
2019-11-20 01:37:13vstinnersetpull_requests: + pull_request16766
2019-11-19 23:38:21vstinnersetmessages: + msg357005
2019-11-19 23:14:22vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request16757
2019-11-19 23:11:09vstinnercreate