Title: _PySys_EndInit() doesn't copy main interpreter configuration
Type: Stage: resolved
Components: Interpreter Core Versions: Python 3.8, Python 3.7
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: eric.snow, miss-islington, ncoghlan, vstinner
Priority: normal Keywords: patch

Created on 2018-11-13 23:08 by vstinner, last changed 2018-11-22 15:12 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 10532 merged vstinner, 2018-11-14 00:33
PR 10566 merged miss-islington, 2018-11-16 11:15
PR 10660 merged vstinner, 2018-11-22 14:38
Messages (7)
msg329872 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-11-13 23:08
Extract of _PySys_EndInit():

SET_SYS_FROM_STRING_BORROW("path", config->module_search_path);

sys.path is initialized from _PyMainInterpreterConfig.module_search_path object: the list of strings is not copied. As a consequence, when sys.path is modified, _PyMainInterpreterConfig is modified as well. For example, "import site" modifies sys.path.

I dislike this behavior. I prefer to see _PyMainInterpreterConfig as "constant": a snapshot of the configuration used to startup Python.

A side effect is that Py_NewInterpreter() copies the modified configuration, whereas starting from the "original" configuration. For the specific case of Py_NewInterpreter(), I'm not sure of what is the expected behavior... Py_NewInterpreter() imports the "site" module again, so it should reapply the same sys.path change.

Note: I found this bug while working on better tests for global, core and main configurations:
msg329879 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-11-14 01:10
Attached PR 10532 modifies _PySys_EndInit() to copy lists and dictionaries from the config.
msg329905 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-11-14 12:59
Python has 3 kind of configurations:

* global configuration variables like Py_VerboseFlag
* core configuration: _PyCoreConfig
* main interpreter configuration: _PyMainInterpreterConfig

I tried to keep them consistency. Yesterday, I rewrote test_embed.InitConfigTests to really test that these 3 configurations are consistent... And I found multiple bugs :-) (I fixed them as well)

sys.flags is immutable, but some configurations are only used to "initialize" Python which can then be modified. sys.path is a good example.

I don't think that we can ensure that sys.path is always consistent with the main/core configuration (module_search_paths). It's possible to write "sys.path = ['/new/path']". There is no machinery at the module level to call a function when a sys module is *replaced*.

I propose to try to ensure that the configuration is not modified during Python lifecycle, and so that sys.path is a *copy* of the configuration. Same rationale for sys.warnoptions (list) and sys._xoptions (dict).

Attached PR 10532 implements this solution: copy lists and dicts.
msg329985 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-11-16 10:55
New changeset 37cd982df02795905886ab36a2378ed557cb6f60 by Victor Stinner in branch 'master':
bpo-35239: _PySys_EndInit() copies module_search_path (GH-10532)
msg329987 - (view) Author: miss-islington (miss-islington) Date: 2018-11-16 11:34
New changeset d2be9a5c13221fb84c2221bbfd93efac6111e697 by Miss Islington (bot) in branch '3.7':
bpo-35239: _PySys_EndInit() copies module_search_path (GH-10532)
msg330264 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-11-22 15:11
New changeset a5194115733f6ca8fc1ddbee43eabbde536900e6 by Victor Stinner in branch '3.7':
Revert "bpo-35239: _PySys_EndInit() copies module_search_path (GH-10532)" (GH-10660)
msg330265 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-11-22 15:12
I has been decided to not backport the change to Python 3.7: see PR 10532 discusssion.
Date User Action Args
2018-11-22 15:12:08vstinnersetmessages: + msg330265
2018-11-22 15:11:19vstinnersetmessages: + msg330264
2018-11-22 14:38:42vstinnersetpull_requests: + pull_request9913
2018-11-16 11:35:44vstinnersetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2018-11-16 11:34:38miss-islingtonsetnosy: + miss-islington
messages: + msg329987
2018-11-16 11:15:10miss-islingtonsetpull_requests: + pull_request9815
2018-11-16 10:55:39vstinnersetmessages: + msg329985
2018-11-14 12:59:36vstinnersetmessages: + msg329905
2018-11-14 01:10:43vstinnersetmessages: + msg329879
2018-11-14 00:33:30vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request9786
2018-11-13 23:08:32vstinnercreate