classification
Title: PEP 432: Rewrite Py_Main()
Type: Stage: resolved
Components: Versions: Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: barry, eric.snow, miss-islington, ncoghlan, ned.deily, serhiy.storchaka, vstinner
Priority: normal Keywords: patch

Created on 2017-11-15 01:30 by vstinner, last changed 2018-10-30 13:31 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 4399 merged vstinner, 2017-11-15 01:31
PR 4412 merged vstinner, 2017-11-16 01:44
PR 4485 merged vstinner, 2017-11-21 01:32
PR 4504 merged vstinner, 2017-11-22 17:50
PR 4511 merged vstinner, 2017-11-23 00:14
PR 4513 merged vstinner, 2017-11-23 01:44
PR 4521 merged vstinner, 2017-11-23 14:52
PR 4526 merged vstinner, 2017-11-23 17:03
PR 4542 merged vstinner, 2017-11-24 14:52
PR 4548 merged vstinner, 2017-11-24 22:36
PR 4551 merged vstinner, 2017-11-25 00:33
PR 4625 merged vstinner, 2017-11-29 00:54
PR 4637 merged vstinner, 2017-11-29 16:30
PR 4649 merged vstinner, 2017-11-30 16:43
PR 4663 merged vstinner, 2017-12-01 17:01
PR 4665 merged vstinner, 2017-12-01 18:07
PR 4667 merged vstinner, 2017-12-01 18:49
PR 4668 merged vstinner, 2017-12-01 19:13
PR 4669 merged vstinner, 2017-12-01 19:55
PR 4673 merged vstinner, 2017-12-02 01:39
PR 4679 closed serhiy.storchaka, 2017-12-02 18:18
PR 4681 merged serhiy.storchaka, 2017-12-02 18:50
PR 4694 merged vstinner, 2017-12-04 09:40
PR 4728 merged vstinner, 2017-12-05 13:44
PR 4735 merged vstinner, 2017-12-06 15:43
PR 4736 merged vstinner, 2017-12-06 15:54
PR 4737 merged vstinner, 2017-12-06 16:27
PR 4838 merged vstinner, 2017-12-13 14:04
PR 4845 merged vstinner, 2017-12-13 18:18
PR 4854 merged vstinner, 2017-12-14 00:58
PR 4855 merged vstinner, 2017-12-14 01:29
PR 4868 closed vstinner, 2017-12-14 16:01
PR 4873 merged vstinner, 2017-12-14 23:16
PR 4874 merged vstinner, 2017-12-14 23:39
PR 4876 merged vstinner, 2017-12-15 00:47
PR 4899 merged vstinner, 2017-12-16 03:10
PR 4919 merged vstinner, 2017-12-18 14:27
PR 4921 merged vstinner, 2017-12-19 01:13
PR 4934 merged vstinner, 2017-12-19 22:26
PR 4935 merged vstinner, 2017-12-20 00:04
PR 4936 merged vstinner, 2017-12-20 03:00
PR 4946 merged vstinner, 2017-12-20 17:48
PR 4953 merged vstinner, 2017-12-20 21:52
PR 4954 merged vstinner, 2017-12-21 00:50
PR 4960 merged vstinner, 2017-12-21 14:27
PR 4961 merged vstinner, 2017-12-21 14:48
PR 4963 merged vstinner, 2017-12-21 15:23
PR 7712 merged vstinner, 2018-06-15 17:41
PR 7730 merged miss-islington, 2018-06-15 22:07
PR 10236 merged vstinner, 2018-10-30 12:13
Messages (68)
msg306245 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-11-15 01:30
Python has a lot of code for its initialization. It's very hard to touch this code without risking to break something. It's hard to move code since many parts of the code are interdepent. The code rely on global "Py_xxx" configuration variables like Py_IsolateFlag (set by -I command line option).

Moreover, currently Python uses the "Python runtime" early. For example, the code to parse the -W command line option uses PyUnicode_FromWideChar() and PyList_Append(). We need a stricter separation for the code before the "Python runtime" is initialized, at least partially initialized.

Nick Coghlan and Eric Snow are already working on all these issues as part of the implementation of PEP 432. They redesigned Py_Initialize() and Py_Finalize().

I would like to finish the work on the step before: the Py_Main() function.

Attached PR is a work-in-progress to rework deeply the Py_Main() function. I have different goals:

* Enhance error handling:

  * Avoid whenever possible calls to Py_FatalError() -- currently, Py_FatalError() is still called, but at a single place
  * My patch adds missing checks on PyDict_SetItem() or PyList_Append() calls, catch errors when adding warnings options and XOptions

* Reorder code to initialize: initialize Python in the "correct" order
* Better "finalization": pymain_free() is now responsible to free memory of all data used by Py_Main(). The ownership of strings is now better defined. For example, Py_SetProgramName() memory was not released before.
* pymain_init() is now the code which must not use the Python runtime
* pymain_core() uses the Python runtime. Its code to initialize the Python runtime should be easier to follow

 
Since pymain_free() now wants to release the memory, we need to force a memory allocator for PyMem_RawMalloc(), since pymain_core() changes the memory allocator. The main() already does something similar, but with simpler code since main() is a private function, whereas Py_Main() seems to be part of the public C API!
msg306246 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-11-15 01:41
I rewrote Py_Main() to prepare the code to be able to implement my "-X dev" idea:
https://mail.python.org/pipermail/python-dev//2017-November/150514.html

The problem is that currently the code parsing command line options and the code setting the memory allocator (handle PYTHONMALLOC environment variable) are mixed, it's not possible to touch this code.

I had similar technical issues when trying to implement properly my PEP 540 idea (Add a new UTF-8 mode): it's hard to change the Python filesystem encoding to UTF-8 after parsing command line arguments, since the current code to parse command line arguments already rely on the Python filesystem encoding and other parts of the Python runtime like the memory allocators.

If I implemented my PR correctly, it should become much more simpler to control PYTHONMALLOC and the filesystem encoding from the command line: from an -X option.
msg306250 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-11-15 02:18
While it doesn't necessarily need to be in this patch, something else I recently realised (by breaking it *cough* [1]) is that the interaction between our command line options and our environment variables isn't really clearly defined anywhere.

https://github.com/python/cpython/commit/d7ac06126db86f76ba92cbca4cb702852a321f78 restored the handling of simple on/off toggles as "toggle enabled = env var is set OR CLI flag is passed", but I noticed the other day that the interaction between PYTHONWARNINGS, the `-W` option, sys.warnoptions, and _warnings.filters is a bit confusing:

```
$ PYTHONWARNINGS=always,default python3 -Wignore -Wonce
Python 3.6.2 (default, Oct  2 2017, 16:51:32) 
[GCC 7.2.1 20170915 (Red Hat 7.2.1-2)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys, _warnings
>>> sys.warnoptions
['always', 'default', 'ignore', 'once']
>>> [f[0] for f in _warnings.filters[:4]]
['once', 'ignore', 'default', 'always']
```

The ordering makes *sense* (where sys.warnoptions just lists filter definitions in the order they're given, and later filters take priority over earlier ones, just as they do for any "warnings.filterwarnings" call), but it isn't immediately intuitive (since the outcome relies on filters being prepended by default).

That said, I've checked and the current warnings configuration behaviour *is* explicitly covered by the test suite (in https://github.com/python/cpython/blob/master/Lib/test/test_warnings/__init__.py), so a passing test suite provides confidence we haven't broken anything on that front.

[1] https://bugs.python.org/issue31845
msg306288 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-11-15 17:27
I wrote a new "_PyInitError" type to report more information when something goes wrong:

* indicate if it's an user error: don't abort() in that case
* function name where the error was raised
* error message

Example:
---
$ PYTHONHASHSEED=x ./python
Fatal Python error: _Py_HashRandomization_Init: PYTHONHASHSEED must be "random" or an integer in range [0; 4294967295]
---

=> Python doesn't fail with abort() anymore
=> notice the new "_Py_HashRandomization_Init" function name which gives context to the error message


Previously, Python called abort() and so might dump a core file:
---
$ PYTHONHASHSEED=x python3
Fatal Python error: PYTHONHASHSEED must be "random" or an integer in range [0; 4294967295]

Aborted (core dumped)
---
msg306321 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-11-15 23:48
New changeset f7e5b56c37eb859e225e886c79c5d742c567ee95 by Victor Stinner in branch 'master':
bpo-32030: Split Py_Main() into subfunctions (#4399)
https://github.com/python/cpython/commit/f7e5b56c37eb859e225e886c79c5d742c567ee95
msg306328 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-11-16 02:04
Nick:
"""
I noticed the other day that the interaction between PYTHONWARNINGS, the `-W` option, sys.warnoptions, and _warnings.filters is a bit confusing:


$ PYTHONWARNINGS=always,default python3 -Wignore -Wonce
(...)
>>> sys.warnoptions
['always', 'default', 'ignore', 'once']
>>> [f[0] for f in _warnings.filters[:4]]
['once', 'ignore', 'default', 'always']
"""

IMHO the command line must have the priority over environment variables, since environment variables are inherited, whereas the command line can be finely tuned *on purpose*. So it's correct, no? It's just that warnoptions gives options in the reverse order than the "expected" order, no?

At least, with my commit, if you want to try to change the priority, you just have to exchange two lines in pymain_add_warnings_options() :-)

    if (pymain_add_warnings_optlist(&pymain->env_warning_options) < 0) ...
    if (pymain_add_warnings_optlist(&pymain->cmdline.warning_options) < 0) ...

--

By the way, I'm not sure that it was a good idea to expose sys.warnoptions to users, but I don't think that we can remove it anymore :-)
msg306329 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-11-16 02:11
New changeset a7368ac6360246b1ef7f8f152963c2362d272183 by Victor Stinner in branch 'master':
bpo-32030: Enhance Py_Main() (#4412)
https://github.com/python/cpython/commit/a7368ac6360246b1ef7f8f152963c2362d272183
msg306332 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-11-16 02:52
I rewrote Py_Main() in two large commits. The code isn't perfect yet, for example, init_sys_streams() still reads the PYTHONIOENCODING environment variable. IMHO it should be read earlier. But as soon, my changes should allow to fix such issues much more easily.

Since I was now able to implement my -X dev option in bpo-32043, I don't plan to push further changes in short term on Py_Main().

I keep this issue open a little bit to check if everything is fine on buildbots, and check if someone has complains :-)
msg306616 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-11-21 02:12
New changeset 25420fe290b98171e6d30edf9350292c21ef700e by Victor Stinner in branch 'master':
bpo-32030: Add more options to _PyCoreConfig (#4485)
https://github.com/python/cpython/commit/25420fe290b98171e6d30edf9350292c21ef700e
msg306759 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-11-22 23:12
New changeset d4341109746aa15e1909e63b30b93b6133ffe401 by Victor Stinner in branch 'master':
bpo-32030: Add _PyCoreConfig.module_search_path_env (#4504)
https://github.com/python/cpython/commit/d4341109746aa15e1909e63b30b93b6133ffe401
msg306763 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-11-22 23:56
Victor, *please* don't add the external import settings to CoreConfig. That struct should only contain the absolute bare minimum of settings needed to get an interpreter that *can't* access the filesystem, such that builtin modules and frozen modules work, but nothing else does.

If you need some extra structures to hold command line and environment state, that's fine, but the full external import system attributes should go in the main interpreter config, as described in https://www.python.org/dev/peps/pep-0432/#supported-configuration-settings
msg306767 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-11-23 00:16
Nick: Ok, I created PR 4511.
msg306768 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-11-23 00:18
Nice, thanks for that. Good call on keeping the current data types for now, so we can focus on consolidating the configuration settings first, and then look at upgrading from C level types to Python level types later.
msg306775 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-11-23 00:49
New changeset e32e79f7d8216b78ac9e61bb1f2eee693108d4ee by Victor Stinner in branch 'master':
bpo-32030: Move PYTHONPATH to _PyMainInterpreterConfig (#4511)
https://github.com/python/cpython/commit/e32e79f7d8216b78ac9e61bb1f2eee693108d4ee
msg306794 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-11-23 09:43
New changeset 1f15111a6e15d52f6b08907576ec61493cd59358 by Victor Stinner in branch 'master':
bpo-32030: Add _PyMainInterpreterConfig.pythonhome (#4513)
https://github.com/python/cpython/commit/1f15111a6e15d52f6b08907576ec61493cd59358
msg306824 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-11-23 16:03
New changeset 0327bde9da203bb256b58218d012ca76ad0db4e4 by Victor Stinner in branch 'master':
bpo-32030: Rewrite calculate_path() (#4521)
https://github.com/python/cpython/commit/0327bde9da203bb256b58218d012ca76ad0db4e4
msg306851 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-11-23 18:02
New changeset b9197959c186f9061bd74d0adc20e96556426db4 by Victor Stinner in branch 'master':
bpo-32030: Fix calculate_path() on macOS (#4526)
https://github.com/python/cpython/commit/b9197959c186f9061bd74d0adc20e96556426db4
msg306924 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-11-24 21:55
New changeset 46972b7bc385ec2bdc7f567bbd22c9e56ffdf003 by Victor Stinner in branch 'master':
bpo-32030: Add _PyMainInterpreterConfig_ReadEnv() (#4542)
https://github.com/python/cpython/commit/46972b7bc385ec2bdc7f567bbd22c9e56ffdf003
msg306929 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-11-24 23:01
New changeset f04ebe2a4d68b194deeb438e9185efdafc10a832 by Victor Stinner in branch 'master':
bpo-32030: Add _PyMainInterpreterConfig.program_name (#4548)
https://github.com/python/cpython/commit/f04ebe2a4d68b194deeb438e9185efdafc10a832
msg306935 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-11-25 02:17
New changeset 9316ee4da2dcc217351418fc4dbe9205995689e0 by Victor Stinner in branch 'master':
bpo-32030: Add _PyPathConfig_Init() (#4551)
https://github.com/python/cpython/commit/9316ee4da2dcc217351418fc4dbe9205995689e0
msg307239 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-11-29 16:20
New changeset 5d39e0429029324cae90bba2f19fb689b007c7d6 by Victor Stinner in branch 'master':
bpo-32030: Rework memory allocators (#4625)
https://github.com/python/cpython/commit/5d39e0429029324cae90bba2f19fb689b007c7d6
msg307268 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-11-29 22:51
New changeset 06be9daf6f03c1c65b9dd9896bc2b17f3c4bbc3a by Victor Stinner in branch '3.6':
bpo-32030: Fix test_sys.test_getallocatedblocks() (#4637)
https://github.com/python/cpython/commit/06be9daf6f03c1c65b9dd9896bc2b17f3c4bbc3a
msg307329 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-11-30 17:04
New changeset bc8ac6b00efcb3c601224b73f64071281f434bcd by Victor Stinner in branch 'master':
bpo-32030: Fix _Py_InitializeEx_Private() (#4649)
https://github.com/python/cpython/commit/bc8ac6b00efcb3c601224b73f64071281f434bcd
msg307395 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-12-01 17:27
New changeset b64de46aae148cfab0980e0ad478da7aafc44900 by Victor Stinner in branch 'master':
bpo-32030: Cleanup "path config" code (#4663)
https://github.com/python/cpython/commit/b64de46aae148cfab0980e0ad478da7aafc44900
msg307399 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-12-01 18:30
New changeset 9ac3d8882712c9675c3d2f9f84af6b5729575cde by Victor Stinner in branch 'master':
bpo-32030: Fix Py_GetPath(): init program_name (#4665)
https://github.com/python/cpython/commit/9ac3d8882712c9675c3d2f9f84af6b5729575cde
msg307402 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-12-01 19:09
New changeset ebac19dad6263141d5db0a2c923efe049dba99d2 by Victor Stinner in branch 'master':
bpo-32030: Don't call _PyPathConfig_Fini() in Py_FinalizeEx() (#4667)
https://github.com/python/cpython/commit/ebac19dad6263141d5db0a2c923efe049dba99d2
msg307407 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-12-01 19:51
New changeset 0ea395ae964c9cd0f499e2ef0d0030c971201220 by Victor Stinner in branch 'master':
bpo-32030: Add Python/pathconfig.c (#4668)
https://github.com/python/cpython/commit/0ea395ae964c9cd0f499e2ef0d0030c971201220
msg307410 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-12-01 20:19
New changeset e23c06e2b03452c9aaf0dae52296c85e572f9bcd by Victor Stinner in branch 'master':
bpo-32030: Fix config_get_program_name() on macOS (#4669)
https://github.com/python/cpython/commit/e23c06e2b03452c9aaf0dae52296c85e572f9bcd
msg307422 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-12-02 09:11
New changeset af5a895073c24637c094772b27526b94a12ec897 by Victor Stinner in branch 'master':
bpo-32030: _PyPathConfig_Init() sets home and program_name (#4673)
https://github.com/python/cpython/commit/af5a895073c24637c094772b27526b94a12ec897
msg307434 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-12-02 15:27
main.c now has compile errors on macOS:

../../source/Modules/main.c:904:20: error: expected expression
            return SET_DECODE_ERROR("PYTHONEXECUTABLE environment "
                   ^
../../source/Modules/main.c:46:5: note: expanded from macro 'SET_DECODE_ERROR'
    do { \
    ^
../../source/Modules/main.c:919:24: error: expected expression
                return SET_DECODE_ERROR("__PYVENV_LAUNCHER__ environment "
                       ^
../../source/Modules/main.c:46:5: note: expanded from macro 'SET_DECODE_ERROR'
    do { \
    ^
2 errors generated.
msg307453 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-12-02 19:36
New changeset 13badcbc60cdbfae1dba1683fd2fae9d70717143 by Serhiy Storchaka in branch 'master':
bpo-32197: Try to fix a compiler error on OS X introduced in bpo-32030. (#4681)
https://github.com/python/cpython/commit/13badcbc60cdbfae1dba1683fd2fae9d70717143
msg307560 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-12-04 12:39
New changeset 31a8393cf6a74c870c3484dd68500619f6232c6d by Victor Stinner in branch 'master':
Revert "bpo-32197: Try to fix a compiler error on OS X introduced in bpo-32030. (#4681)" (#4694)
https://github.com/python/cpython/commit/31a8393cf6a74c870c3484dd68500619f6232c6d
msg307665 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-12-05 14:12
New changeset 33c377ed9b6cb3b9493005314c4e0cfa7517ea65 by Victor Stinner in branch 'master':
bpo-32030: Simplify _PyCoreConfig_INIT macro (#4728)
https://github.com/python/cpython/commit/33c377ed9b6cb3b9493005314c4e0cfa7517ea65
msg307746 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-12-06 16:25
New changeset 672b6baa71010f236ee8c8ce912e98cb542385c6 by Victor Stinner in branch 'master':
bpo-32030: pass interp to _PyImport_Init() (#4736)
https://github.com/python/cpython/commit/672b6baa71010f236ee8c8ce912e98cb542385c6
msg307748 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-12-06 16:26
New changeset 6bf992a1ac6f3f4d0f83ead9c6403a76afdbe6eb by Victor Stinner in branch 'master':
bpo-32030: Add pymain_get_global_config() (#4735)
https://github.com/python/cpython/commit/6bf992a1ac6f3f4d0f83ead9c6403a76afdbe6eb
msg307756 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-12-06 17:13
New changeset 92a3c6f493ad411e4cf0acdf305ef4876aa90669 by Victor Stinner in branch 'master':
bpo-32030: Add _PyImport_Fini2() (#4737)
https://github.com/python/cpython/commit/92a3c6f493ad411e4cf0acdf305ef4876aa90669
msg308201 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-12-13 14:13
Wow, 28 PRs for a single issue! This is a record.
msg308202 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-12-13 14:47
> Wow, 28 PRs for a single issue! This is a record.

You can expect much more :-) One of the goal of the PEP 432 is to put compute sys.path and put it in _PyMainInterpreterConfig. I'm trying to implement that, but we are still far from being able to do it. At least, we are getting closer at each commit.

While it might be possible to squash 28 changes into a single change, I wouldn't be able to review it (I review my own changes on GitHub :-)), and it would very annoying if it causes any regression :-(
msg308212 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-12-13 16:31
New changeset d5dda98fa80405db82e2eb36ac48671b4c8c0983 by Victor Stinner in branch 'master':
pymain_set_sys_argv() now copies argv (#4838)
https://github.com/python/cpython/commit/d5dda98fa80405db82e2eb36ac48671b4c8c0983
msg308240 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-12-13 20:05
New changeset 11a247df88f15b51feff8a3c46005676bb29b96e by Victor Stinner in branch 'master':
bpo-32030: Add _PyPathConfig_ComputeArgv0() (#4845)
https://github.com/python/cpython/commit/11a247df88f15b51feff8a3c46005676bb29b96e
msg308271 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-12-14 01:20
New changeset b5fd9ad05e0f15f8272b8f1b829af22077230584 by Victor Stinner in branch 'master':
bpo-32030: Rewrite _PyMainInterpreterConfig (#4854)
https://github.com/python/cpython/commit/b5fd9ad05e0f15f8272b8f1b829af22077230584
msg308288 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-12-14 11:05
New changeset 374c6e178a7599aae46c857b17c6c8bc19dfe4c2 by Victor Stinner in branch 'master':
bpo-32030: Add _PyMainInterpreterConfig.warnoptions (#4855)
https://github.com/python/cpython/commit/374c6e178a7599aae46c857b17c6c8bc19dfe4c2
msg308349 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-12-14 23:51
New changeset 358e5e17a51ba00742bfaee4557a94c3c4179c22 by Victor Stinner in branch 'master':
bpo-32329: Fix -R option for hash randomization (#4873)
https://github.com/python/cpython/commit/358e5e17a51ba00742bfaee4557a94c3c4179c22
msg308356 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-12-15 00:46
New changeset da273412c4374de07a500e7f23f89a6bb7527398 by Victor Stinner in branch 'master':
bpo-32030: Add _PyCoreConfig_Copy() (#4874)
https://github.com/python/cpython/commit/da273412c4374de07a500e7f23f89a6bb7527398
msg308359 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-12-15 01:05
New changeset 41264f1cd4d6066b2797ff07cae465c1e06ff3b2 by Victor Stinner in branch 'master':
bpo-32030: Add _PyMainInterpreterConfig.executable (#4876)
https://github.com/python/cpython/commit/41264f1cd4d6066b2797ff07cae465c1e06ff3b2
msg308447 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-12-16 03:54
New changeset 9454060e84a669dde63824d9e2fcaf295e34f687 by Victor Stinner in branch 'master':
bpo-29240, bpo-32030: Py_Main() re-reads config if encoding changes (#4899)
https://github.com/python/cpython/commit/9454060e84a669dde63824d9e2fcaf295e34f687
msg308593 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-12-18 22:42
New changeset 6efcb6d3d5911aaf699f9df3bb3bc26e94f38e6d by Victor Stinner in branch 'master':
bpo-32030: Fix compilation on FreeBSD, #include <fenv.h> (#4919)
https://github.com/python/cpython/commit/6efcb6d3d5911aaf699f9df3bb3bc26e94f38e6d
msg308626 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-12-19 10:36
New changeset 5d8624647d0b8ccb22b17b9e819a8e0c3fb4fe4a by Victor Stinner in branch 'master':
bpo-32030: Fix compiler warnings (#4921)
https://github.com/python/cpython/commit/5d8624647d0b8ccb22b17b9e819a8e0c3fb4fe4a
msg308689 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-12-19 22:48
New changeset c4bca951065f4b2b6833f6ce7a0721e863e2343e by Victor Stinner in branch 'master':
bpo-32030: Add _PyCoreConfig.argv (#4934)
https://github.com/python/cpython/commit/c4bca951065f4b2b6833f6ce7a0721e863e2343e
msg308698 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-12-20 00:42
New changeset 19760863623b636a63ccf649107d9504c6465a92 by Victor Stinner in branch 'master':
bpo-32030: Cleanup pymain_main() (#4935)
https://github.com/python/cpython/commit/19760863623b636a63ccf649107d9504c6465a92
msg308700 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-12-20 01:55
I like the new shape of Py_Main(). The main parts of Py_Main() are now well identified:

* init "cmdline"
* init "python core"
* init "python main"
* run python
* cleanup
msg308750 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-12-20 17:00
New changeset ca719ac42b3d58f7c3bcdf63f45b6d62b08b0d01 by Victor Stinner in branch 'master':
bpo-32030: Add _PyCoreConfig.warnoptions (#4936)
https://github.com/python/cpython/commit/ca719ac42b3d58f7c3bcdf63f45b6d62b08b0d01
msg308766 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-12-20 18:36
New changeset 9cfc00262c7f6b93072762eed1dc5d94fa3897f0 by Victor Stinner in branch 'master':
bpo-32030: Complete _PyCoreConfig_Read() (#4946)
https://github.com/python/cpython/commit/9cfc00262c7f6b93072762eed1dc5d94fa3897f0
msg308833 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-12-20 22:41
New changeset 31e99080f6f8cf7faaba9fe3a4e0996e49163317 by Victor Stinner in branch 'master':
bpo-32030: Fix usage of memory allocators (#4953)
https://github.com/python/cpython/commit/31e99080f6f8cf7faaba9fe3a4e0996e49163317
msg308835 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-12-20 23:27
Summary of the visible changes:

* sys.argv is now set earlier, before "import site"

* The implementation of the PEP 538 now respects -E and -I options.

* _PyCoreConfig now allows to modify a lot of configuration options which wasn't possible to configure previously. (_PyMainInterpreterConfig allows to configure options using Python types.) It should be very useful for embedded Python, once the API will be public.


Summary of the most important changes:

* On Unix, char **argv of main() is now decoded again and the configuration (cmdline, env vars, etc.) is read again if the C locale is coerced (PEP 538) or if the UTF-8 Mode is enabled (PEP 540).

* The "path configuration" (sys.path, sys.prefix, sys.exec_prefix, etc.) is now computed before _Py_InitializeMainInterpreter(), by _PyMainInterpreterConfig_Read()

* Warning options are now computed in a list from -W options and PYTHONWARNINGS env var, before setting sys.warnoptions. Similar change has been done for sys._xoptions: xoptions are first stored in a wchar_t** list.

* A new _PyInitError structure has been introduced to be able to report failures instead of calling Py_FatalError(). Most of the Python initializatioin code has been modified to use it.

* Py_Main() now only uses regular C functions (using wchar_t* type for example) before _Py_InitializeCore(), the Python C API is now only used after _Py_InitializeCore().

* A new _PyCoreConfig_Read() has been added to read most configuration options using C types. _PyMainInterpreterConfig_Read() is now mostly responsible to convert C types to Python types.

* It becomes simpler to decide the priority between env vars, -X options, command line options, Py_xxx global variables, etc. The code to read these options and to code to "merge" these options is now decoupled.

* Each interpreter now has its copy of the "core_config" (_PyCoreConfig) and the main "config" (_PyMainInterpreterConfig). When using multiple interpreters, it is important to get Python objects created with the right interpreter for the main config. The ownership of these configurations becomes more obvious.

* Many global buffers with a fixed size like "static wchar_t progpath[MAXPATHLEN+1];" in Modules/getpath.c have been replaced with memory allocated on the heap with functions to release memory.

* Python/pathconfig.c (360 lines) has been added. More code is shared between Windows and Unix to compute the module search path (sys.path).
msg308844 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-12-21 03:39
Oh, the following command does crash:

PYTHONMALLOC=pymalloc ./Programs/_testembed forced_io_encoding
msg308874 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-12-21 14:45
New changeset e47e698da6bd982da277960c14afa9d9939e3155 by Victor Stinner in branch 'master':
bpo-32030: Add _Py_EncodeUTF8_surrogateescape() (#4960)
https://github.com/python/cpython/commit/e47e698da6bd982da277960c14afa9d9939e3155
msg308876 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-12-21 15:20
New changeset 9dd762013fd9fcf975ad51700b55d050ca9ed60e by Victor Stinner in branch 'master':
bpo-32030: Add _Py_EncodeLocaleRaw() (#4961)
https://github.com/python/cpython/commit/9dd762013fd9fcf975ad51700b55d050ca9ed60e
msg308877 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-12-21 15:49
New changeset 9bee329130aae5a13050c08dab9d349b76e66835 by Victor Stinner in branch 'master':
bpo-32030: Add _Py_FindEnvConfigValue() (#4963)
https://github.com/python/cpython/commit/9bee329130aae5a13050c08dab9d349b76e66835
msg308905 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-12-21 21:28
Hum, _PyCoreConfig.ignore_environment is redundant with Py_IgnoreEnvironmentFlag. I don't recall why I added it to _PyCoreConfig. Maybe it should be removed.
msg309053 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-12-26 00:31
_PyCoreConfig.ignore_environment was part of the initial PEP 432 implementation that I wrote.

It's that due to the design goal that once the refactoring is complete, an embedding application should be able to control *all* the settings through the config structs, without *ever* touching the global variables directly.
msg310606 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-01-24 16:03
New changeset 8ded5b803705328749622256701b3f08a9d6c5ab by Victor Stinner in branch 'master':
bpo-32030: Add _PyCoreConfig.module_search_paths (#4954)
https://github.com/python/cpython/commit/8ded5b803705328749622256701b3f08a9d6c5ab
msg310612 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-01-24 16:32
I made most, if not all, changes that I wanted to do. It's time to close this huge issue to continue the work in new more specific issues.

Notes on Py_Main().

(*) _PyPathConfig_Init() is called even if it's not needed (if all "Path configuration outputs" fileds of PyCoreConfig are filled).

(*) pymain_cmdline() uses _Py_CommandLineDetails structure which contains a copy of each global configuration variable like Py_UTF8Mode. Internally, the function has to "set" or "get" these variables when calling some functions like _PyPathConfig_Init().

This code is fragile. *But* pymain_read_conf() is complex, it had to read again the whole configuration a second time if the encoding changed.

It would be nice to remove global variables from _Py_CommandLineDetails to avoid the get/set dance which introduces a risk of loosing changes by mistake. But I'm not sure that it's doable?

(*) Should we make Py_FileSystemDefaultEncoding and Py_FileSystemDefaultEncodeErrors configurable in _PyCoreConfig? Same question for _Py_StandardStreamEncoding, _Py_StandardStreamErrors and PYTHONIOENCODING environment variable.
msg310613 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-01-24 16:32
I close the issue.
msg310618 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-01-24 16:56
Oh, my latest commit introduced a regression in test_distutils: bpo-32652.
msg319682 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-06-15 22:06
New changeset 6c5a4b315664f21bffc36ff6987fb4c4d1590897 by Victor Stinner in branch 'master':
bpo-33824, bpo-32030: Fix pymain_read_conf() (GH-7712)
https://github.com/python/cpython/commit/6c5a4b315664f21bffc36ff6987fb4c4d1590897
msg319685 - (view) Author: miss-islington (miss-islington) Date: 2018-06-15 22:26
New changeset 046da1669598d6112d0a6fb056081f3eb5a4d4e7 by Miss Islington (bot) in branch '3.7':
bpo-33824, bpo-32030: Fix pymain_read_conf() (GH-7712)
https://github.com/python/cpython/commit/046da1669598d6112d0a6fb056081f3eb5a4d4e7
msg328915 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-10-30 13:31
New changeset e1b29950bf751381538e3c8ea6a3e0a98d01dbfb by Victor Stinner in branch 'master':
bpo-32030: Make _PySys_AddXOptionWithError() private (GH-10236)
https://github.com/python/cpython/commit/e1b29950bf751381538e3c8ea6a3e0a98d01dbfb
History
Date User Action Args
2018-10-30 13:31:47vstinnersetmessages: + msg328915
2018-10-30 12:13:30vstinnersetpull_requests: + pull_request9548
2018-06-15 22:26:31miss-islingtonsetnosy: + miss-islington
messages: + msg319685
2018-06-15 22:07:49miss-islingtonsetpull_requests: + pull_request7343
2018-06-15 22:06:31vstinnersetmessages: + msg319682
2018-06-15 17:41:15vstinnersetpull_requests: + pull_request7327
2018-01-24 16:56:05vstinnersetmessages: + msg310618
2018-01-24 16:32:35vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg310613

stage: patch review -> resolved
2018-01-24 16:32:24vstinnersetmessages: + msg310612
2018-01-24 16:03:30vstinnersetmessages: + msg310606
2017-12-26 00:31:21ncoghlansetmessages: + msg309053
2017-12-21 21:28:30vstinnersetmessages: + msg308905
2017-12-21 15:49:20vstinnersetmessages: + msg308877
2017-12-21 15:23:29vstinnersetpull_requests: + pull_request4854
2017-12-21 15:20:34vstinnersetmessages: + msg308876
2017-12-21 14:48:48vstinnersetpull_requests: + pull_request4852
2017-12-21 14:45:21vstinnersetmessages: + msg308874
2017-12-21 14:27:07vstinnersetpull_requests: + pull_request4851
2017-12-21 03:39:29vstinnersetmessages: + msg308844
2017-12-21 00:50:00vstinnersetpull_requests: + pull_request4845
2017-12-20 23:27:16vstinnersetmessages: + msg308835
2017-12-20 22:41:41vstinnersetmessages: + msg308833
2017-12-20 21:52:40vstinnersetpull_requests: + pull_request4844
2017-12-20 18:36:48vstinnersetmessages: + msg308766
2017-12-20 17:48:47vstinnersetpull_requests: + pull_request4838
2017-12-20 17:00:26vstinnersetmessages: + msg308750
2017-12-20 03:00:55vstinnersetpull_requests: + pull_request4828
2017-12-20 01:55:47vstinnersetmessages: + msg308700
2017-12-20 00:42:01vstinnersetmessages: + msg308698
2017-12-20 00:04:15vstinnersetpull_requests: + pull_request4827
2017-12-19 22:48:19vstinnersetmessages: + msg308689
2017-12-19 22:26:01vstinnersetpull_requests: + pull_request4826
2017-12-19 10:36:04vstinnersetmessages: + msg308626
2017-12-19 01:13:44vstinnersetpull_requests: + pull_request4816
2017-12-18 22:42:57vstinnersetmessages: + msg308593
2017-12-18 14:27:51vstinnersetpull_requests: + pull_request4814
2017-12-16 03:54:25vstinnersetmessages: + msg308447
2017-12-16 03:10:30vstinnersetpull_requests: + pull_request4794
2017-12-15 01:05:33vstinnersetmessages: + msg308359
2017-12-15 00:47:14vstinnersetpull_requests: + pull_request4770
2017-12-15 00:46:04vstinnersetmessages: + msg308356
2017-12-14 23:51:24vstinnersetmessages: + msg308349
2017-12-14 23:39:06vstinnersetpull_requests: + pull_request4768
2017-12-14 23:16:46vstinnersetpull_requests: + pull_request4767
2017-12-14 16:01:12vstinnersetpull_requests: + pull_request4760
2017-12-14 11:05:28vstinnersetmessages: + msg308288
2017-12-14 01:29:40vstinnersetpull_requests: + pull_request4744
2017-12-14 01:20:54vstinnersetmessages: + msg308271
2017-12-14 00:58:31vstinnersetpull_requests: + pull_request4743
2017-12-13 20:05:59vstinnersetmessages: + msg308240
2017-12-13 18:18:16vstinnersetpull_requests: + pull_request4733
2017-12-13 16:31:18vstinnersetmessages: + msg308212
2017-12-13 14:47:05vstinnersetmessages: + msg308202
2017-12-13 14:13:52serhiy.storchakasetmessages: + msg308201
2017-12-13 14:04:28vstinnersetpull_requests: + pull_request4728
2017-12-06 17:13:01vstinnersetmessages: + msg307756
2017-12-06 16:27:26vstinnersetpull_requests: + pull_request4641
2017-12-06 16:26:12vstinnersetmessages: + msg307748
2017-12-06 16:25:57vstinnersetmessages: + msg307746
2017-12-06 15:54:01vstinnersetpull_requests: + pull_request4639
2017-12-06 15:43:25vstinnersetpull_requests: + pull_request4638
2017-12-05 14:12:47vstinnersetmessages: + msg307665
2017-12-05 13:44:29vstinnersetpull_requests: + pull_request4632
2017-12-04 12:39:17vstinnersetmessages: + msg307560
2017-12-04 09:40:10vstinnersetpull_requests: + pull_request4607
2017-12-02 19:36:02serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg307453
2017-12-02 18:50:57serhiy.storchakasetpull_requests: + pull_request4593
2017-12-02 18:18:43serhiy.storchakasetpull_requests: + pull_request4591
2017-12-02 15:27:07ned.deilysetnosy: + ned.deily
messages: + msg307434
2017-12-02 09:11:39vstinnersetmessages: + msg307422
2017-12-02 01:39:37vstinnersetpull_requests: + pull_request4581
2017-12-01 20:19:51vstinnersetmessages: + msg307410
2017-12-01 19:55:33vstinnersetpull_requests: + pull_request4578
2017-12-01 19:51:01vstinnersetmessages: + msg307407
2017-12-01 19:13:39vstinnersetpull_requests: + pull_request4577
2017-12-01 19:09:57vstinnersetmessages: + msg307402
2017-12-01 18:49:56vstinnersetpull_requests: + pull_request4576
2017-12-01 18:30:45vstinnersetmessages: + msg307399
2017-12-01 18:07:10vstinnersetpull_requests: + pull_request4575
2017-12-01 17:27:11vstinnersetmessages: + msg307395
2017-12-01 17:01:52vstinnersetpull_requests: + pull_request4573
2017-11-30 17:04:02vstinnersetmessages: + msg307329
2017-11-30 16:43:25vstinnersetpull_requests: + pull_request4561
2017-11-29 22:51:43vstinnersetmessages: + msg307268
2017-11-29 16:30:13vstinnersetpull_requests: + pull_request4552
2017-11-29 16:20:41vstinnersetmessages: + msg307239
2017-11-29 00:54:21vstinnersetpull_requests: + pull_request4540
2017-11-25 02:17:59vstinnersetmessages: + msg306935
2017-11-25 00:33:30vstinnersetpull_requests: + pull_request4482
2017-11-24 23:01:27vstinnersetmessages: + msg306929
2017-11-24 22:36:22vstinnersetpull_requests: + pull_request4480
2017-11-24 21:55:43vstinnersetmessages: + msg306924
2017-11-24 14:52:13vstinnersetpull_requests: + pull_request4475
2017-11-23 18:02:06vstinnersetmessages: + msg306851
2017-11-23 17:03:34vstinnersetpull_requests: + pull_request4462
2017-11-23 16:03:22vstinnersetmessages: + msg306824
2017-11-23 14:52:25vstinnersetpull_requests: + pull_request4458
2017-11-23 09:43:16vstinnersetmessages: + msg306794
2017-11-23 01:44:13vstinnersetpull_requests: + pull_request4450
2017-11-23 00:49:47vstinnersetmessages: + msg306775
2017-11-23 00:18:17ncoghlansetmessages: + msg306768
2017-11-23 00:16:43vstinnersetmessages: + msg306767
2017-11-23 00:14:37vstinnersetpull_requests: + pull_request4448
2017-11-22 23:56:08ncoghlansetmessages: + msg306763
2017-11-22 23:12:11vstinnersetmessages: + msg306759
2017-11-22 17:50:29vstinnersetpull_requests: + pull_request4442
2017-11-21 02:12:29vstinnersetmessages: + msg306616
2017-11-21 01:32:05vstinnersetpull_requests: + pull_request4422
2017-11-16 02:52:42vstinnersetmessages: + msg306332
2017-11-16 02:11:47vstinnersetmessages: + msg306329
2017-11-16 02:04:32vstinnersetmessages: + msg306328
2017-11-16 01:44:31vstinnersetpull_requests: + pull_request4362
2017-11-15 23:48:13vstinnersetmessages: + msg306321
2017-11-15 17:27:04vstinnersetmessages: + msg306288
2017-11-15 02:18:56ncoghlansetmessages: + msg306250
2017-11-15 01:41:45vstinnersetnosy: + ncoghlan, eric.snow
2017-11-15 01:41:33vstinnersetmessages: + msg306246
2017-11-15 01:37:05barrysetnosy: + barry
2017-11-15 01:36:05vstinnersettitle: Rewrite Py_Main() -> PEP 432: Rewrite Py_Main()
2017-11-15 01:31:01vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request4347
2017-11-15 01:30:24vstinnercreate