classification
Title: Document functions safe to be called before Py_Initialize()
Type: Stage: resolved
Components: Documentation Versions: Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: docs@python Nosy List: docs@python, eric.snow, ncoghlan, serhiy.storchaka, vstinner
Priority: normal Keywords: patch

Created on 2017-11-24 13:19 by vstinner, last changed 2017-12-15 15:09 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 4540 merged vstinner, 2017-11-24 13:22
PR 4621 closed vstinner, 2017-11-28 23:26
Messages (10)
msg306894 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-11-24 13:19
Follow-up of bpo-32086, bpo-32096 and "[Python-Dev] Python initialization and embedded Python" thread:
https://mail.python.org/pipermail/python-dev/2017-November/150605.html

I propose to explicitly list functions that can be safetely called before Py_Initialize(). This safety warranty must be part of the C API.

Maybe we should even test all tests function in test_capi using Programs/_testembed, as we did for Py_DecodeLocale() and Py_SetProgramName() in commit 9e87e7776f7ace66baaf7247233afdabd00c2b44 ("pre_initialization_api" test).

Attached PR adds proposed documentation. It also documents "global configuration variables" like Py_DebugFlag.
msg306898 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-11-24 14:09
Are you sure about PyMem_Malloc() and PyObject_Malloc()? What functions require them? I thought only PyMem_RawMalloc() can be called before Py_Initialize().

I think that for all functions that *can* or *should* be called before Py_Initialize() this should be explicitly documented in the documentation of this function, like for Py_SetProgramName() and PyImport_AppendInittab().
msg306902 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-11-24 16:05
> Are you sure about PyMem_Malloc() and PyObject_Malloc()?

Technically, the pymalloc memory allocator is initialized statically by the compiler, from the first instruction of the process.

But maybe we should not suggest users to call them, especially because the allocator can be modified by the PYTHONMALLOC environment variable.

> What functions require them?

No function to initalize Python require PyMem or PyObject allocators. Only PyMem_Raw allocator is needed.

Py_EncodeLocale() uses it, but this function also uses Python objects (str, bytes), and so Py_EncodeLocale() must no be called before Py_Initialize().
msg306912 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-11-24 18:28
Should PyMem_SetAllocator() and PyObject_SetArenaAllocator() be called before Py_Initialize(), or they can be called after it?

If PyMem_Malloc() and PyObject_Malloc() are not needed for pre-initialization, should we support calling them before Py_Initialize()? There are other functions and macros that can be safely used before Py_Initialize().
msg306916 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-11-24 20:38
> Should PyMem_SetAllocator() and PyObject_SetArenaAllocator() be called before Py_Initialize(), or they can be called after it?

I'm quite sure that calling PyMem_SetAllocator() or PyObject_SetArenaAllocator() after Py_Initialize() will quickly crash.

> If PyMem_Malloc() and PyObject_Malloc() are not needed for pre-initialization, should we support calling them before Py_Initialize()?

We don't have to support them.

Ok, I remove them from the pre-init documentation.
msg306917 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-11-24 20:43
> I'm quite sure that calling PyMem_SetAllocator() or
> PyObject_SetArenaAllocator() after Py_Initialize() will quickly crash.

Then document this explicitly like for other functions that *should* be called 
before Py_Initialize() if called at all.
msg306921 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-11-24 21:30
New changeset 84c4b1938fade2b425ac906730beabd413de094d by Victor Stinner in branch 'master':
bpo-32124: Document C functions safe before init (#4540)
https://github.com/python/cpython/commit/84c4b1938fade2b425ac906730beabd413de094d
msg306922 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-11-24 21:33
> Then document this explicitly like for other functions that *should* be called before Py_Initialize() if called at all.

I agree that it would be even better to document if a function must not be called after Py_Initialize().

*But* I'm not sure of what I wrote, I have to check the code, and maybe even test manually to "see what happens" (ensure that it works) :-)

So I decided to push my first PR, and will work on a second PR later.
msg306925 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2017-11-24 22:05
I've left a review (writing it as you merged the PR).

My main concern is that we not promise more than we must.  Every pre-init function or variable we promise to embedders represents global state that is hard to get rid of.  It also entrenches pre-init API and state that we're aiming to deprecate (via PEP 432).
msg306937 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-11-25 03:32
Key point to note regarding PEP 432: at least personally, I'm not actually aiming to deprecate the legacy embedding API.

Instead, I'm just aiming to eventually stop *adding* to it, with new config structs replacing the current ad hoc mix of pre-init function calls, C globals, environment variables, and filesystem state.

That means I'm quite willing to accept maintaining compatibility for applications using the current single phase initialisation approach as a design constraint for the PEP.

We have a similar constraint in place for extension modules: 
even though any *new* features we introduce are likely to be dependent on switching over to PEP 489's multi-phase initialisation APIs, we still ensure that single-phase initialisation continues working for existing modules.
History
Date User Action Args
2019-03-21 23:05:12cheryl.sabellalinkissue31470 superseder
2017-12-15 15:09:10vstinnersetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2017-11-28 23:26:20vstinnersetpull_requests: + pull_request4536
2017-11-25 03:32:11ncoghlansetmessages: + msg306937
2017-11-24 22:05:36eric.snowsetmessages: + msg306925
2017-11-24 21:33:03vstinnersetmessages: + msg306922
2017-11-24 21:30:29vstinnersetmessages: + msg306921
2017-11-24 20:43:35serhiy.storchakasetmessages: + msg306917
2017-11-24 20:38:22vstinnersetmessages: + msg306916
2017-11-24 18:28:07serhiy.storchakasetmessages: + msg306912
2017-11-24 16:05:00vstinnersetmessages: + msg306902
2017-11-24 14:09:12serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg306898
2017-11-24 13:22:20vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request4474
2017-11-24 13:19:16vstinnercreate