Created on 2017-03-15 11:27 by ncoghlan, last changed 2017-03-15 11:28 by ncoghlan.
|msg289668 - (view)||Author: Nick Coghlan (ncoghlan) *||Date: 2017-03-15 11:27|
For PEP 538, setting PYTHONIOENCODING turned out to have undesirable side effects on Python 2 instances in subprocesses, since Python 2 has no 'surrogateescape' error handler. So I switched to using the "Py_SetStandardStreamEncoding" API defined in http://bugs.python.org/issue16129 instead, but this turns out to have problematic interactions with the dynamic memory allocator management, so it fails with a fatal exception in debug mode. An example of the error can be seen here: https://travis-ci.org/python/cpython/jobs/211293576 The problem appears to be that between the allocation of the memory with `_PyMem_RawStrdup` in `Py_SetStandardStreamEncoding` and the release of that memory in `initstdio`, the active memory manager has changed (at least in a debug build), so the deallocation as part of the interpreter startup fails. That interpretation is based on this comment in Programs/python.c: ``` /* Force again malloc() allocator to release memory blocks allocated before Py_Main() */ (void)_PyMem_SetupAllocators("malloc"); ``` The allocations in Py_SetStandardStreamEncoding happen before the call to Py_Main/Py_Initialize, but the deallocation happens in Py_Initialize. The "fix" I applied to the PEP branch was to make the default allocator conditional in Programs/python.c as well: ``` #ifdef Py_DEBUG (void)_PyMem_SetupAllocators("malloc_debug"); # else (void)_PyMem_SetupAllocators("malloc"); # endif ``` While that works (at least in the absence of a PYTHONMALLOC setting) it seems fragile. It would be nicer if there was a way for Py_SetStandardStreamEncoding to indicate which allocator should be used for the deallocation.
|2017-03-15 11:28:01||ncoghlan||set||versions: + Python 3.6, Python 3.7|