Message289668
For PEP 538, setting PYTHONIOENCODING turned out to have undesirable side effects on Python 2 instances in subprocesses, since Python 2 has no 'surrogateescape' error handler.
So I switched to using the "Py_SetStandardStreamEncoding" API defined in http://bugs.python.org/issue16129 instead, but this turns out to have problematic interactions with the dynamic memory allocator management, so it fails with a fatal exception in debug mode. An example of the error can be seen here: https://travis-ci.org/python/cpython/jobs/211293576
The problem appears to be that between the allocation of the memory with `_PyMem_RawStrdup` in `Py_SetStandardStreamEncoding` and the release of that memory in `initstdio`, the active memory manager has changed (at least in a debug build), so the deallocation as part of the interpreter startup fails.
That interpretation is based on this comment in Programs/python.c:
```
/* Force again malloc() allocator to release memory blocks allocated
before Py_Main() */
(void)_PyMem_SetupAllocators("malloc");
```
The allocations in Py_SetStandardStreamEncoding happen before the call to Py_Main/Py_Initialize, but the deallocation happens in Py_Initialize.
The "fix" I applied to the PEP branch was to make the default allocator conditional in Programs/python.c as well:
```
#ifdef Py_DEBUG
(void)_PyMem_SetupAllocators("malloc_debug");
# else
(void)_PyMem_SetupAllocators("malloc");
# endif
```
While that works (at least in the absence of a PYTHONMALLOC setting) it seems fragile. It would be nicer if there was a way for Py_SetStandardStreamEncoding to indicate which allocator should be used for the deallocation. |
|
Date |
User |
Action |
Args |
2017-03-15 11:27:40 | ncoghlan | set | recipients:
+ ncoghlan, vstinner |
2017-03-15 11:27:39 | ncoghlan | set | messageid: <1489577259.98.0.691600629889.issue29818@psf.upfronthosting.co.za> |
2017-03-15 11:27:39 | ncoghlan | link | issue29818 messages |
2017-03-15 11:27:39 | ncoghlan | create | |
|