This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author ncoghlan
Recipients ncoghlan, vstinner
Date 2017-03-15.11:27:39
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1489577259.98.0.691600629889.issue29818@psf.upfronthosting.co.za>
In-reply-to
Content
For PEP 538, setting PYTHONIOENCODING turned out to have undesirable side effects on Python 2 instances in subprocesses, since Python 2 has no 'surrogateescape' error handler.

So I switched to using the "Py_SetStandardStreamEncoding" API defined in http://bugs.python.org/issue16129 instead, but this turns out to have problematic interactions with the dynamic memory allocator management, so it fails with a fatal exception in debug mode. An example of the error can be seen here: https://travis-ci.org/python/cpython/jobs/211293576

The problem appears to be that between the allocation of the memory with `_PyMem_RawStrdup` in `Py_SetStandardStreamEncoding` and the release of that memory in `initstdio`, the active memory manager has changed (at least in a debug build), so the deallocation as part of the interpreter startup fails.

That interpretation is based on this comment in Programs/python.c:

```
    /* Force again malloc() allocator to release memory blocks allocated
       before Py_Main() */
    (void)_PyMem_SetupAllocators("malloc");
```

The allocations in Py_SetStandardStreamEncoding happen before the call to Py_Main/Py_Initialize, but the deallocation happens in Py_Initialize.

The "fix" I applied to the PEP branch was to make the default allocator conditional in Programs/python.c as well:

```
#ifdef Py_DEBUG
    (void)_PyMem_SetupAllocators("malloc_debug");
#  else
    (void)_PyMem_SetupAllocators("malloc");
#  endif
```

While that works (at least in the absence of a PYTHONMALLOC setting) it seems fragile. It would be nicer if there was a way for Py_SetStandardStreamEncoding to indicate which allocator should be used for the deallocation.
History
Date User Action Args
2017-03-15 11:27:40ncoghlansetrecipients: + ncoghlan, vstinner
2017-03-15 11:27:39ncoghlansetmessageid: <1489577259.98.0.691600629889.issue29818@psf.upfronthosting.co.za>
2017-03-15 11:27:39ncoghlanlinkissue29818 messages
2017-03-15 11:27:39ncoghlancreate