note, I was asked to report this issue, posted on the py dev mailing list: see - http://code.activestate.com/lists/python-dev/118015/

---

We've run into an issue recently with blender3d on ms-windows where we
want to enforce the encoding is UTF-8 with the embedded python
interpreter.
(the encoding defaults to cp437).

I naively thought setting the environment variable before calling
Py_Initialize() would work, but the way python DLL loads, it gets its
own environment variables that cant be modified directly [1].
eg, _putenv("PYTHONIOENCODING=utf-8:surrogateescape");

We had bug reports by windows users not able to export files because
the stdout errors on printing paths with unsupported encoding. [2],[3]

---

Of course we could distribute blender with a bat file launcher that
sets env variables, or ask the user to set their env variable - but I
dont think this is really a good option.

I tried overriding the stderr & stdout, but this caused another bug on exiting, giving an assert in MSVCR90.DLL's write.c (called from python32_d.dll):
_VALIDATE_CLEAR_OSSERR_RETURN((_osfile(fh) & FOPEN), EBADF, -1);

import sys, io
sys.__stdout__ = sys.stdout =
io.TextIOWrapper(io.open(sys.stdout.fileno(), "wb", -1),
encoding='utf-8', errors='surrogateescape', newline="\n",
line_buffering=True)
sys.__stderr__ = sys.stderr =
io.TextIOWrapper(io.open(sys.stderr.fileno(), "wb", -1),
encoding='utf-8', errors='surrogateescape', newline="\n",
line_buffering=True)



IMHO either of these solutions would be fine.

* have a PyOS_PutEnv() function, gettext has gettext_putenv() to
workaround this problem.

* manage this the same as Py_GetPythonHome(), which can be defined by
the embedding application to override the default.


[1] http://stackoverflow.com/questions/5153547/environment-variables-are-different-for-dll-than-exe
[2] http://projects.blender.org/tracker/index.php?func=detail&aid=32750
[3] http://projects.blender.org/tracker/index.php?func=detail&aid=31555

See also issue #15216.

> IMHO either of these solutions would be fine.
> 
> * have a PyOS_PutEnv() function, gettext has gettext_putenv() to
> workaround this problem.

This solution would help in many other cases as well, so adding
such an API would certainly help more than specialized interfaces.

> * manage this the same as Py_GetPythonHome(), which can be defined by
> the embedding application to override the default.

I think you meant Py_SetPythonHome().

Given that the IO encoding is very important for Python 3.x, a special
API just for setting the encoding may be useful to have as well.

Care must be taken, though, that the encoding cannot be set after
Py_Initialize() has been called.

It may overall be easier to go with the PyOS_PutEnv() solution to
not run into the problems with having to check for an initialized
interpreter first.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Oct 05 2012)
>>> Python Projects, Consulting and Support ...   http://www.egenix.com/
>>> mxODBC.Zope/Plone.Database.Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2012-09-27: Released eGenix PyRun 1.1.0 ...       http://egenix.com/go35
2012-09-26: Released mxODBC.Connect 2.0.1 ...     http://egenix.com/go34
2012-09-25: Released mxODBC 3.2.1 ...             http://egenix.com/go33
2012-10-23: Python Meeting Duesseldorf ...                 18 days to go

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

Agree PyOS_PutEnv would be good since its not restricted to string encoding and resolves the problem of not being able to control env vars for an embedded interpreter in general.

Having ways to change encoding is good too but a bit outside the scope of this report and possibly not the best solution either since its possible (through unlikely), that you need to set the encoding at the very start of python initialization- rather than site & builtin modules loads.

patch attached, simply wraps putenv()

At first glance your proposed fix looks like an easy hack to get around the issue. However it's not going to work properly. Embedded Python interpreters should isolate themselves from the user's environment. When `Py_IgnoreEnvironmentFlag` is enabled, Py_GETENV() always returns NULL.

IMHO we can't get around Py_GetIOEncoding(), Py_SetIOEncoding() and Py_IOEncoding.

Claiming this one, mainly because I want people to largely leave the already hairy initialisation process alone until we get a chance to discuss it at the language summit next year.

I plan to write up a comprehensive overview of the initialisation sequence before then, because we need to be figuring out how to *delete* code here, instead of adding even more.