Python 3.0 doesn't run from a directory with umlauts and possible other
non ASCII chars.

I renamed my development folder from C:\dev\ to c:\test äöüß name\.
Python crashes after a few moments before it can reach its shell.

 	python30.dll!PyErr_SetObject(_object * exception=0x1e1b9888, _object *
value=0x00a0b8a0)  Zeile 56 + 0xb Bytes	C
 	python30.dll!PyErr_SetString(_object * exception=0x1e1b9888, const
char * string=0x1e18c358)  Zeile 77 + 0xd Bytes	C
 	python30.dll!find_module(char * fullname=0x0021fcc0, char *
subname=0x00000000, _object * path=0x00000000, char * buf=0x0021fb70,
unsigned int buflen=257, _iobuf * * p_fp=0x0021fb64, _object * *
p_loader=0x0021fb68)  Zeile 1228 + 0x10 Bytes	C
 	python30.dll!import_submodule(_object * mod=0x1e1c6a88, char *
subname=0x0021fcc0, char * fullname=0x00000000)  Zeile 2313 + 0x27 Bytes	C
 	python30.dll!load_next(_object * mod=0x1e1c6a88, _object *
altmod=0x1e1c6a88, char * * p_name=0x00000000, char * buf=0x0021fcc0,
int * p_buflen=0x0021fcbc)  Zeile 2127 + 0x15 Bytes	C
 	python30.dll!import_module_level(char * name=0x00000000, _object *
globals=0x00000000, _object * locals=0x1e069ec3, _object *
fromlist=0x00000000, int level=0)  Zeile 1908 + 0x1a Bytes	C
 	python30.dll!PyImport_ImportModuleLevel(char * name=0x1e184b04,
_object * globals=0x00000000, _object * locals=0x00000000, _object *
fromlist=0x00000000, int level=0)  Zeile 1979 + 0x18 Bytes	C
 	python30.dll!_PyCodecRegistry_Init()  Zeile 841 + 0x12 Bytes	C
 	python30.dll!PyCodec_LookupError(const char * name=0x00000000)  Zeile
436 + 0xc Bytes	C
 	python30.dll!unicode_decode_call_errorhandler(const char *
errors=0x00000000, _object * * errorHandler=0x00000009, const char *
encoding=0x1e1979ec, const char * reason=0x00000000, const char * *
input=0x0021fe80, const char * * inend=0x0021fe84, int *
startinpos=0x0021fe6c, int * endinpos=0x0021fe68, _object * *
exceptionObject=0x00000000, const char * * inptr=0x0021fe90, _object * *
output=0x0021fe70, int * outpos=0x0021fe88, unsigned short * *
outptr=0x0021fe74)  Zeile 1384 + 0xa Bytes	C
 	python30.dll!PyUnicodeUCS2_DecodeUTF8Stateful(const char *
s=0x1e1dd010, int size=48, const char * errors=0x00000000, int *
consumed=0x00000000)  Zeile 1967 + 0x47 Bytes	C
 	python30.dll!PyUnicodeUCS2_FromStringAndSize(const char *
u=0x1e1dd008, int size=48)  Zeile 464 + 0xb Bytes	C
 	python30.dll!PyUnicodeUCS2_FromString(const char * u=0x1e1dd008) 
Zeile 482 + 0x7 Bytes	C
 	python30.dll!_PySys_Init()  Zeile 1084 + 0xb Bytes	C
 	python30.dll!Py_InitializeEx(int install_sigs=1)  Zeile 220	C
 	python30.dll!Py_Initialize()  Zeile 292 + 0x7 Bytes	C
 	python30.dll!Py_Main(int argc=2, char * * argv=0x00000001)  Zeile 432	C
>	python.exe!mainCRTStartup()  Zeile 398 + 0xe Bytes	C
 	kernel32.dll!7c816fd7()

The patch fixes parts of the problem. At least Python doesn't crash any
more when run from a directory with non ASCII chars. It just fails with
an import error in initstdio().

I've added a fprintf(stderr, "%s", path) to makepathobject(). I suspect
that PC/getpathp.c doesn't handle non ASCII chars correctly. It's using
char instead of w_char all over the place. Could that be related to the
issue, Neal?

Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

c:\testäöü\PCBuild8\win32release>set PYTHONPATH=c:\testäöü\Lib

c:\testäöü\PCBuild8\win32release>python
c:\testõ÷³\Lib;c:\testõ÷³\PCBuild8\win32release\python30.zip;c:\testõ÷³\DLLs;c:\
testõ÷³\lib;c:\testõ÷³\lib\plat-win;c:\testõ÷³\lib\lib-tk;c:\testõ÷³\PCBuild8\wi
n32releaseFatal Python error: Py_Initialize: can't initialize sys
standard strea
ms
object  : ImportError('No module named encodings.utf_8',)
type    : ImportError
refcount: 4
address : 00A43540
lost sys.stderr

This application has requested the Runtime to terminate it in an unusual
way.
Please contact the application's support team for more information.

The bug is related to http://bugs.python.org/issue1262

Hi Martin!

Thomas Wouters said on #python that you have the Windows Fu to fix the
problem. Parts of the Python API for file paths, sys.path and os.environ
have to be reimplemented using the wide char API.

I've checked in part of the patch in r58837. It doesn't solve the
problem but at least it prevents Python from seg faulting on Windows.

I like to move _PyExc_Init() before _PySys_Init() and set sys.prefix,
exec_prefix and executable with PyUnicode_DecodeFSDefault().

Without the changes Python is seg faulting on Windows when the path
contains non ASCII chars. With the patch it is failing with a fatal
error which is a tiny bit nicer.

If this doesn't cause any problems on other platforms, go for it.

I'm setting the priority to normal. The issue isn't resolved but it's
not critical for the next alpha release. By the way what's your ETA for
the next alpha, Guido?

Assign to myself.
Among the things to do, use Py_FileSystemDefaultEncoding (=mbcs on
Windows) to encode sys.path items; likewise in NullImporter_init and
other functions.
So many places to change, we need serious testcases.

Please don't use the FileSystemEncoding on Windows for sys.path items.
Instead, it should use the wide API to perform all system calls. Py3k
shouldn't ever use the file system encoding for anything on Windows.

Agreed. I will try to stay with PyObjects* until really needed by a system call.

I'm increasing the severity of the bug. It's a still a major show
stopper for non-English Windows users. For example see #2780

There are some problems under Linux too:

$ pwd
/home/antoine/py3k/héhé
$ ./python
Fatal Python error: Py_Initialize: can't initialize sys standard streams
Traceback (most recent call last):
  File "/home/antoine/py3k/pristine/Lib/encodings/__init__.py", line 32,
in <module>
TypeError: zipimporter() argument 1 must be string without null bytes,
not str
Abandon

See #2798 for the non-Windows case, with a patch.

Here is a quick fix, that decodes filenames using 
Py_FileSystemDefaultEncoding, to let the release pass.

I am still working on a version that keep PyObjects* as long as
possible, but it will be a major change.

Fixed as r64126, using Py_FileSystemDefaultEncoding.

I close this issue, and open issue3080 to rewrite all functions in
import.c with full unicode in mind.