There's a path in imp.find_module that mixes encodings.  The module name
is encoded to char* using UTF-8 by the 's' format passed to
PyArg_ParseTuple().  But the path name is converted (in the loop over
the path in find_module()) to char* using the filesystem encoding.  On
Windows this ends up constructing a char* that mixes MBCS and UTF8 in
one string.

(We discovered this when researching bug 4352, but this is not the cause
of the problem reported there -- the module name in that bug is ASCII.)

Andrew Svetlov is looking into producing a patch.

Problem fixed, patch attached
I inserted conversion path parameters to using Py_FileSystemDefaultEncoding for:

* load_module
* load_compiled
* load_dynamic
* load_source
* load_package

find_module is already has conversion.

PyMem_Free is needed when "es" is used with PyArg_ParseTuple. See other
part of import.c. I did same mistake before. ;-)

Thank you.

On Mon, Mar 30, 2009 at 3:13 PM, Hirokazu Yamamoto
<report@bugs.python.org> wrote:
>
> Hirokazu Yamamoto <ocean-city@m2.ccsnet.ne.jp> added the comment:
>
> PyMem_Free is needed when "es" is used with PyArg_ParseTuple. See other
> part of import.c. I did same mistake before. ;-)
>
> ----------
> nosy: +ocean-city
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue5604>
> _______________________________________
>

According to Hirokazu Yamamoto memory cleanup added.
Patch is updated.

Thanks Andrew!  Committed to 3.0.2 as 70756.

Should be merged into 3.1, but should *not* be backported to 2.x.

Continuing work over import.c I fixed bad error message encoding in 
generated ImportError exception.
Tests for checking in case of non-ascii characters added.

Martin von Loewis added to nosy list

Martin, can you review latest patch and apply it if this one is correct.

I want to start working on conversion import.c to use unicode strings 
(we spoke about Tuesday) this weekend.
It will be nice if I will have synchronized svn before making new 
changes.

Continuing work on problem I figured out:
* on Windows it's impossible to convert filenames to file system 
encoding without and don't miss something.
* Windows can work properly only with unicode (wchar_t) characters.
* all other systems feels itself good using utf-8 (or another filesystem 
encoding).
* it's very errorprone to change 'char*' to 'PyUnicode*'.

To solve this problem I assume: 
* all char* in Python API is utf-8.
* if there are need call to operation system api like fopen - call 
imp_fopen, this function will do need conversions. Inside import.c there 
are 4 calls: fopen, open_exclusive, unlink, stat. I want to write stubs 
for this calls. 
* also loaders for dynamic modules aka 'C extensions' have to expect 
utf-8 as  pathname parameter, not 'filesystem encoded'.

Patch for windows is applied (STILL NOT CONVERTED TO OTHER OS).
But for Windows it works (regression tests passed).

If this solution is applicable for 3.1 (as I know Cannon works on excellent importlib but this library will replace imp functionality only 
in 3.2) - I can continue switching. Unfortunately I cannot test py3k 
trunk on non-windows machines - but I can 'make all OS calls as 
expected' and wait for buildbot answer.

Please review import_patch_4th_edition.zip and if I ran in wrong way - 
let me know.

Still an issue for some buildbot:

http://www.python.org/dev/buildbot/all/builders/x86%20XP-4%203.x/builds/1487
http://www.python.org/dev/buildbot/all/builders/x86%20XP-4%203.x/builds/1491


It is loosely related with #7712, because now the tests are run in TEMP (on C:) instead of the build dir (on D:).

Also the test has a few problems:
1) the keys of known_locales are lowercase, but locale_encoding = locale.getpreferredencoding() can return uppercase encodings (e.g. UTF-8);
2) this masks another error: the b'\xe4' is not a valid utf-8 byte and it can be decoded;
3) the test should be skipped properly if the preferred encoding is not among the ones of the known_locales dict;
4) the 'encoded_char' should be 'decoded_char'.

It seems that in the failure linked by Florent, find_module tries to encode the filename with the wrong encoding and with error='replace' and the char at the end of 'test_imp_helper_' is converted to U+FFFD.
If the file is created with the correct name (e.g. 'test_imp_helper_\xc0') and find_module tries to search the wrong name (i.e. 'test_imp_helper_\ufffd'), then the error is raised (but then cp1252 used by the terminal can't encode that char and the second exception is raised).
Now the tests are run on C: and the filesystem encoding might be different, so it might not match anymore the encoding returned by locale.getpreferredencoding().

I fixed all the things listed in the previous message in r78689, but that just enabled the test on several Linux buildbots and some started to fail too.
In r78696 (and r78697) I tried to use sys.getfilesystemencoding() instead of locale.getpreferredencoding() and that turned at least the Windows buildbots green, but there are still a few linux bots that fail:
http://www.python.org/dev/buildbot/builders/AMD64%20Ubuntu%20wide%203.x/builds/545/steps/test/logs/stdio
http://www.python.org/dev/buildbot/builders/AMD64%20Ubuntu%203.x/builds/561/steps/test/logs/stdio
http://www.python.org/dev/buildbot/builders/x86%20Ubuntu%203.x/builds/525/steps/test/logs/stdio
http://www.python.org/dev/buildbot/builders/amd64%20gentoo%203.x/builds/513/steps/test/logs/stdio

Thanks for fixing this. Now Win7 buildbot is green on trunk.

The Linux buildbots were running the tests using ./python ./Lib/test/regrtest.py instead of ./python -m test.regrtest and '' was missing from sys.path, so imp.find_module couldn't find the module.
This is now fixed in r78711 and backported r78716.
Thanks to Florent that noticed that those buildbots were using a different command to run the tests.