imp.find_module() seems to cause an UnicodeDecodeError when the path
list contains paths with non-ASCII names. Tested on Windows [1]; see the
attached test case which demonstrates the problem.

[1] Python 3.0rc2 (r30rc2:67141, Nov 7 2008, 11:43:46) [MSC v.1500 32
bit (Intel)] on win32

The example works correctly on Linux (py3k trunk). The problem is maybe
specific to Windows?

Indeed. It happens when the filesystem encoding is not utf-8.

I have several changes in my local workspace about this, which also deal
with zipimport and other places that import modules.
I suggest to let 3.0 go out and correct all this for 3.1.

Indeed. It happens when the filesystem encoding is not utf-8.

How can I test it on Linux?

Oh, I found sys.setfilesystemencoding("latin-1")! But even with that,
your example find_module.py works correctly with py3k trunk. The
problem has maybe gone?

Well, latin-1 can decode any arbitrary array of bytes, so of course it
won't fail. :)

I can reproduce this problem on Windows Vista, fresh py3k sources.
Looks like bug occurs only with Latin-1 characters.
At least Cyrillic works ok.

From my understanding (after tracing/debugging) problem lies in import.c
find_module tries to convert path from unicode to bytestring using Py_FileSystemDefaultEncoding (line 1397). For Windows it is 'mbcs'.

Conversion done with decode_mbcs (unicodeobject.c:4244) what uses MultiByteToWideChar with codepage CP_ACP. Problem is: converting
composite characters ('\u00e4' is 'a'+'2 dots over letter', I don't know
true name for this sign) this function returns only 'a'.

>>> repr('h\u00e4kkinen'.encode('mbcs'))
"b'hakkinen'"

MSDN says (http://msdn.microsoft.com/en-
us/library/dd374130(VS.85).aspx):
For strings that require validation, such as file, resource, and user
names, the application should always use the WC_NO_BEST_FIT_CHARS flag
with WideCharToMultiByte. This flag prevents the function from mapping
characters to characters that appear similar but have very different
semantics. In some cases, the semantic change can be extreme. For
example, the symbol for "∞" (infinity) maps to 8 (eight) in some code
pages.

Writing encoding function in opposite to PyUnicode_DecodeFSDefault with
setting this flag also cannot help - problematic character just replaced
with 'default' ('?' if not specified).
Hacking specially for 'latin-1' encoding sounds ugly.

Changing all filenames to unicode (with possible usage of fileio instead
of direct calls of open/fdopen) in import.c looks good for me but takes
long time and makes many changes.

At the sprint, Andrew Svetlov, Martin von Loewis and I looked into this
a bit, and discovered that Andrew's Vista copy uses a Russian locale for
the filesystem encoding (despite using English as the language). In
this locale, a-umlaut cannot be represented in the ANSI code page (which
has only 256 values), because the Russian locale uses those byte values
to represent Cyrillic.

As long as the import code (written in C) uses bytes in the filesystem
encoding to represent paths, this problem will remain.

Two possible solutions would be to switch to Brett's importlib, or to
change the import code to use wide characters everywhere (like
posixmodule.c). Both are extremely risky and a lot of work, and I don't
expect we'll get to this for 3.1.

(In 2.x the same problem exists, but is perhaps less real because module
names are limited to ASCII.)

We also discovered another problem, which I'll report separately: the
*module* name is decoded to UTF8, while the *path* name uses the
filesystem encoding...

See also bpo-8611.

About the mbcs encoding: issue bpo-850997 proposes to make it more strict.

I closed issue bpo-850997, mbcs is now really strict by default:

>>> 'h\u00e4kkinen'.encode('mbcs')
UnicodeEncodeError: ...
>>> 'h\u00e4kkinen'.encode('mbcs', 'replace')
"b'hakkinen'"

PyUnicode_EncodeFSDefault(), PyUnicode_DecodeFSDefault() and os.fsencode() use mbcs with strict error handler on Windows. On other OS, these functions use surrogateescape error handler, but mbcs only supports strict and replace (to encode, and strict and ignore to decode).

I wrote a patch to fix this issue, see bpo-9425.

Good news: this issue is now fixed in py3k (Python 3.2). I cannot give a commit number, because there are too much commits related to this problem (see bpo-8611 and bpo-9425), but it works ;-)

@serg Asminog: What is your Python version? What is your locale encoding (print(sys.getfilesystemencoding())? What is your Windows version?

print(sys.getfilesystemencoding())
print(os.name)
print(sys.version)
print(sys.version_info)
print(sys.platform)

mbcs
nt
3.2.2 (default, Sep 4 2011, 09:07:29) [MSC v.1500 64 bit (AMD64)]
sys.version_info(major=3, minor=2, micro=2, releaselevel='final', serial=0)
win32

-----------
Windows 7 64bit

Also

Traceback (most recent call last):
  File "D:\temp\python bug\test.py", line 20, in <module>
    file_object, file_path, description = imp.find_module(basename, [dirname])
ImportError: No module named mymodule

with python  2.6.6 (r266:84297, Aug 24 2010, 18:13:38) [MSC v.1500 64 bit (AMD64)]

Oops, it's not sys.getfilesystemencoding(), but locale.getpreferredencoding() which is interesting. Can you give me your locale encoding?

cp1251