From: https://code.google.com/p/psutil/issues/detail?id=416


# -*- coding: utf-8 -*-
from __future__ import unicode_literals
import os, errno
name = "ƒőő"
try:
    os.mkdir(name)
except OSError as err:
    if err.errno != errno.EEXIST:
        raise
os.statvfs(name)


The script above works fine on Python 3.3 but on 2.7 you'll get:

Traceback (most recent call last):
  File "foo.py", line 10, in <module>
    os.statvfs(name)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-2: ordinal not in range(128)

Patch in attachment fixes the issue.

Functions such as rename(), popen(), mkfifo(), mknod(), etc have the same issue.

> The script above works fine on Python 3.3 but on 2.7 you'll get: ...

Cool, you now have a good reason to upgrade to Python 3 ;-)

I'm not sure that it's a good idea to invest time on fixing Unicode issues in Python 2, especially in a minor version (Python 2.7.x).

> I'm not sure that it's a good idea to invest time on fixing Unicode 
> issues in Python 2, especially in a minor version (Python 2.7.x).

I admit I sort of share the same doubts, but considering 2.7 a "minor python version", especially at this point, would be a mistake.

2.7.x is a minor version, not 2.7. We fixed Unicode issues in Python 2 bugfixes many times.

Giampaolo, do you want to provide a test?

Attached patch includes tests. I took test_sax.py as an example.

While I'm at it I'm going to fix also mkfifo(), mknod() and others. Hold on a bit more.

Ok, patch in attachment fixes mkfifo(), mknod() and statvfs() and also includes Unicode tests for all os module's path-related functions.

You have eaten "return NULL;" in posix_mkfifo.

"from test.test_support import TESTFN_UNICODE, TESTFN_ENCODING" can fail. The simplest solution is just initialize them to None by default in test_support.

If TESTFN_UNICODE.encode(TESTFN_ENCODING) fails (on POSIX locale), it will be better to run tests with unicode(TESTFN, 'ascii') than skip them.

Tests should check that results for unicode filename is same as for str filename.

As far as Victor have doubts, we should ask Benjamin.

> As far as Victor have doubts, we should ask Benjamin.

Well, if you begin to patch some os functions, we will find much functions which don't support Unicode path.

I prefer to consider that Python 2 doesn't support Unicode filenames to avoid bugs.

If you want to support Unicode filename, we will have to modify a lot of code. What's the point since Python 3 has a very good support of Unicode? Much better than Python 2?

Either way it's fine with me.

Regardless I think these tests have some value because all those os functions are not currently tested so it might makes sense to apply  Serhiy's suggestions and port them to Python 3.4.

BDFL says (http://permalink.gmane.org/gmane.comp.python.devel/146074):

"""Given that the claim "Python 2 doesn't support Unicode filenames" is factually incorrect (in Python 2.7, most filesystem calls in fact do support Unicode, at least on some platforms), I think individual functions in the os module that are found lacking should be considered bugs, and if someone goes through the effort to supply an otherwise acceptable fix, we shouldn't reject it on the basis that we don't want to consider supporting Unicode filenames."""

Ok, I will go on then.

> You have eaten "return NULL;" in posix_mkfifo.

What do you mean?

> If TESTFN_UNICODE.encode(TESTFN_ENCODING) fails (on POSIX locale), it will be better to run tests with 
> unicode(TESTFN, 'ascii') than skip them.

Agreed.

> Tests should check that results for unicode filename is same as for str filename.

What do you mean? Can you provide an example?

> > You have eaten "return NULL;" in posix_mkfifo.
> What do you mean?

You deleted "return NULL;" after "if (!PyArg_ParseTuple(...))" in the 
posix_mkfifo() function.

> > Tests should check that results for unicode filename is same as for str
> > filename.
> What do you mean? Can you provide an example?

For example test_statvfs should check that os.statvfs(TESTFN_UNICODE) == 
os.statvfs(TESTFN_UNICODE_ENCODED) (where TESTFN_UNICODE_ENCODED is relevant 
8-bit str).

Python 2.7 is no longer supported.