In the process of trying to use optparse with russian messages, I found
several problems with gettext and unicode handling:

1. optparse.OptionParser.error function doesn't work with unicode argument
2. optparse doesn't work when its error messages are gettext-translated
3. optparse fails running 'prog.py --help > out.txt' with unicode help
(at least on my system: Windows XP, Russian)

I have attached a file demonstrating these problems: test_optparse.py.
You can run it either using nose[1] or directly, manually uncommenting
test_* functions one-by-one.

[1]: http://www.somethingaboutorange.com/mrl/projects/nose/

Here's the result of running `nosetests test_optparse.py`:

EEF
======================================================================
ERROR: OptionParser.error function doesn't work with unicode argument
----------------------------------------------------------------------
Traceback (most recent call last):
  File
"c:\python25\lib\site-packages\nose-0.10.2-py2.5.egg\nose\case.py", line
182, in runTest
    self.test(*self.arg)
  File "C:\work\test_optparse.py", line 10, in test_unicode_error
    optparse.OptionParser().error(russian_text)
  File "C:\Python25\lib\optparse.py", line 1562, in error
    self.exit(2, "%s: error: %s\n" % (self.get_prog_name(), msg))
  File "C:\Python25\lib\optparse.py", line 1551, in exit
    sys.stderr.write(msg)
UnicodeEncodeError: 'ascii' codec can't encode characters in position
28-34: ordinal not in range(128)

======================================================================
ERROR: optparse doesn't work when its error messages are gettext-translated
----------------------------------------------------------------------
Traceback (most recent call last):
  File
"c:\python25\lib\site-packages\nose-0.10.2-py2.5.egg\nose\case.py", line
182, in runTest
    self.test(*self.arg)
  File "C:\work\test_optparse.py", line 25, in
test_translated_unicode_error_message
    optparse.OptionParser().parse_args(["--unknown"])
  File "C:\Python25\lib\optparse.py", line 1380, in parse_args
    self.error(str(err))
UnicodeEncodeError: 'ascii' codec can't encode characters in position
0-6: ordinal not in range(128)

======================================================================
FAIL: optparse fails running 'prog.py --help > out.txt' with unicode help
----------------------------------------------------------------------
Traceback (most recent call last):
  File
"c:\python25\lib\site-packages\nose-0.10.2-py2.5.egg\nose\case.py", line
182, in runTest
    self.test(*self.arg)
  File "C:\work\test_optparse.py", line 42, in test_redirected_unicode_help
    assert '?????' not in dummy_stdout.getvalue()
AssertionError

----------------------------------------------------------------------
Ran 3 tests in 0.000s

FAILED (errors=2, failures=1)

I've also attached a patch that fixes all these issues and also allows
the word "error" to be translated with gettext.

Regarding the use of `locale.getpreferredencoding` instead of
`sys.getdefaultencoding`. On my system (Windows XP, Russian) I get:

Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit
(Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys, locale
>>> sys.getdefaultencoding()
'ascii'
>>> locale.getpreferredencoding()
'cp1251'

Using cp1251 on my system makes much more sense. It's used as a default
encoding everywhere in the system. For example, in Notepad.

Using non-ASCII characters in an optparse help string also causes 
UnicodeDecodeErrors. Here's the relevant part of the traceback:

File "/home/spk30/opt/ActivePython-2.5/lib/python2.5/optparse.py", line 
1655, in print_help
    file.write(self.format_help().encode(encoding, "replace"))

NB. Adding an encoding declaration at the beginning of the python 
script which used a non-ASCII character in an optparse help string 
didn't solve the problem.

sampablokuper, I don't think your problem is relevant to this issue. In
addition to encoding declaration you should use unicode strings: u"your
non-ASCII text". Or wait for Python 3.0, where strings will be unicode
by default.

ash, you are correct; my bad. Thanks for the heads-up.

What I find most bothersome is that ``optparse`` is being inconsistent
in the types of localised strings it expects. It needs Unicode strings
for snippets forming part of the help message, while it expects normal
strings in other places like ``OptionParser.error()`` --a fact which
isn't documented at all, BTW.

I've been developing a medium app lately with localised messages all
over the place using several packages in the standard library and
``optarparse``'s help messages are the only place where Unicode strings
have been required. I'm not saying that ``optparse`` shouldn't use
Unicode, but it'd be nice if it was consistent and the fact was documented.

I'm attaching a tiny script which uses ``optparse``.  Just try to change
any appearance of the ``s`` normal string to Unicode ``us`` or
vice-versa, then call the program with ``--help`` or no arguments (it
requires one) and you get a ``UnicodeError``. Thanks!

The attached version of ``optparse_unicode.py`` doensn't depend on a
UTF-8 locale, sorry.

More than a year passed since I reported this... Could someone suggest
how to move this forward? If needed, I can try to improve patch, test or
description of this issue. Should I, for example, split this into
separate issues?

Alexy, there would be a much better chance of getting this accepted if you could supply a patch file that also included unit tests.

It would be nice to test argparse for the same behavior.

Yep, argparse almost certainly has the same kind of problems - I basically copied the optparse gettext behavior into argparse because I don't really know how that stuff works but figured people must have wanted what was in there. ;-)

After so much time I've checked again with the little script I sent and I see that it doesn't happen under Python 2.7 (2.7.1+), but it does under 2.6 (2.6.6) and 2.5 (2.5.5).

I’m afraid 2.5 and 2.6 don’t get bug fixes any more, only security fixes.  For 2.7 and 3.x, even if your bug can’t be reproduced, I think it would be useful to add the test to prevent a regression.

+1