Hey everyone,

I'm having issues writing unicode strings with ConfigParser.write. I don't know
if this is python's fault or my own but I couldn't find help on this, neither by
googling, nor by asking on the python irc channels.

Attached to this description I'll add an example script reproducing the error
and hope that someone will enlighten me on what I'm doing wrong.

It seems that this only occurs in python2, doing the same with python3 (omitting
the u before the unicode string), everything works fine.

Thanks in advance!

>>> str(u"\u0411")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\u0411' in position 0: ordinal not in range(128)

So, clearly configparser in 2.x doesn't support unicode.  Now the question is, is this a bug or would adding support be a feature? (If the latter it can't be fixed in 2.7.)  I'll leave the answer to that question up to Lucaz.

> Now the question is, is this a bug 
> or would adding support be a feature?

That may be a good question for python-dev.

Since ConfigParser is a very old module,
if there were a pressing need, we probably
would have heard about it before now.

Well, python3 is probably pushing some people to try to add better unicode support to their python2 versions.  I think it is more a question of "is this an easy fix?" or would it require extensive changes to support unicode properly.  If it is easy (ie: it is really just a bug in configparser's string handling) I don't see a downside to fixing it.

It's also a question of adding a feature to a point release.  If dev's relied on the new feature, they would also have to test for version > 2.7.1.  We usually try to avoid that (after a minor fiasco with booleans many years ago).

I understand what you are saying, and I thought about that, too; but you could say the same thing about any bug fix that makes code work that didn't work before, yet we don't.  So I guess you are right that it should be discussed on python-dev.

My vote would be that this is a new feature and therefore uneligible for 2.x.

In my opinion we should unfortunately close this with WONTFIX because:

* adding a feature in a point release is not an option
* this may be poorly documented but most of the standard library in 2.x assumes bytestrings (and fails with Unicode strings). Same goes for instance for csv.
* storing encoded data internally enables for direct reading and writing to compatible files
* having Unicode instead requires the possibility to specify an `encoding` parameter to read* and write* methods. I added this parameter in 3.2 exactly for this reason.

So, unless I missed something obvious and there's more discussion, I am closing this issue next week.

No, the need for an encoding parameter for read/write makes it unambiguous that it is indeed a feature.

Well, the only thing I can add to this is that the json module (which I ended up
using) supports unicode with no problem. So I think the argument that most of
the standard library in 2.x assumes bytestrings is a bit... shaky.

Other than that, I can follow your reasoning and will just hope that all my
needed external libraries will soon make it to python 3 so I don't have to fight
such inconsistencies anymore :)

As another core dev aptly said, most standard library Unicode support is probably accidental. As for `json`, this is one of the "newest" additions to stdlib, introduced in Python 2.6 (released at the same time as Python 3.0). Not the best example if you compare it to libraries that were added in 1.5 or so.

> I think it is more a question of "is this an easy fix?" 
> or would it require extensive changes to support unicode properly.

First of all, the question is: who would like to develop it. You can vote for an issue, but it doesn't change anything if you don't find a developer to implement your idea :-)

Anyway, try to use Python everywhere in Python 2 is a waste of time. The work has already been done in Python 3, and it's much easier to use Unicode in Python 3, just because everything uses Unicode (exceptions, filenames, file content, modules, etc.).

> Anyway, try to use Python everywhere in Python 2 is a waste of time.

Oh... I mean "use Unicode in Python 2"

the_isz, for serious Unicode support you might try using the configparser 3.2 backport:

http://pypi.python.org/pypi/configparser

Thanks for the hint, Łukasz, but like I stated earlier, I already worked around
this problem by using the json module instead.

some workaround

import configparser
import codecs

cfg = configparser.ConfigParser()
cfg.write(codecs.open('filename','wb+','utf-8'))

In file ConfigParser.py, def write, replace line

    key = " = ".join((key, str(value).replace('\n', '\n\t')))

by

    key = " = ".join((key, str(value).decode('utf-8').replace('\n', '\n\t')))

Tested in Windows7 and Ubuntu 15.04.