Title: Encoding a unicode with unicode() and ignoring errors
Type: behavior Stage: resolved
Components: Versions: Python 2.7
Status: closed Resolution: rejected
Dependencies: Superseder:
Assigned To: Nosy List: G..Scott.Johnston, ned.deily
Priority: normal Keywords:

Created on 2013-08-28 06:28 by G..Scott.Johnston, last changed 2013-08-28 06:45 by ned.deily. This issue is now closed.

Messages (2)
msg196350 - (view) Author: G. Scott Johnston (G..Scott.Johnston) Date: 2013-08-28 06:28
I've come up with the following series of minimal examples to demonstrate my bug. 

>>> unicode("")
>>> unicode("", errors="ignore")

>>> unicode("abcü")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 3: ordinal not in range(128)
>>> unicode("abcü", errors="ignore")

>>> unicode(3)
>>> unicode(3, errors="ignore")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: coercing to Unicode: need string or buffer, int found

>>> unicode(unicode(""))
>>> unicode(unicode(""), errors="ignore")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: decoding Unicode is not supported

The first two pairs of mini-programs are reasonable behaviour.  If the errors parameter is set to "ignore", no additional errors are thrown, but characters that produce encoding errors are skipped in the output, as expected.  

The third pair of mini-programs can be solved by instead writing unicode(str(3), errors="ignore").  This should likely be done automatically, given the fact that unicode(3) behaves as expected, and properly converts between types.  The fact that the conversion is done automatically without the errors parameter leads me to believe that there is a logic problem with the code, where the setting errors="ignore" changes the path of execution by more than just skipping characters that cause encoding errors.

The fourth pair of mini-programs is simply baffling.  The first mini-program clearly demonstrates that decoding a Unicode object is in fact supported.  The fact that the second mini-program claims it's not supported further demonstrates that the logic depends on the errors="ignore" parameter more than it should.
msg196351 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-08-28 06:45
See  It appears to me that unicode() is behaving exactly as documented. In particular:

"If encoding and/or errors are given, unicode() will decode the object which can either be an 8-bit string or a character buffer using the codec for encoding."

"If no optional parameters are given, unicode() will mimic the behaviour of str() except that it returns Unicode strings instead of 8-bit strings. More precisely, if object is a Unicode string or subclass it will return that Unicode string without any additional decoding applied."

One can argue about whether this documented behavior makes the most sense but, since it is documented to behave that way and that any significant changes to that behavior at this late stage of the life of Python 2 could break existing programs, I think there will be little support for making such a change now.  Sorry!
Date User Action Args
2013-08-28 06:45:08ned.deilysetstatus: open -> closed

type: behavior

nosy: + ned.deily
messages: + msg196351
resolution: rejected
stage: resolved
2013-08-28 06:28:31G..Scott.Johnstoncreate