This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author serhiy.storchaka
Recipients ezio.melotti, lemburg, loewis, serhiy.storchaka, vstinner
Date 2015-08-21.20:52:30
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <>
There is a reason for behavior in case 2. This is likely a truncated data and it is safer to raise an exception than silently produce lone surrogate. Current UTF-7 encoder always adds '-' after ending shift sequence. I suppose this is not a bug.

However there are yet three bugs.

4. Decoder can emit lone low surrogate before replacement character in case of error.

>>> b'+2DTdI-'.decode('utf-7', 'replace')

A low surrogate is a part of incomplete astral character and shouldn't emitted in case of error in encoded astral character.

5. According to RFC 2152: "A "+" character followed immediately by any character other than members of set B or "-" is an ill-formed sequence." But this is accepted by current decoder as empty shift sequence that is decoded to empty string.

>>> b'a+,b'.decode('utf-7')
>>> b'a+'.decode('utf-7')

6. Replacement character '\ufffd' can be replaced with character 'ý' ('\xfd'):

>>> b'\xff'.decode('utf-7', 'replace')
>>> b'a\xff'.decode('utf-7', 'replace')
>>> b'a\xffb'.decode('utf-7', 'replace')
>>> b'\xffb'.decode('utf-7', 'replace')

This bug is reproduced only in 3.4+.

Following patch fixes bugs 1 and 4 and adds more tests.

Corner cases 2 and 3 are likely not bugs.

I doubt about fixing bug 5. iconv accepts such ill-formed sequences. In any case I think the fix of this bug can be applied only for default branch.

I have no idea how to fix bug 6. I afraid it can be a bug in _PyUnicodeWriter and therefore can affect other decoders.
Date User Action Args
2015-08-21 20:52:43serhiy.storchakasetrecipients: + serhiy.storchaka, lemburg, loewis, vstinner, ezio.melotti
2015-08-21 20:52:40serhiy.storchakasetmessageid: <>
2015-08-21 20:52:39serhiy.storchakalinkissue24848 messages
2015-08-21 20:52:36serhiy.storchakacreate