This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: Wrong behavior for '\xff\n'.decode('gb2312', 'ignore')
Type: behavior Stage: test needed
Components: Unicode Versions: Python 3.3
Status: closed Resolution: fixed
Dependencies: 12057 Superseder:
Assigned To: Nosy List: cdqzzy, ezio.melotti, hyeshik.chang, lemburg, python-dev, terry.reedy, vstinner
Priority: normal Keywords: patch

Created on 2011-05-06 09:16 by cdqzzy, last changed 2022-04-11 14:57 by admin. This issue is now closed.

File name Uploaded Description Edit
cjk_decode.patch vstinner, 2011-06-03 22:20 review
Messages (14)
msg135268 - (view) Author: zy (cdqzzy) Date: 2011-05-06 09:16
let s='\xff\n' 
The expected result of s.decode('gb2312', 'ignore') is u"\n", while in 2.6.6 it is u"".
  s can be replaced with chr(m) + chr(n) , where m is in range of 128~255, and n in 0~127.
  In the above cases, try decoding from chr(n) will never interfere with later parts in the string if there is any, since chr(n) do not start a multibyte sequence.
msg135410 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-05-07 07:41
u'' in 2.7.1 also, on winxp
msg135419 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-05-07 08:40
So the correct result for b'\xff\n'.decode('gb2312', 'replace') is u'?\n'?
msg135422 - (view) Author: zy (cdqzzy) Date: 2011-05-07 09:08
> So the correct result for b'\xff\n'.decode('gb2312', 'replace') is u'?\n'?

I think it should be so. This behavior does not leave out possible information, has no side-effect on later decodings, and should the '\n'  indeed be redundant, an output of u'?\n' would unlikely cause confusions.

Though, I have no knowledge on this subject code-wise. If a change of the behavior will have an impact on performance, maybe the change should not come in.
msg135423 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-05-07 09:21
_codecs_cn implements different multibyte encodings: gb2312, gbkext, gbcommon, gb18030ext, gbk, gb18030.

And there are other Asian multibyte encodings: big5 family, ISO 2202 family, JIS family, korean encodings (KSX1001, EUC_KR, CP949, ...), Big5, CP950, ...

All of them ignore the all bytes if one byte of a multibyte sequence is invalid (lile 0xFF 0x0A: replaced by ? instead of ?\n using replace error handler).

I don't think that you can/should patch only one encoding: we should use the same rule for all encodings.

By the way, do you have any document explaining which result is the good one (? or ?\n)? For UTF-8, we have well defined standards explaining exactly what to do with invalid byte sequences => see issue #8271. It is easy to fix the decoders, but I would like to be sure that your proposed change is the right way to decode these encodings.

Change the multibyte encodings can also concern the security. Read for example the following section "Check byte strings before decoding them to character strings" of my book:
msg135447 - (view) Author: zy (cdqzzy) Date: 2011-05-07 11:32
I do not have documents on this subject. Though, I found that GNU iconv(1) behaves the same as my proposed behavior. My reading of the source code suggests that iconv(1) treat all encodings equally, which I think should also be true for python.

As of security concerns, I do not think the change in decoding function itself would introduce any security vulnerabilities. If a security issue arises because of the proposed change, there must be improper code out side of python, which is out of python's control. That said, the proposed change is unlikely to introduce new security vulnerability, as all it does in effect is retaining a few ascii characters in the string to the output as opposed to removing.  In the issue of wordpress, if we suppose that wordpress was written in python, and that the attacker was using gb2312 encoded strings instead of gbk, then my proposed change would by chance fix the issue, as the backslash would be retained when we decode the string.
msg135767 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-05-11 09:51
I asked if the change is correct on iconv mail list. Here is a copy of an answer.

De: 	Bruno Haible
À: 	[iconv mailing list]
Cc: 	Victor Stinner
Sujet: 	Re: [bug-gnu-libiconv] Invalid byte sequences and multiybyte encodings
Date: 	Tue, 10 May 2011 14:52:09 +0200


> Someone opened an issue in Python bug tracker asking to change how
> invalid multibyte sequences are handled.

For UTF-8 the recommended way of handling malformed input is written down
in <>. But the
principle applies to any encoding with a variable number of bytes per
  When an invalid or malformed byte sequence is found, the smallest
  such byte sequence is transformed to U+FFFD (replacement character).

In particular, normally, if the first byte that is considered "wrong"
or "invalid" is a valid starter byte, the malformed byte sequence should
be considered to end before that byte. If it is not a valid starter
byte, then use your judgement.

For an example implementation, see
Here the return value is the number of bytes consumed. Look carefully
when it is 1, 2, 3, or 4.

> b'\xffabc'.decode('gb2312', 'replace') gives "�bc". The 'a' character is
> seen as part of a multibyte character of 2 bytes. Because {0xFF, 0x61}
> is invalid in GB2312, the two bytes are replaced by U+FFFD.
> Is it the "right" way to to do?

It is better to replace only the 0xFF byte with U+FFFD, because 0x61 is a
valid first byte (even a complete character).

> UTF-8 decoder changed recently to ignore a single byte and restart the
> decoder, so '\xF1\x80\x41\x42\x43' is now decoded "�ABC" instead "�C".
> Should we do the same for all encodings?

Generally, yes.

> Or at least for asian encodings 
> (gb2312, gbk, gb18030, big5 family, ISO 2202 family, JIS family, EUC_KR,
> CP949, Big5, CP950, ...)?

For stateful encodings of the ISO 2202 family, you may want to ignore/replace
a complete escape sequence, where the syntax of escape sequences is defined
through general rules.

In memoriam Siegfried Rädel <ädel>
msg135769 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-05-11 10:53
Oh, the HZ codec has no test! And what is this horrible BLOB, Lib/test/
msg137340 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-05-30 22:12
- I added tests for the HZ codec and some ISO 2022 codecs: #12057
 - I fixed IncrementalEncoder.encode() (of multibytecodec ): #12100
 - I fixed IncrementalEncoder.reset() (of multibytecodec): #12171

I can now work confidently on this issue. I will try to patch all CJK decoders to only replace 1 invalid byte by U+FFFD (and not 2, 3 or 4 bytes) and try to write a test for each case (each byte sequence generating a different error).
msg137591 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2011-06-03 21:38
New changeset 3610841f7357 by Victor Stinner in branch '3.2':
Issue #12016: Reindent decoders of HK and JP codecs

New changeset aa07c1237f4e by Victor Stinner in branch 'default':
(Merge 3.2) Issue #12016: Reindent decoders of HK and JP codecs

New changeset 685351d65592 by Victor Stinner in branch '2.7':
Issue #12016: Reindent decoders of HK and JP codecs
msg137595 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2011-06-03 21:58
New changeset 8572bf1b56ec by Victor Stinner in branch '3.2':
Issue #12016: Add test_errorhandle() to TestBase_Mapping of

New changeset c3dc94d53ef8 by Victor Stinner in branch 'default':
(Merge 3.2) Issue #12016: Add test_errorhandle() to TestBase_Mapping of

New changeset 53912b58eee6 by Victor Stinner in branch '2.7':
Issue #12016: Add test_errorhandle() to TestBase_Mapping of
msg137602 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-06-03 22:20
 - patch *all* CJK decoders to replace only the first byte of an invalid byte sequence (by U+FFFD). Example from the issue title: b'\xff\n'.decode('gb2312', 'replace') gives now '�\n' instead of just '�'
 - add at least one unit test for *each* path in the decoder (sometimes it was really hard to see how to go into a specific path, especially for the johab decoder!)
 - add testcases for euc_jis_2004 and shift_jis_2004
 - factorize "codec tests" (codectests) of all japanese EUC tests (euc_commontests)

Because I consider this issue as a bug, I would like to apply this patch to 2.7, 3.2 and 3.3.
msg140004 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2011-07-07 23:45
New changeset 16cbd84de848 by Victor Stinner in branch 'default':
Issue #12016: Multibyte CJK decoders now resynchronize faster
msg140006 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-07-07 23:52
> Because I consider this issue as a bug, I would like
> to apply this patch to 2.7, 3.2 and 3.3.

It is maybe a bug but it is also an important change on Python behaviour, so finally I prefer to only change (fix) Python 3.3.

Thanks for reporting the bug zy (cdqzzy). Tell me if it now behaves as you expected.

I'm closing this issue because the initial issue is now fixed.
Date User Action Args
2022-04-11 14:57:17adminsetgithub: 56225
2011-07-07 23:52:16vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg140006

versions: - Python 3.1, Python 2.7, Python 3.2
2011-07-07 23:45:23python-devsetmessages: + msg140004
2011-06-03 22:20:03vstinnersetfiles: + cjk_decode.patch
keywords: + patch
messages: + msg137602
2011-06-03 21:58:06python-devsetmessages: + msg137595
2011-06-03 21:38:16python-devsetmessages: + msg137591
2011-05-30 22:12:22vstinnersetmessages: + msg137340
2011-05-30 21:48:11vstinnersetmessages: - msg137334
2011-05-30 21:47:16python-devsetnosy: + python-dev
messages: + msg137334
2011-05-11 19:02:32vstinnersetnosy: + hyeshik.chang
2011-05-11 13:32:55vstinnersetdependencies: + HZ codec has no test
2011-05-11 10:53:57vstinnersetmessages: + msg135769
2011-05-11 09:52:00vstinnersetmessages: + msg135767
2011-05-07 11:32:46cdqzzysetmessages: + msg135447
2011-05-07 09:21:43vstinnersetmessages: + msg135423
2011-05-07 09:08:38cdqzzysetmessages: + msg135422
2011-05-07 08:40:05vstinnersetmessages: + msg135419
versions: + Python 3.1, Python 3.2, Python 3.3
2011-05-07 07:41:48terry.reedysetnosy: + terry.reedy
messages: + msg135410
2011-05-06 17:16:42eric.araujosetnosy: + lemburg, vstinner
2011-05-06 09:26:34ezio.melottisetnosy: + ezio.melotti
stage: test needed

versions: + Python 2.7, - Python 2.6
2011-05-06 09:16:14cdqzzycreate