Hi,

There is a small bug in GB18030 decoder.

For 4-byte sequence, the legal range is:
0x81-0xFE for the 1st byte
0x30-0x39 for the 2nd byte
0x81-0xFE for the 3rd byte
0x30-0x39 for the 4th byte

The current code forgets to check 0xFE of the 1st and 3rd byte.
Therefore, there are 8630 illegal 4-byte sequences can be decoded by GB18030 codec, here is an example:

# legal sequence 0x81319130 is decoded to U+060A, it's fine.
b = bytes([0x81, 0x31, 0x81, 0x30])
uchar = b.decode('gb18030')
print(ord(uchar))

# illegal sequence 0x8130FF30 can be decoded to U+060A as well.
b = bytes([0x81, 0x30, 0xFF, 0x30])  
uchar = b.decode('gb18030')
print(ord(uchar))

Adding Hye-Shik who wrote the codec.

I found another bug in hz codec.
hz encoding uses 7-bit ASCII to represent Chinese characters, it was popular in USENET networks in the late 1980s and early 1990s.

I will do more check and fix them together, then I will invite you to review the patch.


u = 'hi~python'
b = u.encode('hz')   # bug in this step, the right sequence should be b"hi~~python"
print(b)    # the output is b"hi~python"

u = b.decode('hz')   # so can't decode, UnicodeDecodeError raised
print(u)

I examined all Chinese codecs, here are the patches, please review them, feel free to ask me your question.

Thanks to Hye-Shik, your framework is very easy to understand :)

Do you have authoritative links that describe these standards?

Good question.

GB2312:
I tested those programming languages one by one.

GBK/CP936/GB18030-2000:
I gathered data via Internet as much as I can, then compare them to Python3's codecs. I check key points with authoritative source, and verify every appeared conflicts.
The data come from ICU, Unicode.org, IBM, Chinese researchers, and data found by Google.

I had spent about half-month to do this, not just started from several days ago. I hope those descriptions will help late comers.

>> I examined all Chinese codecs
I said it above, but I forgot Taiwan and HongKong are using Chinese as well.

BIG5 and CP950 are using a wrong convert table, test this:
>>> u = b'\xC6\xA1'.decode('big5')
>>> hex(ord(u))
'0x30fe'

This should not happen, 0xC6A1 is neither in BIG5 nor in CP950.
In BIG5-2003 and HKSCS-2008, 0xC6A1 is mapped to U+2460.

I only had a look roughly, please check more.
I won't check HongKong codec anymore, I suggest check it as well.

> The data come from ICU, Unicode.org, IBM, 

If you could provide links to the relevant pages/section we can verify that the codecs are indeed incorrect.  Also keep in mind that there might people relying on these incorrectness, so we have to be careful while changing them.  This is especially true if there are de-facto standards that diverge from the actual standards.

This is not a de-facto standard, it should be fixed.
I already posted this infomation on a Taiwan Python community, let's wait their inspection.

>> If you could provide links to the relevant pages/section we can verify that the codecs are indeed incorrect. 

Here is CP950, 0xC6A1 is not in it.
https://msdn.microsoft.com/zh-cn/goglobal/cc305155

I can provide one link, but there are many variants of BIG5 convert table on the Interenet, so one link doesn't bring persuasion.

In this page: https://moztw.org/docs/big5/
Listed many variants of BIG5 tables, I found 0xC6A1<->U+30FE in this table "Unicode 1.1", the description of it is "it's a terrible table, many errors exist, sadlly many foreigners are using it", but IIRC Python's BIG5 codec is not fully same as that table.

IMO, the most reliable way is reading a lot of stuff, and verify the key points and conflicts with authoritative source, but this way is very hard for foreigners.

Anyway, let's wait Taiwanese and their opinion for whether this should be fixed.

Did you hear anything back from them?

I posted in a Taiwanese forum: https://groups.google.com/forum/#!forum/pythontw
no reply yet.

Just FYI, cp950 0xC6A1 (\uf6b1) is found in current WindowsBestFit: ftp://ftp.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WindowsBestFit/bestfit950.txt

I closed this issue, because it involved too many things.
 
1, for GB18030 decoder bug, see issue29990.
2, for hz encoder bug, see issue30003.
3, for problem in Traditional Chinese codecs, please create a new issue.