classification
Title: Bad encoding alias cp936 -> gbk: euro sign
Type: behavior Stage:
Components: Unicode, Windows Versions: Python 3.7, Python 3.6, Python 3.5, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Artoria2e5, ezio.melotti, paul.moore, steve.dower, tim.golden, vstinner, zach.ware
Priority: normal Keywords:

Created on 2016-10-03 03:11 by Artoria2e5, last changed 2016-11-24 21:31 by Artoria2e5.

Messages (4)
msg277925 - (view) Author: Mingye Wang (Artoria2e5) * Date: 2016-10-03 03:11
Microsoft's cp936 defines a euro sign at 0x80, but Python would kick the bucket when asked to do something like `u'\u20ac'.encode('cp936')`. This may break things for zh-hans-cn windows users who wants to put a euro sign in their file name (if they insist on using a non-unicode str for open() in py2, well.)

By looking at the codecs documentation, 'cp936' appears to be an alias for the GBK encoder, which by itself has been a very ambiguous name and subject to confusion --

The name "GBK" might refer to any of the four commonly-known members of the family of EUC-CN (gb2312) extensions that has full coverage of Unicode 1.1 CJK Unified Ideographs block:
  1) The original GBK. Rust-Encoding says that it's in a normative annex of GB13000.1-1993, but the closest thing I can find in my archive.org copy of that standard is an annex on an EUC (GB/T 2311) UCS.
  2) IANA GBK, or Microsoft cp936. This is the one with the euro sign I am looking for.
  3) GBK 1.0, a recommendation from the official standardization committees based on cp936. It's roughly cp936 without the euro sign but with some additional 95 PUA code points. 
  4) W3C TR GBK. This GBK is basically gb18030-2005 without four-byte UTF, and with the euro sign. Roughly a union of 2) and 3) with some PUA code points moved into the right place.
 
Looking at Modules/cjkcodecs/_codecs_cn.c @ 104259:36b052adf5a7, Python seems to be doing either 1) or 3). For a quick fix you can just make an additional cp936 encoding around the gbk encoding that handles U+20AC; for some excitement (of potentially breaking stuff) you can join the web people and use either 2) or 4).
msg278393 - (view) Author: Mingye Wang (Artoria2e5) * Date: 2016-10-09 21:43
The "join the web people" solution should look like this:

$ diff -Naurp a/_codecs_cn.c b/_codecs_cn.c
--- a/_codecs_cn.c    2016-10-09 14:24:04.675111500 -0700
+++ b/_codecs_cn.c    2016-10-09 14:27:06.600961500 -0700
@@ -128,6 +128,12 @@ ENCODER(gbk)
             continue;
         }

+        if (c == 0x20AC) { /* cp936, or web GBK */
+            WRITEBYTE1((unsigned char)0x80);
+            NEXT(1, 1);
+            continue;
+        }
+
         if (c > 0xFFFF)
             return 1;

@@ -159,6 +165,12 @@ DECODER(gbk)
             NEXT_IN(1);
             continue;
         }
+
+        if (c == 0x80) { /* cp936, or web GBK */
+            OUTCHAR(0x20AC);
+            NEXT_IN(1);
+            continue;
+        }

         REQUIRE_INBUF(2);

It should be mostly safe as this character is not previously defined in Python's GBK implementation.
msg280847 - (view) Author: Mingye Wang (Artoria2e5) * Date: 2016-11-15 14:25
Also, go to ftp://ftp.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WindowsBestFit/bestfit936.txt for MS reference.
msg280968 - (view) Author: Mingye Wang (Artoria2e5) * Date: 2016-11-16 17:01
Update: the test script at issue28712 can be modified to show this issue too.
History
Date User Action Args
2016-11-24 21:31:57Artoria2e5setversions: - Python 3.3, Python 3.4
2016-11-16 17:01:34Artoria2e5setmessages: + msg280968
2016-11-16 17:00:36Artoria2e5setnosy: + paul.moore, tim.golden, zach.ware, steve.dower
components: + Windows
2016-11-15 14:25:36Artoria2e5setmessages: + msg280847
2016-10-09 21:43:30Artoria2e5setmessages: + msg278393
2016-10-03 03:11:31Artoria2e5create