This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: No EUDC (HKSCS) support in Windows cp950
Type: behavior Stage:
Components: Unicode, Windows Versions: Python 3.7, Python 3.6, Python 3.5, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Artoria2e5, ezio.melotti, paul.moore, steve.dower, tim.golden, vstinner, zach.ware
Priority: normal Keywords:

Created on 2016-11-14 20:09 by Artoria2e5, last changed 2022-04-11 14:58 by admin.

Messages (4)
msg280811 - (view) Author: Mingye Wang (Artoria2e5) * Date: 2016-11-14 20:09
Python's cp950 implementation lacks support for HKSCS ('big5hkscs'). This support, which maps HKSCS Big5-EUDC code points to Unicode PUA code points algorithmically, is found in Windows Vista+ as well as an update for XP.

An experiment session is shown below. I will use '2>>>' to denote a Win32 build of Python 2.7.10 running under a console window set to cp950 (via chcp), and '3>>>' to denote a Python 3.4.3 build running under Cygwin's UTF-8 mintty. HKSCS-2008's table is used  http://www.ogcio.gov.hk/en/business/tech_promotion/ccli/terms/doc/hkscs-2008-big5-iso.txt for a list of HKSCS characters; note though, its non-PUA mappings are not found in Windows.

Let's start with the first character in that list.

3>>> u'\u43F0'
'䏰'
3>>> print(u'\uF266') # provisional PUA

3>>> u'\u43F0'.encode('cp950') # FAIL
3>>> u'\uF266'.encode('cp950') # FAIL
3>>> u'\u43F0'.encode('hkscs')
b'\x87@'
3>>> u'\uF266'.encode('hkscs') # FAIL`

These experiments above show how Python 3 handles HKSCS characters, and how U+43F0 should normally be encoded. Now let's switch to Windows console, which would be using Windows' decode-to-Unicode routine for cp950.

2>>> print b'\x87@'


Let's try to identify this character:

3>>> u''
'\uf266'

So indeed there is some sort of HKSCS going on. But note what Windows has is really not any kind of new HKSCS:

> Big5       ucs93                  ucs00                   ucs03 + 1-6
> 876B       9734                   9734                    9734
> 876C       F292                   F292                   27BEF
> 876D       5BDB                   5BDB                    5BDB

2>>> print b'\x87\x6b,\x87\x6c,\x87\x6d'
,,
3>>> u',,'
'\uf291,\uf292,\uf293'

Just as for all other code pages, you can always find Microsoft's mapping at ftp://ftp.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WindowsBestFit/bestfit950.txt. If you are uncomfortable with adding a whole new table and wasting space (this is done for hkscs btw), use the algorithmic mapping at https://en.wikipedia.org/wiki/Code_page_950.
msg280828 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-11-15 08:29
Python supports native Windows code pages using
codecs.code_page_encode() and codecs.code_page_decode() methods. See
for example Lib/encodings/cp65001.py : this codec is not implemented
in Python, but is a wrapper to native Windows functions
(MultiByteToWideChar and WideCharToMultiByte).
msg280969 - (view) Author: Mingye Wang (Artoria2e5) * Date: 2016-11-16 17:01
Update: the test script at issue28712 can be modified to show this issue too.
msg281654 - (view) Author: Mingye Wang (Artoria2e5) * Date: 2016-11-24 21:31
Windows cp950's EUDC<->PUA mapping is not specific to HKSCS.
History
Date User Action Args
2022-04-11 14:58:39adminsetgithub: 72879
2016-11-24 21:31:35Artoria2e5setmessages: + msg281654
title: No HKSCS support in Windows cp950 -> No EUDC (HKSCS) support in Windows cp950
2016-11-18 18:58:28terry.reedysetversions: - Python 3.3, Python 3.4
2016-11-16 17:01:38Artoria2e5setnosy: + paul.moore, tim.golden, zach.ware, steve.dower
messages: + msg280969
components: + Windows
2016-11-15 08:29:18vstinnersetmessages: + msg280828
2016-11-14 20:09:08Artoria2e5create