Author Suzumizaki
Recipients Suzumizaki
Date 2014-02-02.14:20:26
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1391350828.19.0.622173065149.issue20485@psf.upfronthosting.co.za>
In-reply-to
Content
Currently, the name of .pyd modules is limited within 7 bit US-ASCII.

I want to do "import X" to import X.pyd, where X contains Unicode characters.
I tried to make the patch and my work seems to be done successfully.
I will post the patch with this issue, and next what should I do?

About the solution:
To make the export entry 'PyInit_xxx' kept inside 7 bit, I use the simple encoding 'szm62' for unicode, called in the patch.

1) 'szm62' is used once per module(.pyd), only for 'PyInit_xxx'.
2) 'szm62' is used only when non-ASCII characters are in the name of the module.
3) 'szm62' generates short string as UTF-8, except 0-9A-Za-z are encoded to 2 bytes.
4) 'szm62' is very simple, much easier than UTF-8.
5) I tested it only with MS VC++, but I believe highly compatible with other environments.
6) 'szm62' also can decode 7 bits to unicode, but only the encoding is used for this issue.

Notes:
The simplicity is important for the project like Cython -- it generates .pyd files.
The codepoints over 16bits are also simply supported. They will be encoded to 4 alnum(0-9A-Za-z) characters.
0-9A-Za-z are (always) encoded to 2 alnums. They will be simply prefixed with '0'(U+000030).
When the generating 'Non-ASCII.pyd' with MSVC toolkit, the report 'LINK : warning LNK4232' will be raised on linking. But no problem. The warning says "When someone try to link with LoadLibraryEx'A' function, it may or may not fail, depends on the user locale." Our Python.exe uses always LoadLibraryEx'W', it never fail with locale issue.

Or if you have any question, please tell me that.

Regards,
Suzumizaki-Kimitaka
History
Date User Action Args
2014-02-02 14:20:28Suzumizakisetrecipients: + Suzumizaki
2014-02-02 14:20:28Suzumizakisetmessageid: <1391350828.19.0.622173065149.issue20485@psf.upfronthosting.co.za>
2014-02-02 14:20:28Suzumizakilinkissue20485 messages
2014-02-02 14:20:27Suzumizakicreate