Author stutzbach
Recipients Arfrever, gvanrossum, lemburg, loewis, r.david.murray, scoder, stutzbach, vstinner, zooko
Date 2010-05-09.22:14:46
SpamBayes Score 3.30043e-08
Marked as misclassified No
Message-id <1273443291.26.0.304983963221.issue8654@psf.upfronthosting.co.za>
In-reply-to
Content
Here is a new patch.

Usage, short version: 

By default, modules compile in Unicode-agnostic mode, where Py_UNICODE is an incomplete type.  Unicode-agnostic modules will load in both UCS2 and UCS4 interpreters.

If a module defines PY_REAL_PY_UNICODE before including Python.h, Py_UNICODE will be a complete type.  The module can then dereference Py_UNICODE pointers and use sizeof(Py_UNICODE).  Attempting to load the module into a mismatched interpreter will cause an ImportError.

Longer version:

In Unicode-agnostic mode, extensions can pass around Py_UNICODE pointers freely, but they will get a compilation error if they attempt to dereference the pointer or use sizeof(Py_UNICODE).  In Unicode-agnostic mode, the following are never defined: Py_UNICODE_SIZE, Py_UNICODE_WIDE, HAVE_USABLE_CHAR_T, and functions and macros take Py_UNICODE as parameter or return it.

Passing a Py_UNICODE pointer to a function that expects a wchar_t * (or something similar) will compile, but will generate an incompatible pointer types warning.

Per MvL's suggestion, the patch adds a flag field to PyModuleDef_Base and defines two flags: one to indicate if the module depends on the Unicode width, and another to indicate the width the module was compiled with.  These flags are checked in PyModule_Create2.

The patch includes unit tests to ensure that importing works or doesn't work in the correct cases.

19 of the 77 modules in Modules/ needed PY_REAL_PY_UNICODE.  

18 of them failed to compile without it (good!).

1 of them (_tkinter.c) output an "incompatible pointer types" warning without it, but compiled anyway (resulting in bad code - without evidence to the contrary it assumed Python was UCS2).

I ran the test suite with a UCS2 Python and a UCS4 Python on Linux.  I had hoped to run the test suite on Windows as well, but many tests are failing on Windows even without the patch.
History
Date User Action Args
2010-05-09 22:14:51stutzbachsetrecipients: + stutzbach, lemburg, gvanrossum, loewis, zooko, scoder, vstinner, Arfrever, r.david.murray
2010-05-09 22:14:51stutzbachsetmessageid: <1273443291.26.0.304983963221.issue8654@psf.upfronthosting.co.za>
2010-05-09 22:14:49stutzbachlinkissue8654 messages
2010-05-09 22:14:48stutzbachcreate