classification
Title: UNICODE macro in cPickle conflicts with Windows define
Type: Stage: commit review
Components: Windows Versions: Python 2.7
process
Status: closed Resolution: accepted
Dependencies: Superseder:
Assigned To: loewis Nosy List: eckhardt, loewis, rpetrov
Priority: high Keywords: patch

Created on 2008-10-06 10:28 by eckhardt, last changed 2009-01-02 20:40 by loewis. This issue is now closed.

Files
File name Uploaded Description Edit
Python-unicode-redefine-warning.0.patch eckhardt, 2008-10-06 10:35
PythonCE-2.5-20061219-cPickle.c-patch rpetrov, 2008-10-10 22:39 part of PythonCE-2.5-20061219 patch
Messages (8)
msg74363 - (view) Author: Ulrich Eckhardt (eckhardt) Date: 2008-10-06 10:28
In several places, Python is using TCHAR-based win32 APIs which has 
several consequences:
1. The actually used function depends on the _UNICODE macro, while the 
using code often assumes that this macro is not defined. That means 
that not only the code changes when the macro is defined but that it 
simply breaks.
2. Using the ANSI-variants of functions causes overhead when a Unicode 
string has to be transcoded and possibly data loss, because the ANSI 
encoding can't represent the full Unicode range. Note that I'm 
assuming an NT based system (i.e. NT, win2k etc), because the systems 
where the ANSI encoding is native (win9x, ME(?) etc) have been dropped 
from the supported platforms.
3. The main reason for me wanting a build independent of TCHAR is a 
port to MS Windows CE, which doesn't support the ANSI encoding at all. 
This means that a TCHAR-clean build for desktop win32 is a necessary 
part of the port to CE, but even without the port it would be useful 
to skip the conversion overhead and at the same time gain 
functionality. Concerning the CE port, see this thread as a reference:
http://mail.python.org/pipermail/python-dev/2008-September/082529.html

Note: I'm adding this tracker item mainly in order to attach patches 
for review, so they don't get lost like it's possible on the 
mailinglist.
msg74364 - (view) Author: Ulrich Eckhardt (eckhardt) Date: 2008-10-06 10:35
Add a patch that fixes a warning.
msg74386 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2008-10-06 19:24
Please create a separate tracker item per individual change (or per
group by enourmously similar changes). Each change will be reviewed
separately, and will get accepted or rejected separately.

This specific patch (Python-unicode-redefine-warning) is fine, but I'd
rather wait until after the release of Python 3.0, in order to be able
to merge the changes better into the 3k branch.
msg74481 - (view) Author: Roumen Petrov (rpetrov) * Date: 2008-10-07 20:23
What about to substitute UNICODE macro as example with RAWUNICODE in the
source ?
msg74482 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2008-10-07 20:34
> What about to substitute UNICODE macro as example with RAWUNICODE in the
> source ?

That would work as well. As it stands, the macro names are equivalent to
the ones in pickle.py, which is a useful property to have. So I'm in
favor of the proposed change.
msg74653 - (view) Author: Roumen Petrov (rpetrov) * Date: 2008-10-10 22:39
For the protocol I attach part for PythonCE-2.5-20061219.patch. Both
patches address UNICODE name clash in Modules/cPickle.c.

For windows programs UNICODE is more commonly used define and may be
python C-code only has to avoid issues like this one.

I couldn't found where python C-code use so called "TCHAR meaning" and
the first sentence from comment in Ulrich patch looks irrelevant to the
issue.

Also Ulrich patch is far from complete "TCHAR" implementation in python.
May be issue title has to be changed to address more precise problem.
msg74674 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2008-10-12 17:36
> I couldn't found where python C-code use so called "TCHAR meaning" and
> the first sentence from comment in Ulrich patch looks irrelevant to the
> issue.

I was confused by Ulrich's terminology at first, also (claiming that
Python doesn't use TCHAR, so how could it depend on it); I now think
Ulrich is nearly correct. E.g. in 2.6, we call, in posixmodule.c,
CreateProcess. Now, CreateProcess is defined in the SDK as

#ifdef UNICODE
#define CreateProcess  CreateProcessW
#else
#define CreateProcess  CreateProcessA
#endif // !UNICODE

However, it is *documented* as

BOOL WINAPI CreateProcess(
  __in_opt     LPCTSTR lpApplicationName,
  __inout_opt  LPTSTR lpCommandLine,
  __in_opt     LPSECURITY_ATTRIBUTES lpProcessAttributes,
  __in_opt     LPSECURITY_ATTRIBUTES lpThreadAttributes,
  __in         BOOL bInheritHandles,
  __in         DWORD dwCreationFlags,
  __in_opt     LPVOID lpEnvironment,
  __in_opt     LPCTSTR lpCurrentDirectory,
  __in         LPSTARTUPINFO lpStartupInfo,
  __out        LPPROCESS_INFORMATION lpProcessInformation
);

So according to the documentation, this is a function that takes TCHAR
parameters, hence it is a "TCHAR-based (windows) API". The fact that the
Windows SDK actually defines it differently is a technical hack; for all
practical purposes, it behaves as if it really was a TCHAR API.

> Also Ulrich patch is far from complete "TCHAR" implementation in python.

See msg74386.

> May be issue title has to be changed to address more precise problem.

Certainly.
msg78874 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2009-01-02 20:40
Thanks for the patch. Committed as r68173.
History
Date User Action Args
2009-01-02 20:40:31loewissetstatus: open -> closed
messages: + msg78874
2008-12-30 13:31:24loewissetpriority: high
assignee: loewis
resolution: accepted
stage: commit review
2008-10-12 17:37:39loewissettitle: use of TCHAR under win32 -> UNICODE macro in cPickle conflicts with Windows define
2008-10-12 17:36:31loewissetmessages: + msg74674
2008-10-10 22:39:08rpetrovsetfiles: + PythonCE-2.5-20061219-cPickle.c-patch
messages: + msg74653
2008-10-07 20:34:09loewissetmessages: + msg74482
2008-10-07 20:23:48rpetrovsetnosy: + rpetrov
messages: + msg74481
2008-10-06 19:24:42loewissetnosy: + loewis
messages: + msg74386
2008-10-06 10:35:19eckhardtsetfiles: + Python-unicode-redefine-warning.0.patch
keywords: + patch
messages: + msg74364
2008-10-06 10:28:45eckhardtcreate