classification
Title: locale.strxfrm raises MemoryError
Type: crash Stage: resolved
Components: Extension Modules Versions: Python 3.1, Python 3.2
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: drukker, loewis, pitrou
Priority: high Keywords: patch

Created on 2009-10-07 23:01 by drukker, last changed 2009-10-19 19:49 by pitrou. This issue is now closed.

Files
File name Uploaded Description Edit
strxfrm_MemoryError.patch drukker, 2009-10-07 23:01
strxfrm_fixes_and_collation_tests.patch drukker, 2009-10-17 19:29
strxfrm_fixes_and_collation_tests_2.patch drukker, 2009-10-19 15:33
Messages (10)
msg93728 - (view) Author: Derk Drukker (drukker) Date: 2009-10-07 23:01
The strxfrm function in the locale module can potentially raise a
MemoryError.

The failing malloc is in Modules/_localemodule.c, line 291.

This is because the variable n0 of type Py_ssize_t is passed to
PyArg_ParseTuple, which expects an int when PY_SSIZE_T_CLEAN is not defined.

Patch attached, which also fixes an unrelated memory leak.
msg94079 - (view) Author: Derk Drukker (drukker) Date: 2009-10-15 08:55
Could someone triage this?  And review my simple fix?  I think it should
be a release blocker.

It seems to me to be a problem on all 64-bit non-Windows platforms, or
any platform if using UCS-4.  The 4 high-order bytes of an 8-byte
Py_ssize_t variable are not initialized.

Thanks.
msg94085 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-10-15 11:51
Nice catch. I wonder why not all of our modules are PY_SSIZE_T_CLEAN in
py3k.
This function is not actually tested, could you add a test to the
regression suite?
msg94092 - (view) Author: Derk Drukker (drukker) Date: 2009-10-15 15:13
All right.

The function strcoll also isn't tested (except for issue #3303).  I'll
look into that one, too.
msg94186 - (view) Author: Derk Drukker (drukker) Date: 2009-10-17 19:29
I've added the tests.

I found that on Windows, strxfrm has been unavailable in py3k since
r61306, because of an omission in PC/pyconfig.h.  (cf. patch)

In determining whether I should first test for the existence of strcoll
(or strxfrm) in the locale module, as was done in TestMiscellaneous, I
found this is no longer needed in py3k since r61339.  (cf. patch)  (In
python2 such checks are still necessary.)
msg94242 - (view) Author: Derk Drukker (drukker) Date: 2009-10-19 15:33
The tests in TestEnUSCollation I added don't work for all encodings
(e.g. Asian ones, or ASCII).  Now checked for encodings which are known
to work.

Found and fixed a bug (result not returned) in getpreferredencoding in
Lib/locale.py.

This test is skipped on Mac, since r30377.  I've tested it on 10.3
(PPC), and it passed, so I enabled it again.

I did, however, find that FreeBSD and Mac OS X have known bugs in
wcscoll and wcsxfrm: http://www.freebsd.org/cgi/man.cgi?query=wcsxfrm
(cf. patch)
msg94254 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-10-19 19:23
Nice work, thanks!
One detail: when giving you credit, who should I mention? just "egreen"?
msg94255 - (view) Author: Derk Drukker (drukker) Date: 2009-10-19 19:35
I've entered my real name.
msg94256 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-10-19 19:35
Ok, thanks.
msg94258 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-10-19 19:49
The patch is now committed to py3k (r75539) and 3.1 (r75541), closing.
History
Date User Action Args
2009-10-19 19:49:59pitrousetstatus: open -> closed
resolution: fixed
messages: + msg94258

stage: resolved
2009-10-19 19:35:53pitrousetmessages: + msg94256
2009-10-19 19:35:18drukkersetmessages: + msg94255
2009-10-19 19:23:22pitrousetmessages: + msg94254
2009-10-19 15:33:47drukkersetfiles: + strxfrm_fixes_and_collation_tests_2.patch

messages: + msg94242
2009-10-17 19:29:27drukkersetfiles: + strxfrm_fixes_and_collation_tests.patch

messages: + msg94186
2009-10-15 15:13:37drukkersetmessages: + msg94092
2009-10-15 11:51:54pitrousetpriority: high
nosy: + pitrou
messages: + msg94085

2009-10-15 08:55:05drukkersetmessages: + msg94079
components: + Extension Modules, - Library (Lib)
2009-10-07 23:01:41drukkercreate