Python 2.7.4 fails while extracting zip files when 'member' is a unicode path.

---
Traceback (most recent call last):
  ...
    my_zip.extract(item, tmp_folder_path)
  File "D:\Apps\Python274\lib\zipfile.py", line 1024, in extract
    return self._extract_member(member, path, pwd)
  File "D:\Apps\Python274\lib\zipfile.py", line 1057, in _extract_member
    arcname = arcname.translate(table)
TypeError: character mapping must return integer, None or unicode
---
2.7.3 had no problems because the call to translate() is new.


The following, copied from ZipFile.py, will recreate the error.
--
import string
illegal = ':<>|"?*'
table = string.maketrans(illegal, '_' * len(illegal))

arcname = "hi"
arcname = arcname.translate(table)
# ascii strings are fine

arcname = u"hi"
arcname = arcname.translate(table)
# unicode fails
# Traceback (most recent call last):
#   File "<stdin>", line 1, in <module>
# TypeError: character mapping must return integer, None or unicode
---

I tried using unicode literals for the illegal string and maketrans underscore arg, but that didn't work. Suggestions?


Here's a link to the doc for translate().
http://docs.python.org/2/library/stdtypes.html#str.translate

Apparently namelist() can return either ascii or unicode strings for its members, depending on the archive. Obviously this'd apply to literal unicode strings as well.

Oops, passing a unicode literal to extract()'s member arg wouldn't be sufficient.

The extract() method quietly converts strings to ZipInfo objects via getinfo(member_string). Then _extract_member() takes the filename attribute of that ZipInfo object, which causes problems when when THAT is unicode.

So I guess this bug only applies to archives with unicode member paths.

Attached is one such file to aid in troubleshooting.

It appears that this is a consequence of the changes in issue 6972, in particular change 4d1948689ee1.

Yes, it's my fault. Here is a patch (with test) which fixes this regression in 2.7. This is 2.7 only issue, in Python 3 arcnames always are unicode. Please test on Windows.

The 2013-04-08 patch worked on Windows XP.

Perhaps this would deserve a 2.7.5?

Yes; I won't have time for a few days, though.

I guess I will join with 3.2 and 3.3 for #17666.

Perhaps we should hold off for a week or two to see if any other critical problems show up.

Yes, although the new releases will get the standard rc period anyway.

A week's notice to push any almost ready IDLE bugfixes before the .rc's would be nice. (I am assuming there are some, but would have to ask Roger.)

New changeset d02507c9f973 by Serhiy Storchaka in branch '2.7':
Issue #17656: Fix extraction of zip files with unicode member paths.
http://hg.python.org/cpython/rev/d02507c9f973

heads-up: Tests are still failing on FreeBSD (gcc & clang) buildbots:

http://buildbot.python.org/all/builders/AMD64%20FreeBSD%209.0%20dtrace%202.7/builds/472/steps/test/logs/stdio
http://buildbot.python.org/all/builders/AMD64%20FreeBSD%209.0%20dtrace%2Bclang%202.7/builds/468/steps/test/logs/stdio

it seems like file() can't handle unicode file names on FreeBSD. The FS encoding is 'US-ASCII' on Snakebite's FreeBSD box.

> /home/cpython/users/christian.heimes/2.7/Lib/zipfile.py(1078)_extract_member()
-> with self.open(member, pwd=pwd) as source, \
(Pdb) self.open(member, pwd=pwd)
<zipfile.ZipExtFile object at 0x801eb5fd0>
(Pdb) n
> /home/cpython/users/christian.heimes/2.7/Lib/zipfile.py(1079)_extract_member()
-> file(targetpath, "wb") as target:
(Pdb) file(targetpath, "wb")
*** UnicodeEncodeError: 'ascii' codec can't encode characters in position 47-48: ordinal not in range(128)
(Pdb) sys.getfilesystemencoding()
'US-ASCII'

Here is a patch which skips test_extract_unicode_filenames if no Unicode filesystem semantics on this platform.

I guess that  test_extract_unicode_filenames_skip.patch will not fix the failing test. The test fails because u"\xf6.txt" cannot be encoded to sys.getfilesystemencoding() (which is ASCII on the FreeBSD buildbot). You should test u"\xf6.txt". You should move the try/except inside the function.

The test is still failling:

http://buildbot.python.org/all/builders/AMD64 OpenIndiana 2.7/builds/1670/steps/test/logs/stdio

"""
======================================================================
ERROR: test_extract_unicode_filenames (test.test_zipfile.TestsWithSourceFile)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/export/home/buildbot/64bits/2.7.cea-indiana-amd64/build/Lib/test/test_zipfile.py", line 436, in test_extract_unicode_filenames
    writtenfile = zipfp.extract(fname)
  File "/export/home/buildbot/64bits/2.7.cea-indiana-amd64/build/Lib/zipfile.py", line 1024, in extract
    return self._extract_member(member, path, pwd)
  File "/export/home/buildbot/64bits/2.7.cea-indiana-amd64/build/Lib/zipfile.py", line 1079, in _extract_member
    file(targetpath, "wb") as target:
UnicodeEncodeError: 'ascii' codec can't encode characters in position 85-86: ordinal not in range(128)

"""

New changeset 8952fa2c475f by Serhiy Storchaka in branch '2.7':
Issue #17656: Skip test_extract_unicode_filenames if the FS encoding
http://hg.python.org/cpython/rev/8952fa2c475f

Sorry, I thought I had corrected this test.

Shouldn't it left opened until regression fix release has released.

I don't think so. The bug is fixed, and the fix will be in the release.

http://mail.python.org/pipermail/python-dev/2013-April/125761.html asked to leave bugs open.

Ah, fair enough.