Title: UnicodeDecodeError when creating tar.gz with unicode name
Type: behavior Stage:
Components: Versions: Python 2.7
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: lars.gustaebel Nosy List: jaraco, lars.gustaebel, python-dev, terry.reedy, vstinner
Priority: low Keywords: patch

Created on 2011-12-20 01:23 by jaraco, last changed 2011-12-26 17:22 by python-dev. This issue is now closed.

File name Uploaded Description Edit
tarfile-stream-gzip-unicode-fix.diff lars.gustaebel, 2011-12-21 09:42
smime.p7s jaraco, 2011-12-25 17:00
Messages (15)
msg149896 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2011-12-20 01:23
python -c "import tarfile;'hello.tar.gz', 'w|gz')"


Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\jaraco\projects\public\cpython\Lib\", line 1687, in open
    _Stream(name, filemode, comptype, fileobj, bufsize),
  File "C:\Users\jaraco\projects\public\cpython\Lib\", line 431, in __init__
  File "C:\Users\jaraco\projects\public\cpython\Lib\", line 459, in _init_write_gz
    self.__write( + NUL)
  File "C:\Users\jaraco\projects\public\cpython\Lib\", line 475, in __write
    self.buf += s
UnicodeDecodeError: 'ascii' codec can't decode byte 0x8b in position 1: ordinal not in range(128)

Remove the compression ('|gz') or remove the unicode name or run under Python 3 and the command completes without error.

The error does not occur under Python 3 (even with non-ascii characters), so it should be possible to create a tarfile with a unicode filename on Python 2.7.

This failure is the underlying cause of #11638.
msg149974 - (view) Author: Lars Gustäbel (lars.gustaebel) * (Python committer) Date: 2011-12-21 09:42
tarfile under Python 2.x is not particularly designed to support unicode filenames (the gzip module does not support them either), but that should not be too hard to fix.
msg149985 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2011-12-21 13:33
That looks like a good patch to me. Do you want to commit it, or would you rather I do?
msg150032 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2011-12-21 18:28
New changeset a60a3610a97b by Lars Gustäbel in branch '2.7':
Issue #13639: Accept unicode filenames in"w|gz").
msg150035 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-12-21 19:07
+ ="iso-8859-1", "replace")

Why did you chose ISO-8859-1? I think that the filesystem encoding should be used instead:

-   ="iso-8859-1", "replace")
+   =, "replace")
msg150037 - (view) Author: Lars Gustäbel (lars.gustaebel) * (Python committer) Date: 2011-12-21 19:31
msg150038 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-12-21 19:36
"The gzip format (defined in RFC 1952) allows storing the original filename (without the .gz suffix) in an additional field in the header (the FNAME field). Latin-1 (iso-8859-1) is required."

Hum, it looks like the author of the gzip program (on Linux Fedora 16) didn't read the RFC!

$ tar -cvf hého.tar README
$ gzip hého.tar 
$ hachoir-urwid ~/prog/python/default/hého.tar.gz 
0) file:/home/haypo/prog/python/default/hého.tar.gz: ...
   0) signature= "\x1f\x8b": GZip file signature (\x1F\x8B) (2 bytes)
   2) compression= deflate: Compression method (1 byte)
   3.0) is_text= False: File content is probably ASCII text (1 bit)
   3.1) has_crc16= False: Header CRC16 (1 bit)
   3.2) has_extra= False: Extra informations (variable size) (1 bit)
   3.3) has_filename= True: Contains filename? (1 bit)
   3.4) has_comment= False: Contains comment? (1 bit)
   3.5) reserved[0]= <null> (3 bits)
   4) mtime= 2011-12-21 19:34:54: Modification time (4 bytes)
   8.0) reserved[1]= <null> (1 bit)
   8.1) slowest= False: Compressor used maximum compression (slowest) (1 bit)
   8.2) fastest= False: Compressor used the fastest compression (1 bit)
   8.3) reserved[2]= <null> (5 bits)
   9) os= Unix: Operating system (1 byte)
   10) filename= "hého.tar": Filename (10 bytes)

Raw display:

   10) filename= "h\xc3\xa9ho.tar\0": Filename (10 bytes)
msg150203 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-12-24 02:10
2.7 is closed to new features. This looks like it mignt be one. The 2.7 doc for says "Return a TarFile object for the pathname name." Does the meaning of 'pathname' in 2.7 generally include unicode as well as str objects? (It is not in the Glossary.) 

> The error does not occur under Python 3 (even with non-ascii characters), so it should be possible to create a tarfile with a unicode filename on Python 2.7.

Python 3 has many new features that are not in 2.7, so 'possible' is not exactly the point ;-).
msg150228 - (view) Author: Lars Gustäbel (lars.gustaebel) * (Python committer) Date: 2011-12-24 11:12
I thought about that myself, too. It is clearly no new feature, it is really more some kind of a fix.

Unicode pathnames given to are just passed through to the open() function, which is why this always has been working, except for this particular case. There are 6 different possible write modes: "w:", "w:gz", "w:bz2", "w|", "w|gz" and "w|bz2". And the only one not working with a unicode pathname is "w|gz". Although admittedly is not supposed to be used with a unicode path, people do it anyway, because they don't care, and because it works. The patch does not add a new broad functionality, it merely harmonises the way the six write modes work.

Neither can we retroactively enforce using string pathnames at this point, nor should we let a user run into this strange error. The patch is very small and minimally invasive. The error message you get without the patch is completely incomprehensible.
msg150237 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-12-24 21:42
With that explanation, that it is one case out of six that fails, for whatever reason, I agree.

That leaves the issue of whether the fix is the right one. I currently agree with Victor that we should do what the rest of Python does and what is most universally useful. That fact that an old standard requires a *storage* encoding for a nearly unused field for .gz files that (I believe) only works for Western Europe, does not mean we should use it for *opening* .tar files. WestEuro-centrism is as bad as Anglo-centrism. If the unicode filename cannot be Latin-1 encoded, the filename field should be left blank. But it seems to me that the filename should be converted to the bytes that the user wants, expects, and can use.
msg150248 - (view) Author: Lars Gustäbel (lars.gustaebel) * (Python committer) Date: 2011-12-25 11:26
I think we should wrap this up as soon as possible, because it has already absorbed too much of our time. The issue we discuss here is a tiny glitch triggered by a corner-case. My original idea was to fix it in a minimal sort of way that is backwards-compatible.

There are at least 4 different solutions now:

1. Keep the patch.
2. Revert the patch, leave everything as it was as wontfix.
3. Don't write an FNAME field at all if the filename that is passed is a unicode string.
4. Rewrite the FNAME code the way Terry suggests. This seems to me like the most complex solution, because we have to fix as well, because the code in question was originally taken from the gzip module. (BTW, both the tarfile and gzip module discard the FNAME field when a file is opened for reading.)

My favorites are 1 and 3 ;-)
msg150249 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2011-12-25 17:00
I also feel (1) or (3) is best for this issue. If there is a _better_ 
implementation, it should be reserved for a separate improvement to Python 

I lean slightly toward (3) because it would support filenames with Unicode 
characters other than latin-1 (as long as the file system allows it to be 
saved), because I suspect it would enable tests such as this to pass:
msg150259 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-12-26 07:26
As I understand the patched code, it only fixes the issue for unicode names that can be latin-1 encoded and that other unicode names will raise the same exception with 'latin-1' (or equivalent) substituted for 'ascii'. So it is easy for me to anticipate a new issue reporting such someday.

I would prefer a more complete fix. If 3 is easier than 4, fine with me.
msg150260 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-12-26 07:46
I just took a look as the 3.2 tarfile code and see that it always (because is always unicode) does the same encoding, with 'replace', referencing RFC1952. Although there are a few other differences, they appear inconsequential, so that the code otherwise should behave the same. Reading further on codec error handling, I gather that my previously understanding was off; non-Latin1 chars will just all appear as '?' instead of raising an exception. While that is normally useless, it does not matter since the result is not used. So I agree to call this fixed.
msg150268 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2011-12-26 17:22
New changeset dc1045d08bd8 by Jason R. Coombs in branch '2.7':
Issue #11638: Adding test to ensure .tar.gz files can be generated by sdist command with unicode metadata, based on David Barnett's patch.
Date User Action Args
2011-12-26 17:22:36python-devsetmessages: + msg150268
2011-12-26 07:46:51terry.reedysetstatus: open -> closed
resolution: fixed
messages: + msg150260
2011-12-26 07:26:24terry.reedysetmessages: + msg150259
2011-12-25 17:00:46jaracosetfiles: + smime.p7s

messages: + msg150249
2011-12-25 11:26:25lars.gustaebelsetpriority: normal -> low
assignee: lars.gustaebel
messages: + msg150248
2011-12-24 21:42:54terry.reedysettype: behavior
messages: + msg150237
2011-12-24 11:13:00lars.gustaebelsetmessages: + msg150228
2011-12-24 02:10:46terry.reedysetnosy: + terry.reedy
messages: + msg150203
2011-12-21 19:36:39vstinnersetmessages: + msg150038
2011-12-21 19:31:14lars.gustaebelsetmessages: + msg150037
2011-12-21 19:07:22vstinnersetnosy: + vstinner
messages: + msg150035
2011-12-21 18:28:22python-devsetnosy: + python-dev
messages: + msg150032
2011-12-21 13:33:50jaracosetmessages: + msg149985
2011-12-21 09:42:42lars.gustaebelsetfiles: + tarfile-stream-gzip-unicode-fix.diff
keywords: + patch
messages: + msg149974
2011-12-20 15:25:13pitrousetnosy: + lars.gustaebel
2011-12-20 01:23:23jaracocreate