Title: tarfile extract fails on OS X system python due to chown of gid=-1
Type: Stage:
Components: Versions: Python 2.7
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: Dave.Flogeras, amaury.forgeotdarc, ned.deily, ronaldoussoren
Priority: normal Keywords:

Created on 2011-11-01 18:17 by Dave.Flogeras, last changed 2011-11-03 12:39 by Dave.Flogeras. This issue is now closed.

Messages (6)
msg146800 - (view) Author: Dave Flogeras (Dave.Flogeras) Date: 2011-11-01 18:17
I am trying to unpack boost_1_46_1.tar.bz2 (you can grab it here with the following code:

import tarfile
t = "boost_1_46_1.tar.bz2" );

On OSX (both Lion and Snow Leopard, so python 2.7.1 and 2.6.1 resp.) this works as a normal user, but not as root failing with:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/", line 2028, in extractall
    self.extract(tarinfo, path)
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/", line 2065, in extract
    self._extract_member(tarinfo, os.path.join(path,
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/", line 2157, in _extract_member
    self.chown(tarinfo, targetpath)
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/", line 2269, in chown
    os.chown(targetpath, u, g)
OverflowError: signed integer is greater than maximum

I have confirmed that Python is running as a 64bit application on OSX.

On my 64bit Linux (gentoo) machine, this works both as a user and as root. I tried with Python 2.7.2 as well as downgrading to 2.7.1.

This could be related to 1215928
msg146804 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2011-11-01 20:41
It's more likely an issue with the chown call. Indeed, the tar file contains a gid of 4294967295, which is too large for an int.  Normally this was fixed by issue1747858, which was included in Python 2.6.5 and 2.7.
Are you sure you have the same error with Python 2.7?
msg146812 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2011-11-01 21:44
The issue is indeed with chown call. The call that fails has a gid of -1, truncated to 4294967295, which is a valid gid on OS X ('nogroup').  The Apple-supplied Python 2.7.1 in OS X 10.7 fails running under sudo as root:

Python 2.7.1 (r271:86832, Jun 16 2011, 16:59:05) 
[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)] on darwin
>>> os.chown('/tmp/c', 0, 4294967295)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: signed integer is greater than maximum

But the 64-bit OS X 2.7.2 and current Python 2.7 builds do not fail.  The fix for Issue1747858 should be in all of them.  Investigating further.
msg146814 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2011-11-01 22:05
For some reason, Apple is patching Modules/posixmodule.c in a way that reverts the fixes to chown in Issue1747858.  You can see the patches for their 10.7.2 python 2.7 here:

It probably should be filed as a bug to Apple. They also have a number of other patches to posixmodule in there.  Ronald, any opinions?  In the mean time, marking this issue as pending invalid.
msg146896 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2011-11-03 06:59
There's little chance that Apple will fix this issue in their build (at least not until Lion+1 gets released) because this is not a security bug. Their breaking of 2.7 in Lion is probably due to lack of review for their collection of patches, and is not something we can fix.

I'd close this as invalid as the issue is already fixed in our codebase.

I won't file a bug at Apple for this, although that shouldn't stop anyone else from doing so (and remember: is a popularity contest, the more people file an report about a bug the more likely it is that it will get fixed)
msg146926 - (view) Author: Dave Flogeras (Dave.Flogeras) Date: 2011-11-03 12:39
Thank you sirs.  I've already got a workaround for this situation, and personally don't care to waste my time with apple.  I'm satisfied that Python is handling bugs seriously and it will hopefully get into an apple release in the future.
Date User Action Args
2011-11-03 12:39:40Dave.Flogerassetstatus: open -> closed

messages: + msg146926
2011-11-03 06:59:24ronaldoussorensetstatus: pending -> open

messages: + msg146896
2011-11-01 22:09:49ned.deilysetstatus: open -> pending
2011-11-01 22:09:25ned.deilysetstatus: pending -> open
title: Unable to deal with large tarfile -> tarfile extract fails on OS X system python due to chown of gid=-1
2011-11-01 22:05:59ned.deilysetstatus: open -> pending

nosy: + ronaldoussoren
messages: + msg146814

resolution: not a bug
2011-11-01 21:44:43ned.deilysetnosy: + ned.deily
messages: + msg146812
2011-11-01 20:41:55amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg146804
2011-11-01 18:17:20Dave.Flogerascreate