Created on 2009-11-29 17:21 by titus, last changed 2013-11-16 00:32 by jason.coombs. This issue is now closed.
|msg95809 - (view)||Author: Titus Brown (titus)||Date: 2009-11-29 17:21|
Here's the error: test_distutils test test_distutils failed -- Traceback (most recent call last): File "/private/tmp/tmp8UfLPT/python27/Lib/distutils/tests/test_sdist.py", line 342, in test_make_distribution_owner_group self.assertEquals(member.gid, os.getgid()) AssertionError: 0 != 20 It has been a problem for over a week, at least.
|msg95811 - (view)||Author: Ned Deily (ned.deily) *||Date: 2009-11-29 19:58|
Also seeing it on 10.5 ...
|msg95812 - (view)||Author: Tarek Ziadé (tarek) *||Date: 2009-11-29 20:36|
This test makes sure that a tarball created with sidist has the same owner/group than your current user in each one of its member. Maybe somehow, the previous tarball creation breaks this test. Can you try this on your side: Comment everything in "test_make_distribution_owner_group" between these lines (318) # now building a sdist ... # building a sdist again And see if it works. Thx!
|msg95813 - (view)||Author: David Bolen (db3l)||Date: 2009-11-29 21:01|
I don't think its OSX specific. My FreeBSD slaves (both 6.4 and 7.2) have been getting the same error recently (sounds like probably around the same time frame). The error always seems to be that member.gid (0) is not matching os.getgid (1001). The os.getgid call is correct in terms of that being the group under which the build slave is running. In digging a little deeper I think the the generated distribution is behaving correctly, just not how the test is expecting. On my FreeBSD systems (and my personal OSX 10.4 system for that matter), if I make a directory in /tmp it gets my UID, but a GID of 0 (wheel). Whether that's because my id is also in the wheel group or because of the sticky bit on /tmp I'm not totally sure. Maybe it needs both (/tmp is sticky on my Ubuntu system but my id is not in the root group, and this doesn't happen). So the files being created for the second half of this test are placed in a temporary folder with a group of wheel(0), and the files appear to get the same ownership. This doesn't happen elsewhere, but appears to be special behavior beneath tmp. Perhaps if the test did a comparison to the UID/GID of tmp_dir rather than os.getuid()/os.getgid() it would better reflect that the tarfile was being built with the same ownership as found in the filesystem. Whether that's the correct ownership is a separate question, but as long as distutils is accurately reflecting that it shouldn't be a failure on a distutils test. -- David
|msg95814 - (view)||Author: Tarek Ziadé (tarek) *||Date: 2009-11-29 21:21|
Thanks for the digging David, I'll check for /tmp rights in that case. What's important in distutils, is to make sure it was able to create tarballs with various uid/gid using the new tarfile feature.
|msg95815 - (view)||Author: Martin v. Löwis (loewis) *||Date: 2009-11-29 21:45|
David, could it be that the directories where a change of group occurs also have the s-bit set? The sticky bit should have no such effect.
|msg95816 - (view)||Author: Titus Brown (titus)||Date: 2009-11-29 21:52|
I can generate the error on my iMac but not my laptop, which are equivalent versions/upgrades of Mac OS X AFAIK. The /tmp directory in both has drwxrwxrwt, and the subdirectories within which I'm doing the builds are drwxr-xr-x, so it doesn't seem to be only dir permissions... Both accounts are user accounts with +wheel permissions. Tarek, after commenting out those lines (starting at line 309 of my test_sdist.py), I still get the same error on my iMac.
|msg95817 - (view)||Author: David Bolen (db3l)||Date: 2009-11-29 21:56|
From Tarek: > Thanks for the digging David, I'll check for /tmp rights in that case. > What's important in distutils, is to make sure it was able to create > tarballs with various uid/gid using the new tarfile feature. Right, and that part of the test (explicitly making uid/gid=0/0) is passing. It's just the default case that is failing, and only because the test's assumption doesn't match what is actually happening in the filesystem in terms of default group ownership. From Martin: > David, could it be that the directories where a change of group occurs > also have the s-bit set? The sticky bit should have no such effect. Not sure - I only noticed the "t" suffix on the directory "ls -ald" listing, which the ls man page called sticky, but didn't think about it much more than that. -- David
|msg95818 - (view)||Author: Ned Deily (ned.deily) *||Date: 2009-11-29 21:58|
This does seem to be a long standing Unix*-flavor behavior difference. The open(2) man page on OS X (and presumably FreeBSD) reads: When a new file is created, it is given the group of the directory which contains it. The Linux open(2) page has: O_CREAT If the file does not exist it will be created. The owner (user ID) of the file is set to the effective user ID of the process. The group own‐ ership (group ID) is set either to the effective group ID of the process or to the group ID of the parent directory (depending on file system type and mount options, and the mode of the parent directory, see the mount options bsdgroups and sysvgroups described in mount(8)). The Open Group Base Specifications (Issue 6, 2004) for open(1) has this rationale: "The POSIX.1-1990 standard required that the group ID of a newly created file be set to the group ID of its parent directory or to the effective group ID of the creating process. FIPS 151-2 required that implementations provide a way to have the group ID be set to the group ID of the containing directory, but did not prohibit implementations also supporting a way to set the group ID to the effective group ID of the creating process. Conforming applications should not assume which group ID will be used. If it matters, an application can use chown() to set the group ID after the file is created, or determine under what conditions the implementation will set the desired group ID." So the portable solution is to change the test to not assume which group ID is used.
|msg95819 - (view)||Author: David Bolen (db3l)||Date: 2009-11-29 21:59|
> I can generate the error on my iMac but not my laptop, which are > equivalent versions/upgrades of Mac OS X AFAIK. The /tmp directory > in both has drwxrwxrwt, and the subdirectories within which I'm doing > the builds are drwxr-xr-x, so it doesn't seem to be only dir > permissions... No, it seems to be some combination of ... something. My Ubuntu boxes all have "t" on the /tmp directory as well, but don't seem to exhibit this behavior (whether or not I add myself to the "root" group that owns /tmp). But on the FreeBSD and OSX systems I can show this just though "mkdir" from the command line. At some level, I'm not sure it really matters as long as the test uses whatever actual filesystem ownership is in place checking for default values. -- David
|msg95820 - (view)||Author: Ned Deily (ned.deily) *||Date: 2009-11-29 22:06|
BTW, it's trivial to demonstrate the difference. On OS X: $ cd $HOME $ mkdir test $ touch test/test $ chown :musicg test $ touch test/test2 $ ls -l test total 0 -rw-r----- 1 nad staff 0 Nov 29 14:01 test -rw-r----- 1 nad musicg 0 Nov 29 14:01 test2 On my linux system, the same test results in: $ ls -l test total 0 -rw-r----- 1 nad nad 0 2009-11-29 14:02 test -rw-r----- 1 nad nad 0 2009-11-29 14:02 test2
|msg95821 - (view)||Author: Martin v. Löwis (loewis) *||Date: 2009-11-29 22:08|
> At some level, I'm not sure it really matters as long as the test uses > whatever actual filesystem ownership is in place checking for default > values. I think that's difficult to implement, as the files are not there anymore when the check is made (only the tarfile); somebody correct me if I'm wrong. So I agree that the test for group ownership should be dropped, in the failing case (i.e. where nothing was specified for the tar command).
|msg95822 - (view)||Author: David Bolen (db3l)||Date: 2009-11-29 22:15|
> I think that's difficult to implement, as the files are not there > anymore when the check is made (only the tarfile); somebody correct > me if I'm wrong. The files are created during setup, and I think exist through the duration of the test, only being removed during tearDown (via TempDirManager mixin) - otherwise this test couldn't be creating two separate archives from the same sources. Worst case though, just have setup grab the test file ownership information and hang onto it in general. But I think this specific test could also check filesystem ownership at any point in the test.
|msg95825 - (view)||Author: Tarek Ziadé (tarek) *||Date: 2009-11-29 22:30|
I've dropped the gid control and just check the uid. From Distutils PoV, as long as this uid test pass, I can defer the complete uid/gid test to the dedicated tarfile tests. What is important here is being able to check that I can force a different uid/gid for sdist archives (the first part of my test) and this works well. Thanks everyone for the help ! done in r76588, r76590
|msg95826 - (view)||Author: Titus Brown (titus)||Date: 2009-11-29 22:57|
Fix verified, thanks!
|msg202990 - (view)||Author: Jason R. Coombs (jason.coombs) *||Date: 2013-11-16 00:20|
|msg202992 - (view)||Author: Roundup Robot (python-dev)||Date: 2013-11-16 00:29|
New changeset f4b364617abc by Jason R. Coombs in branch 'default': Issue #7408: Forward port limited test from Python 2.7, fixing failing buildbot tests on BSD-based platforms. http://hg.python.org/cpython/rev/f4b364617abc
|msg202993 - (view)||Author: Jason R. Coombs (jason.coombs) *||Date: 2013-11-16 00:32|
I'm declaring this fixed and watching the buildbots for confirmation.
|2013-11-16 00:32:46||jason.coombs||set||status: open -> closed|
messages: + msg202993
messages: + msg202992
|2013-11-16 00:20:47||jason.coombs||set||status: closed -> open|
assignee: tarek -> jason.coombs
versions: + Python 3.4
nosy: + jason.coombs
messages: + msg202990
resolution: accepted -> (no value)
|2009-11-29 22:57:14||titus||set||messages: + msg95826|
|2009-11-29 22:30:34||tarek||set||status: open -> closed|
messages: + msg95825
|2009-11-29 22:15:50||db3l||set||messages: + msg95822|
|2009-11-29 22:08:45||loewis||set||messages: + msg95821|
|2009-11-29 22:06:14||ned.deily||set||messages: + msg95820|
|2009-11-29 21:59:38||db3l||set||messages: + msg95819|
|2009-11-29 21:58:09||ned.deily||set||messages: + msg95818|
|2009-11-29 21:56:29||db3l||set||messages: + msg95817|
|2009-11-29 21:52:19||titus||set||messages: + msg95816|
messages: + msg95815
|2009-11-29 21:21:59||tarek||set||priority: high|
messages: + msg95814
messages: + msg95813
|2009-11-29 20:36:01||tarek||set||messages: + msg95812|
messages: + msg95811