classification
Title: test_distutils fails on Mac OS X 10.5
Type: behavior Stage:
Components: Distutils Versions: Python 3.4, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: jason.coombs Nosy List: db3l, jason.coombs, loewis, ned.deily, python-dev, tarek, titus
Priority: high Keywords:

Created on 2009-11-29 17:21 by titus, last changed 2013-11-16 00:32 by jason.coombs. This issue is now closed.

Messages (18)
msg95809 - (view) Author: Titus Brown (titus) Date: 2009-11-29 17:21
Here's the error:

test_distutils
test test_distutils failed -- Traceback (most recent call last):
File
"/private/tmp/tmp8UfLPT/python27/Lib/distutils/tests/test_sdist.py",
line 342, in test_make_distribution_owner_group 
    self.assertEquals(member.gid, os.getgid())
AssertionError: 0 != 20

It has been a problem for over a week, at least.
msg95811 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2009-11-29 19:58
Also seeing it on 10.5 ...
msg95812 - (view) Author: Tarek Ziadé (tarek) * (Python committer) Date: 2009-11-29 20:36
This test makes sure that a tarball created with sidist has the same
owner/group than your current user in each one of its member. Maybe
somehow, the previous tarball creation breaks this test.

Can you try this on your side:

Comment everything in "test_make_distribution_owner_group" between these
lines (318)

   # now building a sdist

   ...

   # building a sdist again

And see if it works.  Thx!
msg95813 - (view) Author: David Bolen (db3l) Date: 2009-11-29 21:01
I don't think its OSX specific.  My FreeBSD slaves (both 6.4 and 7.2)
have been getting the same error recently (sounds like probably around
the same time frame).  The error always seems to be that member.gid (0)
is not matching os.getgid (1001).  The os.getgid call is correct in
terms of that being the group under which the build slave is running.

In digging a little deeper I think the the generated distribution is
behaving correctly, just not how the test is expecting.  On my FreeBSD
systems (and my personal OSX 10.4 system for that matter), if I make a
directory in /tmp it gets my UID, but a GID of 0 (wheel).  Whether
that's because my id is also in the wheel group or because of the sticky
bit on /tmp I'm not totally sure.  Maybe it needs both (/tmp is sticky
on my Ubuntu system but my id is not in the root group, and this doesn't
happen).

So the files being created for the second half of this test are placed
in a temporary folder with a group of wheel(0), and the files appear to
get the same ownership.  This doesn't happen elsewhere, but appears to
be special behavior beneath tmp.

Perhaps if the test did a comparison to the UID/GID of tmp_dir rather
than os.getuid()/os.getgid() it would better reflect that the tarfile
was being built with the same ownership as found in the filesystem. 
Whether that's the correct ownership is a separate question, but as long
as distutils is accurately reflecting that it shouldn't be a failure on
a distutils test.

-- David
msg95814 - (view) Author: Tarek Ziadé (tarek) * (Python committer) Date: 2009-11-29 21:21
Thanks for the digging David, I'll check for /tmp rights in that case.
What's important in distutils, is to make sure it was able to create
tarballs with various uid/gid using the new tarfile feature.
msg95815 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2009-11-29 21:45
David, could it be that the directories where a change of group occurs
also have the s-bit set? The sticky bit should have no such effect.
msg95816 - (view) Author: Titus Brown (titus) Date: 2009-11-29 21:52
I can generate the error on my iMac but not my laptop, which are
equivalent versions/upgrades of Mac OS X AFAIK.  The /tmp directory
in both has drwxrwxrwt, and the subdirectories within which I'm doing
the builds are drwxr-xr-x, so it doesn't seem to be only dir permissions...

Both accounts are user accounts with +wheel permissions.

Tarek, after commenting out those lines (starting at line 309 of my
test_sdist.py), I still get the same error on my iMac.
msg95817 - (view) Author: David Bolen (db3l) Date: 2009-11-29 21:56
From Tarek:

> Thanks for the digging David, I'll check for /tmp rights in that case.
> What's important in distutils, is to make sure it was able to create
> tarballs with various uid/gid using the new tarfile feature.

Right, and that part of the test (explicitly making uid/gid=0/0) is
passing.  It's just the default case that is failing, and only because
the test's assumption doesn't match what is actually happening in the
filesystem in terms of default group ownership.

From Martin:

> David, could it be that the directories where a change of group occurs
> also have the s-bit set? The sticky bit should have no such effect.

Not sure - I only noticed the "t" suffix on the directory "ls -ald"
listing, which the ls man page called sticky, but didn't think about it
much more than that.

-- David
msg95818 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2009-11-29 21:58
This does seem to be a long standing Unix*-flavor behavior difference.  

The open(2) man page on OS X (and presumably FreeBSD) reads:

   When a new file is created, it is given the group of the directory
   which contains it.

The Linux open(2) page has:

       O_CREAT
          If the file does not exist it will be created.  The owner  
(user  ID)  of
          the  file is set to the effective user ID of the process.  The 
group own‐
          ership (group ID) is set either to the effective group ID of 
the  process
          or to the group ID of the parent directory (depending on file 
system type
          and mount options, and the mode of the parent directory,  see  
the  mount
          options bsdgroups and sysvgroups described in mount(8)).

The Open Group Base Specifications (Issue 6, 2004) for open(1) has this 
rationale:

"The POSIX.1-1990 standard required that the group ID of a newly created 
file be set to the group ID of its parent directory or to the effective 
group ID of the creating process. FIPS 151-2 required that 
implementations provide a way to have the group ID be set to the group 
ID of the containing directory, but did not prohibit implementations 
also supporting a way to set the group ID to the effective group ID of 
the creating process. Conforming applications should not assume which 
group ID will be used. If it matters, an application can use chown() to 
set the group ID after the file is created, or determine under what 
conditions the implementation will set the desired group ID."

So the portable solution is to change the test to not assume which group 
ID is used.
msg95819 - (view) Author: David Bolen (db3l) Date: 2009-11-29 21:59
> I can generate the error on my iMac but not my laptop, which are
> equivalent versions/upgrades of Mac OS X AFAIK.  The /tmp directory
> in both has drwxrwxrwt, and the subdirectories within which I'm doing
> the builds are drwxr-xr-x, so it doesn't seem to be only dir 
> permissions...

No, it seems to be some combination of ... something.  My Ubuntu boxes
all have "t" on the /tmp directory as well, but don't seem to exhibit
this behavior (whether or not I add myself to the "root" group that
owns /tmp).

But on the FreeBSD and OSX systems I can show this just though "mkdir"
from the command line.

At some level, I'm not sure it really matters as long as the test uses
whatever actual filesystem ownership is in place checking for default
values.

-- David
msg95820 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2009-11-29 22:06
BTW, it's trivial to demonstrate the difference.

On OS X:

$ cd $HOME
$ mkdir test
$ touch test/test
$ chown :musicg test
$ touch test/test2
$ ls -l test
total 0
-rw-r-----  1 nad  staff   0 Nov 29 14:01 test
-rw-r-----  1 nad  musicg  0 Nov 29 14:01 test2

On my linux system, the same test results in:

$ ls -l test
total 0
-rw-r----- 1 nad nad 0 2009-11-29 14:02 test
-rw-r----- 1 nad nad 0 2009-11-29 14:02 test2
msg95821 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2009-11-29 22:08
> At some level, I'm not sure it really matters as long as the test uses
> whatever actual filesystem ownership is in place checking for default
> values.

I think that's difficult to implement, as the files are not there
anymore when the check is made (only the tarfile); somebody correct
me if I'm wrong. So I agree that the test for group ownership should
be dropped, in the failing case (i.e. where nothing was specified for
the tar command).
msg95822 - (view) Author: David Bolen (db3l) Date: 2009-11-29 22:15
> I think that's difficult to implement, as the files are not there
> anymore when the check is made (only the tarfile); somebody correct
> me if I'm wrong.

The files are created during setup, and I think exist through the
duration of the test, only being removed during tearDown (via
TempDirManager mixin) - otherwise this test couldn't be creating two
separate archives from the same sources.

Worst case though, just have setup grab the test file ownership
information and hang onto it in general.  But I think this specific test
could also check filesystem ownership at any point in the test.
msg95825 - (view) Author: Tarek Ziadé (tarek) * (Python committer) Date: 2009-11-29 22:30
I've dropped the gid control and just check the uid.

From Distutils PoV, as long as this uid test pass, I can defer the
complete uid/gid test to the dedicated tarfile tests.

What is important here is being able to check that I can force a
different uid/gid for sdist archives (the first part of my test) and
this works well.

Thanks everyone for the help !

done in r76588, r76590
msg95826 - (view) Author: Titus Brown (titus) Date: 2009-11-29 22:57
Fix verified, thanks!
msg202990 - (view) Author: Jason R. Coombs (jason.coombs) * (Python committer) Date: 2013-11-16 00:20
In issue #19544, #6516 was ported to Python 3.4. After doing so, this issue re-emerges.
msg202992 - (view) Author: Roundup Robot (python-dev) Date: 2013-11-16 00:29
New changeset f4b364617abc by Jason R. Coombs in branch 'default':
Issue #7408: Forward port limited test from Python 2.7, fixing failing buildbot tests on BSD-based platforms.
http://hg.python.org/cpython/rev/f4b364617abc
msg202993 - (view) Author: Jason R. Coombs (jason.coombs) * (Python committer) Date: 2013-11-16 00:32
I'm declaring this fixed and watching the buildbots for confirmation.
History
Date User Action Args
2013-11-16 00:32:46jason.coombssetstatus: open -> closed
resolution: fixed
messages: + msg202993
2013-11-16 00:29:22python-devsetnosy: + python-dev
messages: + msg202992
2013-11-16 00:20:47jason.coombssetstatus: closed -> open

assignee: tarek -> jason.coombs
versions: + Python 3.4
nosy: + jason.coombs

messages: + msg202990
resolution: accepted -> (no value)
2009-11-29 22:57:14titussetmessages: + msg95826
2009-11-29 22:30:34tareksetstatus: open -> closed

messages: + msg95825
2009-11-29 22:15:50db3lsetmessages: + msg95822
2009-11-29 22:08:45loewissetmessages: + msg95821
2009-11-29 22:06:14ned.deilysetmessages: + msg95820
2009-11-29 21:59:38db3lsetmessages: + msg95819
2009-11-29 21:58:09ned.deilysetmessages: + msg95818
2009-11-29 21:56:29db3lsetmessages: + msg95817
2009-11-29 21:52:19titussetmessages: + msg95816
2009-11-29 21:45:09loewissetnosy: + loewis
messages: + msg95815
2009-11-29 21:21:59tareksetpriority: high
resolution: accepted
messages: + msg95814
2009-11-29 21:01:44db3lsetnosy: + db3l
messages: + msg95813
2009-11-29 20:36:01tareksetmessages: + msg95812
2009-11-29 19:58:05ned.deilysetnosy: + ned.deily
messages: + msg95811
2009-11-29 17:21:08tituscreate