classification
Title: Port distutils as found in Python 2.7 to Python 3.x.
Type: behavior Stage: resolved
Components: Distutils Versions: Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: jason.coombs Nosy List: akuchling, eric.araujo, haypo, jason.coombs, ned.deily, python-dev, tarek
Priority: normal Keywords:

Created on 2013-11-10 16:59 by jason.coombs, last changed 2013-12-30 21:09 by python-dev. This issue is now closed.

Files
File name Uploaded Description Edit
ticket-list.txt akuchling, 2013-11-10 20:47
python_2.7_distutil_commits.html jason.coombs, 2013-11-10 22:53
Messages (20)
msg202534 - (view) Author: Jason R. Coombs (jason.coombs) * (Python committer) Date: 2013-11-10 16:59
Following from issue7457, in which a single feature was identified to have gone missing in 29a3eda89995, this ticket captures the need to bring the Python 3 codebase up to match Python 2.7.
msg202571 - (view) Author: A.M. Kuchling (akuchling) * (Python committer) Date: 2013-11-10 20:47
I went through Python 2.7's Misc/NEWS file and collected the entries for Distutils-related issues that were applied.  Perhaps we can check the individual entries on this list, and see which ones are still present in Python 3.x and which ones got reverted.
msg202572 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-11-10 20:48
I wouldn't trust the NEWS items.  I think the only reliable thing to do is diff each file, unfortunately.
msg202574 - (view) Author: A.M. Kuchling (akuchling) * (Python committer) Date: 2013-11-10 21:23
Issue #11104 also made some functionality work in both 2.7 and 3.2, though it's not clear that the problem stemmed from the distutils2 revert.
msg202580 - (view) Author: Jason R. Coombs (jason.coombs) * (Python committer) Date: 2013-11-10 22:53
After spending several hours spelunking, we identified what we believe are the tickets that were backed out in the aforementioned reversion.

issue1180
issue6516
issue7457
issue6466
issue6286

Additionally, issue6377 (renaming .compiler to .compiler_obj) was reverted, but it likely should not be re-applied.

Attached is an export of the etherpad (http://beta.etherpad.org/p/python_2.7_distutil_commits) which we used to keep track of the changes and show our work.

We will flag the above tickets and address each individually.
msg202585 - (view) Author: Roundup Robot (python-dev) Date: 2013-11-10 23:22
New changeset e19441e540ca by Jason R. Coombs in branch '3.3':
Issue 19544 and Issue #7457: Restore the read_pkg_file method to distutils.dist.DistributionMetadata accidentally removed in the undo of distutils2.
http://hg.python.org/cpython/rev/e19441e540ca

New changeset 28059d8b395b by Jason R. Coombs in branch 'default':
Merge with 3.3 for Issue #19544 and Issue #7457
http://hg.python.org/cpython/rev/28059d8b395b
msg202587 - (view) Author: A.M. Kuchling (akuchling) * (Python committer) Date: 2013-11-10 23:45
Patches for the default branch have been added to issue1180 (option to ignore ~/.pydistutils.cfg) and issue6516 (setting the owner/group in Distutils-built tarballs).  Please double-check those patches; I can apply them.
msg202590 - (view) Author: Roundup Robot (python-dev) Date: 2013-11-11 00:00
New changeset 5e98c4e9c909 by Jason R. Coombs in branch '3.3':
Issue #19544 and Issue #6286: Restore use of urllib over http allowing use of http_proxy for Distutils upload command, a feature accidentally lost in the rollback of distutils2.
http://hg.python.org/cpython/rev/5e98c4e9c909

New changeset b1244046f37a by Jason R. Coombs in branch 'default':
Merge with 3.3 for Issue #19544 and Issue #6286. Merge is untested. I was unable to test due to bab0cbf86835.
http://hg.python.org/cpython/rev/b1244046f37a
msg202592 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-11-11 00:54
b1244046f37a appears to have broken buildbots.  See, for example:

http://buildbot.python.org/all/builders/AMD64%20Ubuntu%20LTS%203.x/builds/2984
msg202597 - (view) Author: Jason R. Coombs (jason.coombs) * (Python committer) Date: 2013-11-11 02:05
Thanks Ned. I did see that and have pushed 394ed9deebd4. I believe that corrects the only test failure.
msg202966 - (view) Author: Roundup Robot (python-dev) Date: 2013-11-15 18:07
New changeset b9c9c4b2effe by Andrew Kuchling in branch 'default':
Issue #19544 and Issue #6516: Restore support for --user and --group parameters to sdist command as found in Python 2.7 and originally slated for Python 3.2 but accidentally rolled back as part of the distutils2 rollback. Closes Issue #6516.
http://hg.python.org/cpython/rev/b9c9c4b2effe
msg202973 - (view) Author: Roundup Robot (python-dev) Date: 2013-11-15 22:08
New changeset b08868fd5994 by Christian Heimes in branch 'default':
Issue #19544 and Issue #6516: quick workaround for failing builds
http://hg.python.org/cpython/rev/b08868fd5994
msg202975 - (view) Author: Roundup Robot (python-dev) Date: 2013-11-15 22:13
New changeset 015463176d2e by Victor Stinner in branch 'default':
Issue #19544, #6516: no need to catch AttributeError on import pwd/grp
http://hg.python.org/cpython/rev/015463176d2e
msg202979 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-11-15 23:39
At changeset 015463176d2e2530e4f07cfbe97e41abac540a57, test_make_distribution_owner_group() was failing on some buildbots. test_distutils works fine on my Linux box, I ran the test as my haypo user and as root.

http://buildbot.python.org/all/builders/AMD64%20FreeBSD%209.0%203.x/builds/5780/steps/test/logs/stdio

======================================================================
FAIL: test_make_distribution_owner_group (distutils.tests.test_sdist.SDistTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/buildbot/buildarea/3.x.krah-freebsd/build/Lib/distutils/tests/test_sdist.py", line 477, in test_make_distribution_owner_group
    self.assertEquals(member.gid, os.getgid())
AssertionError: 0 != 1002

http://buildbot.python.org/all/builders/AMD64%20Snow%20Leop%203.x/builds/470/steps/test/logs/stdio

======================================================================
FAIL: test_make_distribution_owner_group (distutils.tests.test_sdist.SDistTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/distutils/tests/test_sdist.py", line 477, in test_make_distribution_owner_group
    self.assertEquals(member.gid, os.getgid())
AssertionError: 0 != 20
msg202981 - (view) Author: Roundup Robot (python-dev) Date: 2013-11-15 23:57
New changeset c35311fcc967 by Andrew Kuchling in branch 'default':
Issue #19544 and Issue #1180: Restore global option to ignore  ~/.pydistutils.cfg in Distutils, accidentally removed in backout of distutils2 changes.
http://hg.python.org/cpython/rev/c35311fcc967
msg202986 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-11-16 00:12
> At changeset 015463176d2e2530e4f07cfbe97e41abac540a57, test_make_distribution_owner_group() was failing on some buildbots.

The problem is that tempfile.mkdtemp() creates a directory with the
group 0. Files created in this directory also have the group 0.

The test uses os.getpid() to get the group identifier.

The test should use os.stat(self.tmp_dir).st_gid (and st_uid) instead
of os.getgid(), or the tmp_dir directory. Or the group of the
temporary directory should be changed to os.getgid().

I don't know if distutils should create tarball with the group 0 or
the group of the current user.
msg202989 - (view) Author: Jason R. Coombs (jason.coombs) * (Python committer) Date: 2013-11-16 00:18
Thanks Victor.

Yes, it appears that there's yet another unported issue #7408, a follow-up to #6516.
msg202997 - (view) Author: Jason R. Coombs (jason.coombs) * (Python committer) Date: 2013-11-16 00:51
I believe all identified issues have been ported/fixed.
msg207109 - (view) Author: Roundup Robot (python-dev) Date: 2013-12-30 20:40
New changeset 83f12a9593db by Zachary Ware in branch 'default':
Issue #19544, #6516: check ZLIB_SUPPORT, not zlib (which might not be bound)
http://hg.python.org/cpython/rev/83f12a9593db
msg207113 - (view) Author: Roundup Robot (python-dev) Date: 2013-12-30 21:09
New changeset 2a872126f4a1 by Zachary Ware in branch 'default':
Issue #19544, #6516: check ZLIB_SUPPORT, not zlib (which might not be bound)
http://hg.python.org/cpython/rev/2a872126f4a1
History
Date User Action Args
2013-12-30 21:09:27python-devsetmessages: + msg207113
2013-12-30 20:40:06python-devsetmessages: + msg207109
2013-11-16 00:51:55jason.coombssetstatus: open -> closed
type: behavior
messages: + msg202997

assignee: eric.araujo -> jason.coombs
resolution: fixed
stage: resolved
2013-11-16 00:18:20jason.coombssetmessages: + msg202989
2013-11-16 00:12:15hayposetmessages: + msg202986
2013-11-15 23:57:09python-devsetmessages: + msg202981
2013-11-15 23:39:22hayposetnosy: + haypo
messages: + msg202979
2013-11-15 22:13:28python-devsetmessages: + msg202975
2013-11-15 22:08:30python-devsetmessages: + msg202973
2013-11-15 18:07:00python-devsetmessages: + msg202966
2013-11-11 02:05:32jason.coombssetmessages: + msg202597
2013-11-11 00:54:07ned.deilysetmessages: + msg202592
2013-11-11 00:00:38python-devsetmessages: + msg202590
2013-11-10 23:45:12akuchlingsetmessages: + msg202587
2013-11-10 23:22:20python-devsetnosy: + python-dev
messages: + msg202585
2013-11-10 22:53:11jason.coombssetfiles: + python_2.7_distutil_commits.html

messages: + msg202580
2013-11-10 21:23:27akuchlingsetmessages: + msg202574
2013-11-10 20:48:31ned.deilysetnosy: + ned.deily
messages: + msg202572
2013-11-10 20:47:19akuchlingsetfiles: + ticket-list.txt
nosy: + akuchling
messages: + msg202571

2013-11-10 16:59:52jason.coombscreate