Created on 2009-10-05 11:28 by ned.deily, last changed 2009-10-18 23:33 by ned.deily. This issue is now closed.
|msg93591 - (view)||Author: Ned Deily (ned.deily) *||Date: 2009-10-05 11:28|
Due to a change in distutils released with Python 2.6.3, packages that use setuptools (version 0.6c9, as of this writing), or the easy_install command, to build C extension modules fail with a cryptic message ending with: ... .egg/setuptools/command/build_ext.py", line 85, in get_ext_filename ext = self.ext_map[fullname] KeyError: <module_name> Among the packages known to be affected include lxml, zope-interface, jinja2, and, hence, packages dependent on these packages (e.g. sphinx, twisted, etc.). The issue is described in greater detail in the Distribute project tracker: http://bitbucket.org/tarek/distribute/issue/41/keyerror-_speedups The solution there was to modify Distribute, a fork of setuptools, to accommodate the change in distutils. An issue has been opened on the setuptools tracker for this problem: http://bugs.python.org/setuptools/issue85 A discussion on the distutils-sig mailing list starts here: http://mail.python.org/pipermail/distutils-sig/2009-October/013534.html The distutils change forces affected users to either: (1) migrate from setuptools to Distribute, which can generally done by simply running: easy_install distribute (2) avoid upgrading to 2.6.3 until a fix for setuptools is released (or patch setuptools themselves) (3) patch or downgrade distutils to restore the previous behavior The primary purpose of this issue is to serve as a placeholder for users searching the Python issue tracker for this failure symptom.
|msg93593 - (view)||Author: Tarek Ziadé (tarek) *||Date: 2009-10-05 11:38|
> (3) patch or downgrade distutils to restore the previous behavior This is not really an option since distutils code was modified to fix some bugs, with no API change and no backward compat break. Users affected with this problems have to ask the Setuptools maintainer to change the monkey-patches done on distutils and release a new version, or have to use Distribute that fixes the problem and provide a setuptools-compatible distribution. As far as Distutils is concerned, its development and bug fixes will not be frozen to make sure a 1 year old release of a tool that does monkey-patches still work. It's up to this project to adapt its patches.
|msg93606 - (view)||Author: Guido van Rossum (gvanrossum) *||Date: 2009-10-05 15:37|
Before we start pointing fingers or throwing mud, was this bug fix which apparently breaking setuptools introduced knowing that this would break setuptools users? I'm surprised that nobody at all tried to use the release candidate with any of the listed packages. Given the number of packages that use setuptools and have been widely distributed, I expect that even if a fixed version of setuptools was released tomorrow, the damage would already be done, and most people would just have to avoid 2.6.3. How hard would it be to release a 2.6.4 which differs from 2.6.3 only by not breaking existing setuptools distros?
|msg93609 - (view)||Author: Tarek Ziadé (tarek) *||Date: 2009-10-05 16:22|
> Before we start pointing fingers or throwing mud, was this bug > fix which apparently breaking setuptools introduced knowing that > this would break setuptools users? > > I'm surprised that nobody at all tried to use the release candidate > with any of the listed packages. The final release of 2.6.3 was done very shortly after the rc, so the community didn't have enough time to test their package with the rc. What should have been done to avoid this (what what was suggested) is to have a special buildbot for this that would run distutils commands over a list of packages from the community. I have a buidbot myself that does this, but it was discontinued lately because of a lack of CPU/Bandwidth. > How hard would it be to release a 2.6.4 which differs from 2.6.3 > only by not breaking existing setuptools distros? I could work on this by making a special internal case in Distutils to avoid calling the patched API that leads to the setuptools bug. This is not hard and can be done quickly if required.
|msg93612 - (view)||Author: Tarek Ziadé (tarek) *||Date: 2009-10-05 17:36|
Done in r75256
|msg93613 - (view)||Author: Guido van Rossum (gvanrossum) *||Date: 2009-10-05 17:55|
Thanks for the quick action Tarek!!!
|msg93616 - (view)||Author: Phillip J. Eby (pje) *||Date: 2009-10-05 18:35|
I appreciate the change being made. The specific code in setuptools is relying on an erroneous reading of get_ext_filename()'s docstring (which as of 2.3 at least, implied to me at least that it is always called with an extension's full dotted name), and so I will fix it in setuptools as well. I would like to point out, however, that subclassing is not "monkeypatching". The distutils explicitly documents the ability to extend and subclass commands, and this particular subclassing was based on code provided by another distutils user, who'd extended build_ext to support building dynamic libraries. It is likely that that user, and anyone else who extended the build_ext command in a similar way (e.g. numpy's and Twisted's distutils extensions), could be affected by the change. So, this was neither a matter of monkeypatching, nor special treatment for setuptools. Setuptools is far from the first or last distutils extension ever written, and build_ext tends to get extended more than most other commands. It's important to remember subclassing use cases when modifying distutils commands; subclasses inherently rely on a stricter interface than a class's public API.
|msg93627 - (view)||Author: Ned Deily (ned.deily) *||Date: 2009-10-06 00:43|
UPDATE for users: With the change to distutils now added in r75256 and a 2.6.4 release in the works, option (3) in the original message should now read: (3) patch distutils to restore the previous behavior or install Python 2.6.4 which is expected to be released within a few weeks
|msg93628 - (view)||Author: Sridhar Ratnakumar (srid)||Date: 2009-10-06 00:53|
> a 2.6.4 release in the works Is this going to have a rc1 as usual? Have we decided on the dates both for rc1 and final?
|msg93629 - (view)||Author: Barry A. Warsaw (barry) *||Date: 2009-10-06 01:27|
yes. rc1 on 6 oct 2009 and final on 18 oct 2009 (tentative)
|msg93900 - (view)||Author: Matthias Klose (doko) *||Date: 2009-10-12 18:45|
This change shows a regression (https://bugs.launchpad.net/bugs/449734), the setup.py file might not be correct, but it did work before this change (with plain distutils). Quoting from the report: Set setup.py file looks like this: setup(name='update-manager', ... ext_modules=[Extension('UpdateManager/fdsend', ['UpdateManager/fdsend/fdsend.c'])], ... WIth 2.6.4~rc1 the logs read: copying build/lib.linux-x86_64-2.6/fdsend.so -> /tmp/buildd/update-manager-0.126.1/debian/tmp/usr/lib/python2.6/dist-packages With 2.6.3: copying build/lib.linux-x86_64-2.6/UpdateManager/fdsend.so -> /home/egon/devel/update-manager/checkout/build-area/update-manager-0.126.1/debian/tmp/usr/lib/python2.6/dist-packages/UpdateManager So it looks like python is eating a prefix. http://bugs.python.org/issue7064 is why this change was made. Reading that, it was done because setuptools assumes that the method is always called with the full dotted name, when it was just being called with the last part of it. However, update-manager isn't using a dotted module path anyway, it's passing a path. http://docs.python.org/distutils/setupscript.html#extension-names-and-packages shows that you would be expected to use ext_modules=[Extension('UpdateManager.fdsend', ['UpdateManager/fdsend/fdsend.c'])], (note . not / in the first line)
|msg93901 - (view)||Author: Barry A. Warsaw (barry) *||Date: 2009-10-12 19:01|
Tarek, Can you please look at this for 2.6.4? We might need another rc and it would be better to do that sooner rather than later. Thanks!
|msg93903 - (view)||Author: Tarek Ziadé (tarek) *||Date: 2009-10-12 20:00|
Looking at it now
|msg93904 - (view)||Author: Phillip J. Eby (pje) *||Date: 2009-10-12 20:30|
FYI, a fix allowing setuptools to work with 2.6.3 is now checked in, and will be released soon (preferably in a few days, unless new bugs turn up). If you are experiencing issues with this and would like to try the fixed version(s), please see the announcement here: http://mail.python.org/pipermail/distutils-sig/2009-October/013822.html
|msg93907 - (view)||Author: Tarek Ziadé (tarek) *||Date: 2009-10-12 22:29|
I am opening another issue to keep a record of the particular case Matthias brought.
|msg93911 - (view)||Author: Sridhar Ratnakumar (srid)||Date: 2009-10-12 23:36|
Tarek - could you give the new issue link?
|msg93925 - (view)||Author: Tarek Ziadé (tarek) *||Date: 2009-10-13 15:39|
The new issue is : #7115
|msg94228 - (view)||Author: Ned Deily (ned.deily) *||Date: 2009-10-18 23:33|
Verified that the original regression reported against 2.6.3 when using setuptools 0.6c9 has been fixed and no longer exists in 2.6.4rc2.
|2009-10-18 23:33:50||ned.deily||set||messages: + msg94228|
|2009-10-13 15:40:26||tarek||set||status: open -> closed|
|2009-10-13 15:39:57||tarek||set||messages: + msg93925|
|2009-10-12 23:36:00||srid||set||messages: + msg93911|
|2009-10-12 22:29:52||tarek||set||messages: + msg93907|
|2009-10-12 20:30:28||pje||set||messages: + msg93904|
|2009-10-12 20:00:50||tarek||set||messages: + msg93903|
|2009-10-12 19:01:03||barry||set||status: closed -> open|
priority: release blocker
messages: + msg93901
messages: + msg93900
|2009-10-06 01:27:05||barry||set||messages: + msg93629|
|2009-10-06 00:53:24||srid||set||messages: + msg93628|
|2009-10-06 00:43:20||ned.deily||set||messages: + msg93627|
|2009-10-05 20:48:30||tarek||set||status: open -> closed|
messages: + msg93616
|2009-10-05 17:55:18||gvanrossum||set||messages: + msg93613|
|2009-10-05 17:36:26||tarek||set||messages: + msg93612|
|2009-10-05 16:22:05||tarek||set||status: closed -> open|
resolution: rejected -> (no value)
messages: + msg93609
messages: + msg93606
|2009-10-05 11:38:59||tarek||set||status: open -> closed|
messages: + msg93593