http://svn.python.org/view?view=rev&revision=75384

IIRC, the relevant modules are pkg_resources, setuptools.sandbox, and setuptools.command.easy_install.

(Btw, "regressed" probably isn't the right word -- I doubt that these corner cases have ever worked correctly in Distribute.)

PJE: This too appears to have regressed in Distribute. Could you tell us which patch fixed this issue in setuptools-0.6c10 so we can figure out why that patch either didn't make it into Distribute or if the behavior subsequently regressed?

setuptools 0.6c10 is released with a fix for this issue.

The following tickets for Tahoe-LAFS may be related.  Hopefully someone will test 
setuptools-0.6c10 with Tahoe-LAFS and report back. 
http://allmydata.org/trac/tahoe/ticket/717 
http://allmydata.org/trac/tahoe/ticket/657

Ok, I've identified what this is, and it has the same root causes as issue 40. 
A patch is on the way.

Hm, I have a new problem which is intimately related to this patch.  The problem
is that package A setup_requires packages B and C, and package B setup_requires
package D, and package C also setup_requires package D.

All four packages are available as sdists.

I get an import error from C's attempt to use D when C is being built.

A lot of investigation in setuptools and pkg_resources suggests to me that what
is happening is that D is added to the pkg_resources working set during the
build of B, and then D's entry in sys.modules is removed by the sandboxing
mechanism described in this ticket (I'm using that patch), and then when C is
built it sees D in the working set so it doesn't attempt to rebuild or active
it, but when it tries to import it, it gets an import error.

Also, by the way, D is installed in .egg form into a temporary directory which
is being used to build B, and that directory is destroyed after B is built, so
by the time C is being built the D.egg is no longer available on the filesystem.

My current thinking is that sandboxing by subprocess would solve this particular
issue as well as being more robust again unanticipated side-effects like this in
general, so I'd like to try a patch that does that.

However, right now, I am instead going to modify by B and C tools so that they
do not require my D tool.

For reference, A == tahoe, B == setuptools_darcs, C == setuptools_trial, and D
== darcsver.

Oh!  I just realized that you didn't include this fix in setuptools-0.6c9. 
cgalvan made the same mistake -- thinking that this was fixed in the release.

Anyway, I just tested this patch on Nevow and the patch worked -- Nevow is now
able to build.

Thanks!

Here's the ticket to fix Nevow to not import nevow (nor twisted nor anything
else outside of the Python Standard Library) in its setup.py:

http://divmod.org/trac/ticket/2699 # build nevow without importing nevow

Thanks for working on this, PJE.

I'm working on a patch to submit to Nevow, but it looks like this might
necessitate a change to the Epsilon release management tool, which is also used
by many other Divmod projects, so it might be hard to get this patch accepted. 
My patch has a higher chance of being accepted if I can explain why it is a bad
idea for a project to import its dependencies during setup.py execution, even
when using distutils.  Can you explain why that is a bad idea or point to some
best-practices doc which warns against it?

Thanks.

Actually, reversing the order doesn't help; the issue isn't the order in which
the requirements are processed, it's the fact that none of the requirements are
added to sys.path until all of them have been built.  The only workaround for
this is to separate the installation of Twisted and Nevow, until such time as
Nevow uses an execfile on its nevow/_version.py to get version information
instead of by importing it.

It appears that Nevow is broken here: it's trying to import twisted.python
during the initial run of its setup script.  You can work around this by
reversing the order of the requirements.  (Currently, pkg_resources resolves
requirements in *reverse* input order.)

However, please note that Nevow is definitely broken; it should not be getting
its version information by importing itself.  Packages with dependencies should
*never* import from those dependencies during setup.py execution, whether
they're using setuptools or distutils.

At this point, I think it's established that the original problem (inability to
build+install in same process) is fixed, so I'm marking this resolved.

Cohfirmed: when I do "sudo python ./setup.py install" using the minimal setup.py
that you posted in msg133, with current setuptools trunk, then it emits:

Traceback (most recent call last):
  File "./setup.py", line 2, in <module>
    setup(name="foo", setup_requires=["Twisted", "Nevow"])
  File "/usr/lib/python2.5/distutils/core.py", line 112, in setup
    _setup_distribution = dist = klass(attrs)
  File "build/bdist.linux-x86_64/egg/setuptools/dist.py", line 219, in __init__
  File "build/bdist.linux-x86_64/egg/setuptools/dist.py", line 243, in
fetch_build_eggs
  File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 522, in resolve
    dist = best[req.key] = env.best_match(req, self, installer)
  File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 758, in best_match
    return self.obtain(req, installer) # try and download/install
  File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 770, in obtain
    return installer(requirement)
  File "build/bdist.linux-x86_64/egg/setuptools/dist.py", line 286, in
fetch_build_egg
  File "/usr/lib/python2.5/site-packages/easy_install.py", line 446, in easy_install
    
  File "/usr/lib/python2.5/site-packages/easy_install.py", line 476, in install_item
    
  File "/usr/lib/python2.5/site-packages/easy_install.py", line 655, in install_eggs
    
  File "/usr/lib/python2.5/site-packages/easy_install.py", line 930, in
build_and_install
    
  File "/usr/lib/python2.5/site-packages/easy_install.py", line 919, in run_setup
    
  File "build/bdist.linux-x86_64/egg/setuptools/sandbox.py", line 24, in run_setup
  File "build/bdist.linux-x86_64/egg/setuptools/sandbox.py", line 63, in run
  File "build/bdist.linux-x86_64/egg/setuptools/sandbox.py", line 26, in <lambda>
  File "setup.py", line 8, in <module>
    
  File "/tmp/easy_install-kgSK5v/Nevow-0.9.31/setupcommon.py", line 2, in <module>
  File "/usr/lib/python2.5/site-packages/PIL/__init__.py", line 5, in <module>
    # package placeholder
  File "/tmp/easy_install-kgSK5v/Nevow-0.9.31/nevow/_version.py", line 2, in
<module>
ImportError: No module named twisted.python

Here is the minimal setup.py I used to reproduce the same "No module
twisted.python" ImportErrot that was mentioned in the original ticket comment.

"""
from setuptools import setup
setup(name="foo", setup_requires=["Twisted", "Nevow"])
"""

This works for me.  Here is a corrected version of a setup.py file (there was a
cut-and-paste error in the version that I posted in msg35).

"""
import setuptools
setuptools.setup(name="minproj_that_needs_nevow", setup_requires=["Twisted"],
                 install_requires=["Nevow"])
"""

When I do "sudo python ./setup.py install" on that file, it fails with
setuptools 0.6c8 but works correctly with current setuptools trunk.

What do you do which yields that ImportError: No module named twisted.python",
cgalvan?

It turns out I was wrong, the fix currently in the trunk only fixes the
"ImportError: No module named components".  But, I still am still getting the
"ImportError: No module named twisted.python" now :/

I can confirm that this fix works for this problem :)

Candidate fix in trunk r65949; please test ASAP to confirm fix.

You needn't bother testing it; it's not compatible with Python 2.3, 
which setuptools still supports.  (The 'set' builtin was added in 2.4.)

Here's the patch that I haven't gotten around to testing yet:

=== modified file 'setuptools/setuptools/sandbox.py'A
--- setuptools/setuptools/sandbox.py	2008-06-22 08:59:18 +0000
+++ setuptools/setuptools/sandbox.py	2008-08-01 21:37:57 +0000
@@ -9,6 +9,7 @@
 def run_setup(setup_script, args):
     """Run a distutils setup script, sandboxed in its directory"""
 
+    old_sysmoduleskeys = set(sys.modules.keys())
     old_dir = os.getcwd()
     save_argv = sys.argv[:]
     save_path = sys.path[:]
@@ -38,6 +39,10 @@
         sys.path[:] = save_path
         sys.argv[:] = save_argv
         tempfile.tempdir = save_tmp
+        delkeys = set(sys.modules.keys()) - old_sysmoduleskeys
+        for delkey in delkeys:
+            print "xxx deleting sys.modules[%s]" % delkey
+            del sys.modules[delkey]
 
 class AbstractSandbox:
     """Wrap 'os' module and 'open()' builtin for virtualizing setup scripts"""

I still think the Right Way to keep your state separate from your other state in
a case like this is to use a subprocess.

But I'm willing to write a patch that does this sys.modules resetting.  Actually
I've already written one, but I haven't tested it yet.

setuptools.sandbox should save/restore sys.modules by saving a copy, then using
.update() to copy the copy back into sys.modules after, and deleting any keys
that are not in the copy.  Something like:

before:
save = sys.modules.copy()

after:
sys.modules.update(save)
for key in list(sys.modules):
    if key not in save:
        del sys.modules[key]

Here is the ticket to track this issue as it effects the Tahoe Least-Authority
Filesystem project:

http://allmydata.org/trac/tahoe/ticket/455 # setuptools sandbox isn't tight
enough to hold Twisted's sand in so that it doesn't get in Nevow's eyes

Okay Brian Warner and I just tracked this issue down a bit, since it is blocking
the release of the next version of the Tahoe Least-Authority Filesystem.

It turns out that when setuptools builds Twisted from source in a "setuptools
sandbox", that this causes the import of a few things including "twisted" and
"twisted._version".  These get added to sys.modules, and then this side-effect
on sys.modules is not undone after Twisted has been built and installed, so in a
sense changes to sys.modules can "leak" out of the sandbox.

Nevow's build process then tries to import twisted.components, which was not
imported by Twisted's build process, so it isn't already in sys.modules, but
also cannot be imported from the newly installed Twisted, because "twisted" is
already in sys.modules (with a directory location of ".", which refers to a
temporary directory when it was being built which has since been deleted).

This is why Nevow cannot be installed in the same setuptools process as Twisted was.

A good way to improve this would be to make the setuptools sandbox "tighter" --
make it so that the act of building and installing one distribution doesn't have
side-effects on the sys.modules which effect the attempt to later build another
distribution.  A good way to do this is to use a subprocess.  If you don't want
the code that you are about to execute to have side-effects on your Python
state, then execute that code in a subprocess.  Another way to fix this would be
to figure out which names had been added to sys.modules during the "build in a
sandbox" and clean them out afterward.

Here's the ticket to track the impact of this issue on allmydata.org Tahoe:

http://allmydata.org/trac/tahoe/ticket/440

It's possible that this issue has something to do with this issue:

http://allmydata.org/trac/setuptools/ticket/3 # setuptools invokes system-wide
setuptools instead of local setuptools, possibly due to interference from nevow

If you try to install this minimal project:

"""
import setuptools
setuptools.setup(name="minproj_that_needs_nevow", install_requires=["Nevow"]
"""

You'll get a reasonable-looking traceback that ends with an ImportError in which
Nevow tried to import Twisted at install time.  (Note that Nevow does not
declare its install-time dependency on Twisted -- see
http://divmod.org/trac/ticket/2629 .)

"""
  File
"/usr/local/stow/python-release25-maint-2008-05-30/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg/setuptools/command/easy_install.py",
line 427, in easy_install
    return self.install_item(None, spec, tmpdir, deps, True)
  File
"/usr/local/stow/python-release25-maint-2008-05-30/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg/setuptools/command/easy_install.py",
line 478, in install_item
    self.process_distribution(spec, dist, deps)
  File
"/usr/local/stow/python-release25-maint-2008-05-30/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg/setuptools/command/easy_install.py",
line 519, in process_distribution
    [requirement], self.local_index, self.easy_install
  File
"/usr/local/stow/python-release25-maint-2008-05-30/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg/pkg_resources.py",
line 522, in resolve
    dist = best[req.key] = env.best_match(req, self, installer)
  File
"/usr/local/stow/python-release25-maint-2008-05-30/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg/pkg_resources.py",
line 758, in best_match
    return self.obtain(req, installer) # try and download/install
  File
"/usr/local/stow/python-release25-maint-2008-05-30/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg/pkg_resources.py",
line 770, in obtain
    return installer(requirement)
  File
"/usr/local/stow/python-release25-maint-2008-05-30/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg/setuptools/command/easy_install.py",
line 446, in easy_install
    return self.install_item(spec, dist.location, tmpdir, deps)
  File
"/usr/local/stow/python-release25-maint-2008-05-30/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg/setuptools/command/easy_install.py",
line 476, in install_item
    dists = self.install_eggs(spec, download, tmpdir)
  File
"/usr/local/stow/python-release25-maint-2008-05-30/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg/setuptools/command/easy_install.py",
line 655, in install_eggs
    return self.build_and_install(setup_script, setup_base)
  File
"/usr/local/stow/python-release25-maint-2008-05-30/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg/setuptools/command/easy_install.py",
line 930, in build_and_install 
    self.run_setup(setup_script, setup_base, args)
  File
"/usr/local/stow/python-release25-maint-2008-05-30/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg/setuptools/command/easy_install.py",
line 919, in run_setup
    run_setup(setup_script, args)
  File
"/usr/local/stow/python-release25-maint-2008-05-30/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg/setuptools/sandbox.py",
line 27, in run_setup
    lambda: execfile(
  File
"/usr/local/stow/python-release25-maint-2008-05-30/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg/setuptools/sandbox.py",
line 63, in run
    return func()
  File
"/usr/local/stow/python-release25-maint-2008-05-30/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg/setuptools/sandbox.py",
line 29, in <lambda>
    {'__file__':setup_script, '__name__':'__main__'}
  File "setup.py", line 8, in <module>

  File "/tmp/easy_install-1aI_wu/Nevow-0.9.31/setupcommon.py", line 2, in <module>
  File "/tmp/easy_install-1aI_wu/Nevow-0.9.31/nevow/__init__.py", line 5, in
<module>
  File "/tmp/easy_install-1aI_wu/Nevow-0.9.31/nevow/_version.py", line 2, in
<module>
ImportError: No module named twisted.python
"""

This is the same behavior as if you had simply executed "easy_install Nevow".

The _version.py file in question is visible here:

http://divmod.org/trac/browser/trunk/Nevow/nevow/_version.py?rev=14968

However, if you try to install this minimal project:

"""
import setuptools
setuptools.setup(name="minproj_that_needs_nevow", setup_requires=["Twisted"],
install_requires=["Nevow"]
"""

You'll get a slightly strange error message which suggests that Nevow got
further in its attempt to use Twisted, but not far enough:

"""
  File
"/usr/local/stow/python-release25-maint-2008-05-30/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg/setuptools/command/easy_install.py",
line 478, in install_item
    self.process_distribution(spec, dist, deps)
  File
"/usr/local/stow/python-release25-maint-2008-05-30/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg/setuptools/command/easy_install.py",
line 519, in process_distribution
    [requirement], self.local_index, self.easy_install
  File
"/usr/local/stow/python-release25-maint-2008-05-30/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg/pkg_resources.py",
line 522, in resolve
    dist = best[req.key] = env.best_match(req, self, installer)
  File
"/usr/local/stow/python-release25-maint-2008-05-30/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg/pkg_resources.py",
line 758, in best_match
    return self.obtain(req, installer) # try and download/install
  File
"/usr/local/stow/python-release25-maint-2008-05-30/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg/pkg_resources.py",
line 770, in obtain
    return installer(requirement)
  File
"/usr/local/stow/python-release25-maint-2008-05-30/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg/setuptools/command/easy_install.py",
line 446, in easy_install
    return self.install_item(spec, dist.location, tmpdir, deps)
  File
"/usr/local/stow/python-release25-maint-2008-05-30/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg/setuptools/command/easy_install.py",
line 476, in install_item
    dists = self.install_eggs(spec, download, tmpdir)
  File
"/usr/local/stow/python-release25-maint-2008-05-30/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg/setuptools/command/easy_install.py",
line 655, in install_eggs
    return self.build_and_install(setup_script, setup_base)
  File
"/usr/local/stow/python-release25-maint-2008-05-30/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg/setuptools/command/easy_install.py",
line 930, in build_and_install
    self.run_setup(setup_script, setup_base, args)
  File
"/usr/local/stow/python-release25-maint-2008-05-30/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg/setuptools/command/easy_install.py",
line 919, in run_setup
    run_setup(setup_script, args)
  File
"/usr/local/stow/python-release25-maint-2008-05-30/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg/setuptools/sandbox.py",
line 27, in run_setup   
    lambda: execfile(
  File
"/usr/local/stow/python-release25-maint-2008-05-30/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg/setuptools/sandbox.py",
line 63, in run
    return func()
  File
"/usr/local/stow/python-release25-maint-2008-05-30/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg/setuptools/sandbox.py",
line 29, in <lambda>
    {'__file__':setup_script, '__name__':'__main__'}
  File "setup.py", line 8, in <module>

  File "/tmp/easy_install-Nib-SJ/Nevow-0.9.31/setupcommon.py", line 2, in <module>
  File "/tmp/easy_install-Nib-SJ/Nevow-0.9.31/nevow/__init__.py", line 10, in
<module>
ImportError: No module named components
"""

The __init__.py file in question is here:

http://divmod.org/trac/browser/trunk/Nevow/nevow/__init__.py?rev=14968

Now if you try to install the actual project -- allmydata.org Tahoe -- from
which this minimal project was extracted:

http://allmydata.org/trac/tahoe/browser/setup.py

You'll get a bizarre error message that takes the name of PIL in vain:

"""
 File
"/home/agl/src/allmydata-tahoe-1.0.0-r2613/setuptools-0.6c8.egg/setuptools/sandbox.py",
line 27, in run_setup
 File
"/home/agl/src/allmydata-tahoe-1.0.0-r2613/setuptools-0.6c8.egg/setuptools/sandbox.py",
line 63, in run
 File
"/home/agl/src/allmydata-tahoe-1.0.0-r2613/setuptools-0.6c8.egg/setuptools/sandbox.py",
line 29, in <lambda>
 File "setup.py", line 8, in <module>
   #
 File "/tmp/easy_install-fqJkNC/Nevow-0.9.18/setupcommon.py", line 2,
in <module>
 File "/usr/lib/python2.5/site-packages/PIL/__init__.py", line 10, in <module>
   #
ImportError: No module named components
"""

Note that allmydata.org Tahoe does not use PIL in any way.  There are no hits
for "grep -r PIL ." in the Tahoe source tree.

Going back to the minimal projects, we can tell that this effect has something
to do with the presence or absence of a Twisted*.egg in the current working
directory when "./setup.py install" is invoked.  That is:

If this minimal project is invoked in an empty directory (and also when no
Twisted, Nevow, or zope.interface installed in the system), then we get the
weird error about failure to import components even though the earlier "from
twisted.python import versions" succeeded.  This will leave a Twisted .egg in
the CWD.

If it is invoked in an otherwise empty directory that has a Twisted egg, then it
will succeed (and leave a Twisted .egg in the CWD).  (Hm.  Actually now that I
look in the target install directory, I see that it installed Nevow into there
but not Twisted or zope.interface (which Twisted requires).  It seems like that
is a problem, but it is probably a separate problem from this ticket.)