The bug tracker for setuptools 0.7 or higher is on BitBucket

 

Issue36

Title warning about "module was already imported from"
Priority bug Status resolved
Superseder Nosy List pje, quentinsf, warner, wdoekes, wonlay, zgoda, zooko
Assigned To Keywords

Created on 2008-08-26.23:55:47 by zooko, last changed 2009-10-19.19:36:46 by pje.

Files
File name Uploaded Type Edit Remove
module_already_imported.diff wonlay, 2008-12-05.12:48:35 text/plain
Messages
msg428 (view) Author: pje Date: 2009-10-19.19:36:46
setuptools 0.6c10 is released with fixes for this issue.
msg381 (view) Author: pje Date: 2009-10-10.23:18:10
It appears that a big source of spurious warnings for this is when pkg_resources
is imported after other modules found in eggs.  This can be resolved by changing
the insert_on() method to only check conflicts when the distribution isn't
already on sys.path.  In other words, if you're re-adding something that's
already there, there's no point in getting the warning more than once.  In
addition, wonlay's patch should help to fix some of the remaining cases.

That leaves handling of packages.  It appears that the issue there is that the
setup.py for 'foolscap' is broken: its 'packages' keyword contains *paths*
rather than package names, which causes setuptools to generate a hosed
'top-level.txt', which should only contains top level package names.  This is
causing attempts to check consistency of these nested packages, which then
breaks with stow.  I'll add a warning for this, so that other people won't make
this mistake with their packages in future.
msg351 (view) Author: warner Date: 2009-08-25.22:18:33
ok, here's a patch which resolves my particular problem, not sure if it's the
right thing to do or not:

diff --git a/pkg_resources.py b/pkg_resources.py
index 9edb6c0..b812371 100644
--- a/pkg_resources.py
+++ b/pkg_resources.py
@@ -2267,8 +2267,15 @@ class Distribution(object):
                 continue
 
             fn = getattr(sys.modules[modname], '__file__', None)
-            if fn and normalize_path(fn).startswith(loc):
+            if fn and (fn.startswith(loc) or normalize_path(fn).startswith(loc)):
                 continue
+            pieces = fn.split(os.path.sep)
+            if (len(pieces) > 2
+                and pieces[-1] in ("__init__.py", "__init__.pyc")
+                and pieces[-2] == modname):
+                fn2 = os.path.sep.join(pieces[:-2])
+                if normalize_path(fn2).startswith(loc):
+                    continue
             issue_warning(
                 "Module %s was already imported from %s, but %s is being added"
                 " to sys.path" % (modname, fn, self.location),
msg350 (view) Author: warner Date: 2009-08-25.22:09:35
wonlay: thanks for the great hint! I've been seeing a similar problem at
home, on my OS-X box, and thanks to your suggestion I was able to figure out
the problem. I don't yet know the solution, though :).

My general policy is to only ever to 'sudo install' when the install target
is to a stow-managed per-package-per-version directory. So in this case, when
I installed the "foolscap" package, I put it in
/usr/local/stow/foolscap-0.4.2/ , and then told stow to symlink it into place
at /usr/local/lib/python2.5/site-packages .

I then see this warning any time anything imports foolscap.

When pkg_resources.Distribution.check_version_conflict() is asked (for some
reason, I don't know why) to check on the 'foolscap' package, it sees that
the __file__ attribute points to:

 /usr/local/lib/python2.5/site-packages/foolscap/__init__.pyc

When it does a normalize_path() on that, it is told that __init__.pyc lives
in:

 /usr/local/stow/foolscap-0.4.2/lib/python2.5/site-packages/foolscap/__init__.pyc

and indeed that is not on sys.path, so it emits the warning.

I guess what I want it to do is to figure out where the module really got
imported from first: chop off the module name and the __init__.pyc to get
"/usr/local/lib/python2.5/site-packages", then do a normalize_path() on that,
and *then* compare it against 'loc'. I don't know how to do that offhand..
will there always be an __init__.pyc file?

cheers,
 -Brian
msg326 (view) Author: zgoda Date: 2009-07-15.08:43:30
In my case importing any systemwide package installed causes this warning:

jarek:~/dev/cruncher$ PYTHONPATH=$HOME/dev/cruncher/cruncher/lib python
>>> import cruncher
>>> import pkg_resources
__main__:1: UserWarning: Module cruncher was already imported from 
cruncher/__init__.pyc, but /Users/jarek/dev/cruncher/cruncher/lib is being added 
to sys.path
>>> 
jarek:~/dev/cruncher$ PYTHONPATH=$HOME/dev/cruncher/cruncher/lib python
>>> import cruncher
>>> import psycopg2
/opt/local/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-
packages/psycopg2-2.0.11-py2.5-macosx-10.5-i386.egg/psycopg2/_psycopg.py:3: 
UserWarning: Module cruncher was already imported from cruncher/__init__.pyc, 
but /Users/jarek/dev/cruncher/cruncher/lib is being added to sys.path
  import sys, pkg_resources, imp
>>>
msg231 (view) Author: quentinsf Date: 2009-01-26.07:43:45
Yes, wonlay's fix (around line 2270) works fine for me.

On my Ubuntu machine, a quick way to show the bug is:

>>> import MySQLdb
>>> import pkg_resources

__main__:1: UserWarning: Module _mysql was already imported from 
/var/lib/python-support/python2.5/_mysql.so, but /var/lib/python-
support/python2.5 is being added to sys.path
msg230 (view) Author: zooko Date: 2009-01-24.04:23:48
Okay, I've learned that one of the instances of this warning was a bug in
nevow's setup.py that was declaring that nevow came with a package named
twisted.plugins:

http://divmod.org/trac/ticket/2830

It sounds like wonlay is onto a bigger issue though.
msg215 (view) Author: wonlay Date: 2008-12-05.12:53:40
Sorry missed some words last submit. It's:

After debuging, I found the warning message is from function
check_version_conflict in pkg_resources.py
msg214 (view) Author: wonlay Date: 2008-12-05.12:48:35
I'm facing this issue now. In my case, the module showed in already import
warning is _mysql.
You can google for:
http://www.google.com/search?hl=en&newwindow=1&safe=off&rlz=1B3GGGL_en___CN232&q=UserWarning%3A+Module+was+already+imported+from+but+%2Fvar%2Flib%2Fpython-support%2Fpython2.5+is+being+added+to+sys.path&btnG=Search

Lots of people have the same problem, and several bug have been reported to
several projects, but it's hard to reproduce, because this is not there issue,
it's issue of pkg_resources in setuptools I think.

I don't know how to reproduce this bug simply, but you can reproduce it follow
these steps (I see this bug when I'm using Django and Protobuf together):

(I'm using debian lenny, python 2.5)
1. Install mysql and its python binding MySQLdb;
2. Install django use mysql as your db backend(http://www.djangoproject.com/);
3. Install protobuf (Both the library and python bindings inside the download
package.)(http://code.google.com/apis/protocolbuffers/);
4. Start a your django project, start a new app, add your app models;
5. Create a simple .proto protobuf file and compile it into .py;
6. Import the compiled model in the end of your django app's models.py;
7. Run ./manage.py shell;

You will see the warning message:
/usr/lib/python2.5/site-packages/protobuf-2.0.2-py2.5.egg/google/__init__.py:1:
UserWarning: Module _mysql was already imported from
/var/lib/python-support/python2.5/_mysql.so, but
/var/lib/python-support/python2.5 is being added to sys.path

After debuging, I found the warning message is from function 

when modname is _mysql, after:
fn = getattr(sys.modules[modname], '__file__', None)

fn will be '/var/lib/python-support/python2.5/_mysql.so'.

But normalize_path(fn) is
'/usr/lib/python-support/python-mysqldb/python2.5/_mysql.so'.

So normalize_path(fn).startswith(loc) failed(loc is
'/var/lib/python-support/python2.5').

Actually, '/var/lib/python-support/python2.5/_mysql.so' is a symbolic link of
'/usr/lib/python-support/python-mysqldb/python2.5/_mysql.so'.

My workaround is first test fn.startswith(loc) and then
normalize_path(fn).startswith(loc).

And the patch attached.
msg158 (view) Author: zooko Date: 2008-09-05.14:02:15
There is a discussion about this issue on the tahoe-dev list:

http://allmydata.org/pipermail/tahoe-dev/2008-September/000785.html
msg145 (view) Author: zooko Date: 2008-08-27.18:53:51
So it seems increasingly likely that this is yet another problem caused by Nevow
importing "nevow" and "twisted" when its setup.py is executed: please see issue20
msg142 (view) Author: pje Date: 2008-08-27.01:33:08
These steps do not reproduce the problem.  I built the Nevow egg using
"easy_install Twisted Nevow".  The subdirectory I used was 't' rather than
'FOO', but I don't think that's relevant.  For that matter, I don't see how the
given steps could possibly produce the error described, unless zope.interface
was doing a require('Nevow'), and importing twisted caused that to happen.
msg140 (view) Author: zooko Date: 2008-08-27.00:36:16
Sorry about that -- I didn't examine the log that I was posting to realize that
it didn't echo the actual Python command-line.
msg138 (view) Author: warner Date: 2008-08-27.00:26:04
Sorry about that, I should have investigated this problem further before raising
it to zooko.

To reproduce this problem independently:
 1: create Nevow-0.9.31-py2.5.egg
 2: put it in a child directory FOO
 3: run PYTHONPATH=FOO python -c 'import twisted'

I haven't yet been able to figure out how to accomplish step 1 outside of our
Tahoe build system. We do it with a "setup.py develop" that sees a dependency on
Nevow that needs satisfying. Apparently there is something broken in the Nevow
setup.py (or rather their setup_egg.py), as a regular "bdist_egg" doesn't seem
to work.
msg137 (view) Author: pje Date: 2008-08-27.00:03:23
Please stop submitting "make" logs as examples; they are useless to me and
anyone else who wants to work on the bug and doesn't have your makefiles,
doesn't run on the same platform, etc.  Thanks.
msg136 (view) Author: zooko Date: 2008-08-26.23:55:47
/usr/lib/python2.5/site-packages/zope/__init__.py:19: UserWarning: Module
twisted was already imported from
/usr/lib/python2.5/site-packages/twisted/__init__.pyc, but
/home/buildslave/tahoe/feisty2.5/build/support/lib/python2.5/site-packages/Nevow-0.9.31-py2.5.egg
is being added to sys.path

Here is an example command line that yields this warning:

http://allmydata.org/buildbot/builders/feisty2.5/builds/1649/steps/pyflakes/logs/stdio

Note that this may be related to issue17 and issue20 -- why is the Nevow egg
declaring the "twisted" namespace and/or module anyway?
History
Date User Action Args
2009-10-19 19:36:46pjesetstatus: in-progress -> resolved
messages: + msg428
2009-10-10 23:18:10pjesetmessages: + msg381
2009-08-25 22:18:33warnersetmessages: + msg351
2009-08-25 22:09:36warnersetmessages: + msg350
2009-07-15 08:43:30zgodasetnosy: + zgoda
messages: + msg326
2009-03-30 09:33:05wdoekessetnosy: + wdoekes
2009-01-26 07:43:45quentinsfsetnosy: + quentinsf
messages: + msg231
2009-01-24 04:23:49zookosetmessages: + msg230
2008-12-05 12:53:40wonlaysetstatus: chatting -> in-progress
messages: + msg215
2008-12-05 12:48:35wonlaysetfiles: + module_already_imported.diff
nosy: + wonlay
messages: + msg214
2008-09-05 14:02:15zookosetmessages: + msg158
2008-08-27 18:53:51zookosetmessages: + msg145
2008-08-27 01:33:08pjesetmessages: + msg142
2008-08-27 00:36:16zookosetmessages: + msg140
2008-08-27 00:26:04warnersetnosy: + warner
messages: + msg138
2008-08-27 00:03:23pjesetstatus: unread -> chatting
nosy: + pje
messages: + msg137
2008-08-26 23:55:47zookocreate