The bug tracker for setuptools 0.7 or higher is on BitBucket

 

Issue139

Title pkg_resources.require can't deal with multiple installed versions if one is present in easy-install.pth
Priority feature Status resolved
Superseder Nosy List mbogosian, pje
Assigned To Keywords

Created on 2012-06-25.07:53:53 by mbogosian, last changed 2013-07-07.18:06:07 by pje.

Messages
msg731 (view) Author: pje Date: 2013-07-07.18:06:02
pkg_resources.require() is not intended for this use case.  If you want to have multiple versions of a package available for import by different scripts, then either there must be no default package, or else those scripts must be generated via a setup.py, buildout, or other tool.

If users can't be bothered to write a setup.py, then they should install their desired default version in the appropriate home subdirectories or via PYTHONPATH or a virtualenv.  Otherwise, the package should not be installed as a default at the system level.  (This can be avoided by using the -m option to easy_install when installing package(s).)

In other words, you can safely install multiple versions of a package as long as none of them is the default version, and then they can be picked up via require().

In short, your options are:

1. Use setup.py to manage dependencies properly, and/or

2. Don't install a default version of a package; use -m to ensure that none of the installed versions is the default, so that require() can be used. (Note that it's not necessary to have a default version of a package unless an operating system level package demands it.)

3. Install user-specific defaults via user-specific site-packages, PYTHONPATH, virtualenv, installation alongside the script, or direct sys.path manipulation prior to importing pkg_resources.

Option 1 works whether or not you do option 2.  If you have users who don't want to use option 1, and you can't do #2 because of OS-level requirements, they will have to fallback to #3 to override the system-level default.
msg730 (view) Author: mbogosian Date: 2013-07-07.07:26:30
Thank you for your patience and thorough explanation of things. I am sorry for not being very clear.

Is there a userland (i.e., not packaging) equivalent for what you described? For example in a multiuser system where Admin wants to support User A, User B, and User C. User A writes usera.py which necessarily depends on Lib==1.4. User B writes userb.py which necessarily depends on Lib==1.5. User C writes userc.py, in which he does a standard "import Lib". User C doesn't care which version he gets, so long as he can use standard import statements. None of the Users are prepared to write setup.py files or package their scripts. They all just want to write and run .py files in their own home directories.

Can Admin accommodate all three users on the same system? If he can, then I apologize for my ignorance, as I was under the misconception that pkg_resources.require was intended to be used as such a mechanism. I did not mean to waste anyone's time. :o(
msg728 (view) Author: pje Date: 2013-07-06.20:07:52
> systems could not install both versions while also supporting a default for script authors who were less picky

This is simply not true.  Scripts which are part of an installed package can depend on any installed version of another package, whether it is the default or not.

To do this, they need only do as I described: be published from a project that declares its dependencies.  virtualenvs are unnecessary.

The default version of a package only affects scripts whose dependencies are *compatible* with the default, and scripts which are not exported by a project with declared dependencies.  A script with declared dependencies incompatible with the default will end up with a non-default version.

But these have to be *declared* static dependencies, not dynamic dependencies requested at runtime.

If you create two projects with dependencies on specific versions of BeautifulSoup, each containing a script, and install them, they will each run with their specific version, regardless of which version is the default.

In other words, if I understand your use case correctly, there was never any reason to be dynamically calling require() in the first place.  Declare your dependencies in setup.py instead of in the script, and the problem goes away.
msg727 (view) Author: mbogosian Date: 2013-07-06.17:28:22
I agree with what you've described, but I think it assumes a simplistic world view. Here's a real world example that might better illustrate the problem:

I frequently use BeautifulSoup (see <http://www.crummy.com/software/BeautifulSoup/>). One great thing about this library is that it (usually) gracefully parses handles non-compliant HTML (much like browsers do).

A couple years ago, the author changed internals between the 3.0.8 and 3.1.0 releases, thereby making the library much more fragile, and much less capable of consuming poorly formed HTML (see <http://www.crummy.com/software/BeautifulSoup/bs3/3.1-problems.html>).

Systems whose package engines dutifully honored the 3.0.8 => 3.1.x upgrade introduced this fragility systemwide. It was possible to have system scripts which could (and wanted) to stay with 3.0.8 alongside of those which needed some feature X introduced in 3.1.x (but which didn't care about parsing bad HTML). Manually requiring a version of an external dependency over which one has no control may be painful, but it's sometimes a fact of life. I'm not sure this is Doing It Wrong for those applications who needed to stay with 3.0.8 but who didn't want to hold up the train for everyone else.

Because of the problems outlined in previous comments, systems could not install both versions while also supporting a default for script authors who were less picky.

BeautifulSoup is not the first library to be used across multiple applications in the same install base where each application may require a specific (sometimes conflicting) version. By way of analogy, many linux systems allow for multiple versions of popular software (like python) to be installed concurrently. However, while systems vary, many have (cleverly) also allowed for the ability to select a default (see, e.g., <http://linux.die.net/man/8/update-alternatives>). This means that typing "python" still works, and so does "python2.7" and "python3.3". If an administrator wants to change what "python" points to, she can. Library authors also do this all the time. (Look at all the symlinks in /usr/lib for example.)

I think tools like virtualenv can be used as a workaround (with some side effects), and I accept that it may be architecturally difficult to implement in setuptools, but I still think it's a valid feature request. If setuptools is not the right place to support it, so be it. I am ignorant of where the limitation is imposed. Perhaps distutils?

Just by $1.05. Now I'll submit. ;o)
msg726 (view) Author: pje Date: 2013-07-06.13:49:30
This won't change in setuptools; the new packaging PEPs have different ways of specifying and managing dependencies and there isn't really a way to do this in the current system.

Note that if you are manually requiring your dependencies, you are probably Doing It Wrong in any event.  The One Obvious Way is to specify all your dependencies in your project's requirements.  Dynamic requirements are for special cases; 99%+ of the time you should let the installer (or "setup.py develop" or "setup.py test") take care of generating your scripts' __require__ lines and allow pkg_resources to manage the conflicts.

tl;dr: __require__ only works for one project because it's supposed to be *your* project, which then declares its own dependencies.  It's not supposed to be used to reference some *other* project.  Add a setup.py to your project and use "setup.py develop" to install it for development.
msg725 (view) Author: mbogosian Date: 2013-07-06.06:57:21
So I guess that's the bug (er, feature?): one should be able to perform a failover or fallback if there's a default version, but currently one can't (see first comment).
msg662 (view) Author: pje Date: 2012-06-27.17:16:17
You can't do failover or fallback if there's a default version.
msg661 (view) Author: mbogosian Date: 2012-06-25.17:39:02
So failover/fallback would work like this?

- - - - - - - - %< - - - - - - - -

easy_install BeautifulSoup==3.0.8
# ...
easy_install -U BeautifulSoup # <- upgrades to >= 3.1.0
# ...
python <<EOF
try:
    __require__ = ( 'BeautifulSoup==3.0.8', )
    import pkg_resources
# Note: we can't catch pkg_resources.DistributionNotFound explicitly
# because we can't yet import pkg_resources; so we have to catch the
# generic Exception instead
except:
    __require__ = ( 'BeautifulSoup>=3.1.0', )
    import pkg_resources
EOF

- - - - - - - - >% - - - - - - - -

Correct me if I'm wrong, but the workaround in msg660 only works for *one* package. In other words, there's no workaround for something like this:

- - - - - - - - %< - - - - - - - -

easy_install mymodule==3.0
# ...
easy_install mymodule==2.0 # <- mymodule 2.0 now globally active version
# ...
easy_install hermodule==5.0
# ...
easy_install hermodule==4.0 # <- hermodule 4.0 now globally active version
# ...
python <<EOF
import pkg_resources

try:
    pkg_resources.require('mymodule==3.0')
except pkg_resources.DistributionNotFound:
    # Fallback to mymodule 2.0 if 3.0 is unavailable
    pkg_resources.require('mymodule==2.0')
    displayMyModuleMissingFeaturesWarning()

try:
    pkg_resources.require('hermodule==5.0')
except pkg_resources.DistributionNotFound:
    # Fallback to hermodule 4.0 if 5.0 is unavailable
    pkg_resources.require('hermodule==4.0')
    displayHerModuleMissingFeaturesWarning()
EOF

- - - - - - - - >% - - - - - - - -
msg660 (view) Author: pje Date: 2012-06-25.16:34:24
On Mon, Jun 25, 2012 at 3:53 AM, mbogosian <setuptools@bugs.python.org>wrote:

>
> Perhaps I have misunderstood the intended behavior, but I thought one
> should be able to do this:
>
> % easy_install BeautifulSoup==3.0.8
> ...
> % easy_install -U BeautifulSoup
> [installs 3.1.0.1]
> ...
> % python -c 'import pkg_resources ; print
> pkg_resources.require("BeautifulSoup==3.0.8")'
> Traceback (most recent call last):
> pkg_resources.VersionConflict: (BeautifulSoup 3.1.0.1
> (.../site-packages/BeautifulSoup-3.1.0.1-py2.5.egg),
> Requirement.parse('BeautifulSoup==3.0.8'))
>

Do this instead:

python -c '__requires__=['BeautifulSoup==3.0.8']; import pkg_resources;
import beautifulsoup'

That is, if the main script declares a __requires__ variable before
importing pkg_resources, pkg_resources will allow you to override what
would normally produce a VersionConflict due to a default version being
present on the command line.

This is an undocumented but supported workaround for the issue, as it's
used internally by the scripts easy_install generates.
msg659 (view) Author: mbogosian Date: 2012-06-25.07:53:53
Perhaps I have misunderstood the intended behavior, but I thought one should be able to do this:

- - - - - - - - %< - - - - - - - -

% easy_install BeautifulSoup==3.0.8
...
% easy_install -U BeautifulSoup
[installs 3.1.0.1]
...
% python -c 'import pkg_resources ; print pkg_resources.require("BeautifulSoup==3.0.8")'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../site-packages/distribute-0.6.10-py2.5.egg/pkg_resources.py", line 648, in require
    needed = self.resolve(parse_requirements(requirements))
  File ".../site-packages/distribute-0.6.10-py2.5.egg/pkg_resources.py", line 550, in resolve
    raise VersionConflict(dist,req) # XXX put more info here
pkg_resources.VersionConflict: (BeautifulSoup 3.1.0.1 (.../site-packages/BeautifulSoup-3.1.0.1-py2.5.egg), Requirement.parse('BeautifulSoup==3.0.8'))

- - - - - - - - >% - - - - - - - -

If I use the '-m' tag (to keep it out of easy_install.pth), I have a different problem:

- - - - - - - - %< - - - - - - - -

% easy_install -m BeautifulSoup==3.0.8
...
% easy_install -U -m BeautifulSoup
[installs 3.1.0.1]
...
% python -c 'import pkg_resources ; print pkg_resources.require("BeautifulSoup==3.0.8")'
[BeautifulSoup 3.0.8 (.../site-packages/BeautifulSoup-3.0.8-py2.5.egg)]
% python -c 'import BeautifulSoup'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named BeautifulSoup

- - - - - - - - >% - - - - - - - -

Can I not have my cake and eat it too? Meaning, can I not have multiple installed versions of a package with a default (i.e., an entry in easy-install.pth), but still be able to explicitly specify a previously installed version?
History
Date User Action Args
2013-07-07 18:06:07pjesetstatus: chatting -> resolved
2013-07-07 18:06:02pjesetmessages: + msg731
2013-07-07 07:26:30mbogosiansetstatus: resolved -> chatting
messages: + msg730
2013-07-06 20:07:52pjesetstatus: chatting -> resolved
messages: + msg728
2013-07-06 17:28:22mbogosiansetstatus: resolved -> chatting
messages: + msg727
2013-07-06 13:49:30pjesetstatus: chatting -> resolved
messages: + msg726
2013-07-06 06:57:21mbogosiansetpriority: bug -> feature
messages: + msg725
2012-06-27 17:16:18pjesetmessages: + msg662
2012-06-25 17:39:03mbogosiansetstatus: resolved -> chatting
messages: + msg661
2012-06-25 16:35:23pjesetstatus: chatting -> resolved
2012-06-25 16:35:10pjesetfiles: - unnamed
2012-06-25 16:34:24pjesetfiles: + unnamed
status: unread -> chatting
messages: + msg660
nosy: + pje
2012-06-25 07:53:53mbogosiancreate