Python needs a .../python2.x.y/vendor-packages directory
for vendor supplied Python files.

See:

http://mail.python.org/pipermail/python-list/2005-September/300029.html

for the full reasoning behind this request.

I also approached Guido w.r.t. this. Here's his reply.

Subject: Re: Python vendor-packages directory in a
future Python release?
Date: Tue, 20 Sep 2005 19:48:40 -0700
From: Guido van Rossum <guido at python.org>
Reply-To: Guido van Rossum <guido at python.org>
To: Rich Burridge <Rich.Burridge at Sun.COM>
References: <4330C108.4030100@sun.com>

I think that's a reasonable request. (In the mean time,
I think that
using site-packages is fine as an interim solution.)

I suggest that you use the SourceForge patch manager
for the Python
project to upload your patch, and then post to
python-dev. You may be
asked to review 5 other patches in order to have
someone look at your
favorite patch.

--Guido

Logged In: YES 
user_id=593130

The reason for this patch given in the referred-to post is:

"We have been told that this directory is inappropriate for vendor 
supplied packages, just as "site_perl" is inappropriate for Perl. 
With Perl, vendor supplied packages go under "vendor_perl". "

where 'this directory' is site-packages, which works fine.

The python-dev thread subequent to this posting starts with
http://mail.python.org/pipermail/python-dev/2005-
September/056682.html
A subsequent post
http://mail.python.org/pipermail/python-dev/2005-
September/056696.html
clarifies that 'vendor supplied packages' here means packages 
installed by the system/OS vendor.

Disconnected (in the pipermail archives) pieces of the thread 
start here
http://mail.python.org/pipermail/python-dev/2005-
September/056697.html
and here
http://mail.python.org/pipermail/python-dev/2005-
September/056702.html

This last suggests that this proposal is on hold while a .pth 
solution is explored.

Logged In: YES 
user_id=511506

A good alternative solution to this problem was given on the
python-devel
mailing list. See:

 
http://mail.python.org/pipermail/python-dev/2005-September/056697.html

 
http://mail.python.org/pipermail/python-dev/2005-September/056699.html

The architectural commitee have approved this solution, so
I'm closing
this bug as "Invalid". If there'd been a "Withdrawn"
resolution, I'd have
closed it that way instead. Perhaps that's what Deleted is
supposed to
do. Feel free to tweak if I've selected the wrong closure.

Guido van Rossum suggests a vendor-packages directory in

http://mail.python.org/pipermail/python-dev/2006-November/070063.html

I'm reopening the patch to encourage further review.

Should be considered for 3.1 and 2.7.

"Further review" never happened. Should be close this?

So this has come up again within the Solaris group.  Since Rich's original request, we've been using a vendor-packages.pth file in the site-packages directory, which enables the vendor-packages directory.  However, I have a concern that this would completely disappear when Py_NoSiteFlag is enabled.  I would like to run the interpreter with -SE on all system-provided scripts, so that modules installed by the user in a private PYTHONPATH or in site-packages via easy_install (or similar) don't inadvertently interpose on system-provided modules which are the ones we've tested against.

I'm attaching a patch (against 2.6.4; sorry, haven't looked at anything newer yet) that modifies pythonrun.c to add initvendor() as a parallel with initmain() and initsite(), and is always run, regardless of Py_NoSiteFlag, but inserts vendor-packages after site-packages for when the user or a script is okay with the potential interposition.

If support for vendor-packages is being considered in general, I'd like to address the request for a rationale that has been brought up a handful of times before.  Specifically, distro vendors (such as ourselves) want to be able to deliver non-core python modules -- some written ourselves, some available from the community at large.  If we do that in the canonical site-packages directory, then those modules are at risk of being overwritten by local administrators installing python packages as they normally would.  This means that system programs -- including critical ones on Solaris, such as the packaging system -- might suddenly stop working.  If we separate the vendor space from the end-user space, then local admins can do anything they want without fear of breaking the system.  And if they really want to replace the vendor-supplied modules, then they can either go to the lengths of making the modules install in vendor-packages, or build using the same mechanisms we do (which for almost all the python we ship are still open-source) and install the resulting (system, not python) package.

I'd love feedback on my patch, regardless.

Thanks for detailing the rationale.  Separating the “vendor”-controlled space (or let’s call it system; not all OSes have vendors :) from the sysadmin-controlled location is indeed a real concern, as shown by the recent-ish dist-packages invention by doko for Debian.  Maybe you could get more feedback from other Python packagers on the distutils-sig, python-dev or distributions@freedesktop.org mailing list?

> not all OSes have vendors
I realized that because French is my native language, I interpret “vendor” as “seller” (the meaning of “vendeur”), but I think that in English it just means “distributor” and has no relation with selling.  vendor-packages is arguably better that system-packages, as system can actually mean a lot of things.

In English also, to vend is to sell, direct from Fr. *vendre*. However, ideas are not necessarily sold for money, but for assent, adoption, or other action. My dictionary has 'publish' as a secondary meaning of vend. So 'vendor-packages' is fine within the common metaphorical meaning of selling ideas. And it rolls off the tongue better than 'publisher-packages' or 'distributor-packages'.

I'm strongly +1 on this one. I package Python RPMs for Fedora and I know what mess can come out of installing through both RPM and easy_install/pip.
In Fedora, both Perl and Ruby use vendor specific dirs for installing RPM packaged modules and site specific dirs for installation via cpan/rubygems. Since I'm also Ruby maintainer, I was able to watch how the confusion of Ruby developers disappeared after we introduced this approach in Fedora 17 (and rubygems stopped uninstalling RPM gems, which was always causing unsatisfied dependency problems). So I think Python should also go this way and provide a configuration option to use vendor-packages. Please note, that Debian packagers have been patching their Python package downstream for quite some time, introducing "dist-packages" into their installation. I think that they didn't experience any significant problems so far, which pretty much proves that this is feasible (and also their patches could probably serve as a base for patch for cPython).

Adding Nick to this discussion, since distutils-sig is talking about these issues right now.

Previous discussions:
http://mail.python.org/pipermail/python-list/2005-September/345116.html
http://mail.python.org/pipermail/python-dev/2005-September/056682.html
http://mail.python.org/pipermail/python-dev/2003-August/037487.html

Barry: if you could summarize how and why doko introduced dist-packages in Debian, or let me copy parts of email we exchanged a couple years ago, I think it could help.  Nick seems inclined to standardize doko’s solution, but IIRC it was an imperfect but working compromise, so we might want to rethink it.

The discussion on distutils-sig that Éric is referring to is one were I ended up pointing out that PEP 439 (bootstrapping pip in 3.4) needs to address this in some manner, so that pip (which installs to site-packages) doesn't end up fighting with system package managers that are also installing to site-packages in many cases.

Since the distro vendors are in a better situation to change their target installation directory, the consensus on the list was that something *like* the current Debian solution is most appropriate (I wasn't aware this issue existed at the time).

The current situation is annoying-but-tolerable with pip as a third party solution, but unacceptable once the command is being provided by Python itself. As with virtual environments, something with upstream support may be able to be cleaner than a third party workaround.

Relevant distutils-sig post:http://mail.python.org/pipermail/distutils-sig/2013-May/020673.html (ignore the initial quoted part - I was just flat out wrong earlier in that thread)

there is more than one thing addressed with the 'dist-packages' choice of name.

The primary reason is to have a directory where you only find python packages as distributed by the linux distribution, and where installers do not install to by default.  Even if this directory is non-writable, people did call 'sudo python setup.py install', and then did report issues on the Ubuntu tracker caused by these installs. Such a vendor place should never be the default to be installed to by default.

There is a Debian policy to support /usr/local, and that's the reason you find a second directory /usr/local/lib/pythonx.y/dist-packages. A 'sudo python setup.py install' installs into this location, distribution maintainers providing Debian packages are supposed to call setup.py install --install-layout=deb to install into the debian system installation.

At this time Barry still new to distro policies, was surprised to find /usr/local/lib/pythonx.y/site-packages being used by the system python, which is also used by a local python build which is configured without any --prefix parameters. So the system python now uses dist-packages in both /usr and /usr/local to not interfere with a local python installation.

Note that for python3, Debian and Ubuntu are trying to share dist-packages across python3 versions to ease upgrades from one version to the other, and trying to support more than one version during the upgrade (calling that /usr/lib/python3/dist-packages).

There is ongoing discussion on pip's github tracker [1] about default location where to install user modules. 

IMO this is something that should be dealt with in Python Interpreter Core [2][3]. I would like to hear some opinion from python devs on this.




[1] https://github.com/pypa/pip/issues/1668
[2] https://github.com/pypa/pip/issues/1668#issuecomment-38418185
[3] https://github.com/pypa/pip/issues/1668#issuecomment-38428857

Note that authority over package design decisions has been delegated to the Python Packaging Authority - it's up to them to decide what they want and ask for it (of they decide they actually need modifications to the interpreter), not for Python core to tell them what to do.

Ok, I have started a thread at pypa-devs google group.
https://groups.google.com/forum/#!topic/pypa-dev/r6qsAmJl9t0