classification
Title: pyvenv 3.3 fails to create symlinks for /local/{bin, lib} to /{bin, lib}
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.4, Python 3.3
process
Status: closed Resolution: works for me
Dependencies: Superseder:
Assigned To: Nosy List: Marco.Amadori, asvetlov, barry, carljm, doko, marten, merwok, ned.deily, vinay.sajip
Priority: normal Keywords:

Created on 2012-11-15 16:55 by Marco.Amadori, last changed 2014-03-12 23:27 by merwok. This issue is now closed.

Messages (19)
msg175626 - (view) Author: Marco Amadori (Marco.Amadori) Date: 2012-11-15 16:55
The solution to work around the bug is there:

http://ipoveraviancarriers.blogspot.it/2012/11/python-33-and-pyvenv-hackish-solution.html

But probably pyvenv should be patched in order to create symlinks.

Thanks!
msg175724 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2012-11-17 02:55
What problem are you trying to solve here?  pyvenv is working as documented.  And, although pyvenv does not claim to be compatible with the third-party virtualenv package, as far as I know virtualenv does not create links for local/bin and friends either.
msg175748 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2012-11-17 14:59
> as far as I know virtualenv does not create links for local/bin and friends either.

AFAIK it never used to, but it seems to do so since 1.6.3, in an attempt to fix a problem with the "posix_local" install scheme. I just tested with virtualenv 1.6.4, and when I create a venv in /tmp/venv, it creates a symbolic link /tmp/venv/local which points back to /tmp/venv. This behaviour causes problems with e.g. PyCharm, which shows ever-nesting directories ("turtles all the way down"), has led to later versions of virtualenv creating an actual directory and symlinking the venv's contents to entries in it.

The question is: is this still necessary? Why exactly does "posix_local" need this, and does it need it for Python 3.3, 3.4 etc? Adding Carl to nosy, hoping he can shed some light on this.
msg175749 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2012-11-17 15:00
Whoops, mouse glitch led to components being wrongly set.
msg175924 - (view) Author: Marco Amadori (Marco.Amadori) Date: 2012-11-18 21:25
> The question is: is this still necessary? Why exactly does
> "posix_local" need this, and does it need it for Python 3.3, 3.4 etc? 
> Adding Carl to nosy, hoping he can shed some light on this.

I hope to see the light on that too. ;-)

Meanwhile, my problem was that after creating the virtualenv, as cited in the blog post above linked, I got stuck in installing pip (and thus distribute) in that virtualenv, because 'VENV/local/bin' is not in the $PATH (and it seems reasonable for a virtualenv that this will remain stable).

So, (IMHO) if changing the $PATH is not a good idea, either python should do:

1. patch distribute and pip to do not install in ./local by default if in a VENV. (enviroment variable based?, whatever)

2. create the links after creating the VENV (my naïve solution)

Am I right or I completely hit the wrong issue?
msg176037 - (view) Author: Carl Meyer (carljm) * Date: 2012-11-20 23:21
Here is the bug filed against virtualenv that led to the addition of the local/ directory: https://github.com/pypa/virtualenv/issues/118

As Vinay pointed out, the original fix was later modified to be friendlier to tools that dislike recursive symlinks.

That's about all I know; I don't know if any of this is still needed in 3.3/3.4.
msg176046 - (view) Author: Marco Amadori (Marco.Amadori) Date: 2012-11-21 08:31
2012/11/21 Carl Meyer <report@bugs.python.org>

>
> Carl Meyer added the comment:
>
> Here is the bug filed against virtualenv that led to the addition of the
> local/ directory: https://github.com/pypa/virtualenv/issues/118
>
> As Vinay pointed out, the original fix was later modified to be friendlier
> to tools that dislike recursive symlinks.
>

I didn't used virtualenv but pyvenv-3.3, so if this is not a bug, my
question become:

how to proper install distribute and pip in python 3.3 venv since they
installs in <VENV>/local/bin and that <VENV>/bin/activate does not
include <VENV>/local/bin in $PATH ?
msg176074 - (view) Author: Carl Meyer (carljm) * Date: 2012-11-21 19:04
What OS are you on, Marco?

It looks to me like pyvenv probably does need the same hack as virtualenv here, to deal with OSes who set posix_local as the default installation scheme.
msg176078 - (view) Author: Marco Amadori (Marco.Amadori) Date: 2012-11-21 20:34
On Wednesday 21 November 2012 21:32:57 Carl Meyer wrote:
> Carl Meyer added the comment:
> 
> What OS are you on, Marco?

Linux. Debian GNU/Linux amd64 sid/experimental.

> It looks to me like pyvenv probably does need the same hack as virtualenv
> here, to deal with OSes who set posix_local as the default installation
> scheme.

nice!
msg176098 - (view) Author: Éric Araujo (merwok) * (Python committer) Date: 2012-11-22 05:21
Is it impossible to edit the install scheme when run in a venv, so that the workaround becomes unnecessary?  We are the upstream after all :)
msg176224 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2012-11-23 19:44
IIUC, Debian Experimental is not a complete distribution. Are there other, non-experimental OS variants where posix_local is used with Python 3.x?

On my Ubuntu Oneiric and Precise 64-bit machines, the default install scheme on the official Python 3.2 - as determined by sysconfig._get_default_scheme() - is "posix_prefix".
msg177837 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2012-12-20 17:00
having the posix_prefix as the default in Debian is an oversight on my side. it always should be posix_local. I'll fix this at least for 3.3 in current development releases.

The rationale for this is that distutils based installs install by default into the path which is used by the distributor and get in conflict with the packages distributed by Debian/Ubuntu.

At one of the language summits in Chicago I asked that the default installation target should be changed to the user dir, but this idea was rejected.

Therefore you do have two site directories, called

  /usr/lib/pythonX.Y/dist-packages
  /usr/local/lib/pythonX.Y/dist-packages

The renaming from site to dist was done to not conflict with a default python installation (without any configure parameters).

The default install should go to /usr/local, packagers for Debian usually should use the Debian local distutils option --install-layout=deb.
msg177839 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2012-12-20 17:45
Even when "posix_local" becomes the default scheme on Debian python builds, it's not clear whether the suggested fix is the correct one to make, unless "posix_local" is added to sysconfig's list of _INSTALL_SCHEMES. Matthias comment makes sense when considering system-wide installs, but when installing into a venv, these considerations don't seem to apply.

Technically the change is not hard to do, but I'm a little uncomfortable whether it's the right thing to do, as it's only for a specific Linux distribution, and it seems a bit of a kludge.

Although the ideal situation is for software not to care whether it's running in a venv, I'm not sure whether installers like pip/distribute need to be completely ignorant of whether they're installing in a venv.

For example, the posix_local scheme says that the site-packages directory is to be called 'dist-packages', which seems unnecessary in a venv since its purpose is really for system-wide installations. Yet, virtualenv names the directory 'site-packages', so it appears not to be following the scheme faithfully as far as that part of it is concerned.
msg177849 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2012-12-20 18:31
virtualenv for 2.7, when used with a Debian/Ubuntu system installation, should only see site-packages.  Same thing should be done for venv for 3.3 and up. Even if needed patches will not go upstream, they are welcome.
msg177875 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2012-12-21 10:45
fixed in experimental, pyvenv's are now handled the same way as python-virtualenvs. closing the issue here.
msg186029 - (view) Author: Marten Lehmann (marten) Date: 2013-04-04 12:06
Could you please tell me, in which exact release or commit this is resolved?

I'm using Python 3.3.0 on Ubuntu Precise and I noticed like the others, that e.g. easy_install and pip are installed to venv/local/bin instead of venv/bin and the site-packages folder being used is venv/local/lib/python3.3/dist-packages instead of venv/lib/python3.3/site-packages.

Are both issues solved in experimental? So far I only saw a symlink-patch for virtualenv but that doesn't actually fix the problem of being double-local (venv and local-folder within venv), neither dist-packages:

https://github.com/pypa/virtualenv/commit/285679cfd326c918676e765e06ed142db66efde0
msg186038 - (view) Author: Éric Araujo (merwok) * (Python committer) Date: 2013-04-04 14:53
From Python’s viewpoint this is closed, please follow up on the Debian bug tracker.
msg186094 - (view) Author: Marten Lehmann (marten) Date: 2013-04-05 17:29
Which bug ID? Couldn't find it anywhere in the previous comments.
msg213329 - (view) Author: Éric Araujo (merwok) * (Python committer) Date: 2014-03-12 23:27
It was linked from the blog post (see first message): https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=695758
History
Date User Action Args
2014-03-12 23:27:59merwoksetmessages: + msg213329
stage: resolved
2013-04-05 17:29:30martensetmessages: + msg186094
2013-04-04 14:53:47merwoksetmessages: + msg186038
2013-04-04 12:06:36martensetnosy: + marten
messages: + msg186029
2012-12-21 10:45:39dokosetstatus: open -> closed
resolution: works for me
messages: + msg177875
2012-12-20 18:31:10dokosetmessages: + msg177849
2012-12-20 18:26:23barrysetnosy: + barry
2012-12-20 17:45:23vinay.sajipsetmessages: + msg177839
2012-12-20 17:00:45dokosetmessages: + msg177837
2012-12-12 12:10:34dokosetnosy: + doko
2012-11-23 19:44:44vinay.sajipsetmessages: + msg176224
2012-11-22 05:21:07merwoksetmessages: + msg176098
versions: + Python 3.4
2012-11-21 20:34:04Marco.Amadorisetmessages: + msg176078
2012-11-21 19:04:24carljmsetmessages: + msg176074
2012-11-21 08:31:45Marco.Amadorisetmessages: + msg176046
title: pyvenv 3.3 fails to create symlinks for <virtualenv>/local/{bin,lib} to <virtualenv>/{bin,lib} -> pyvenv 3.3 fails to create symlinks for <virtualenv>/local/{bin, lib} to <virtualenv>/{bin, lib}
2012-11-20 23:21:42carljmsetmessages: + msg176037
2012-11-18 21:25:27Marco.Amadorisetmessages: + msg175924
2012-11-17 15:00:50vinay.sajipsetmessages: + msg175749
components: + Library (Lib), - Build, 2to3 (2.x to 3.x conversion tool), ctypes, Benchmarks, Cross-Build
2012-11-17 14:59:33vinay.sajipsetnosy: + carljm
messages: + msg175748
components: + Build, 2to3 (2.x to 3.x conversion tool), ctypes, Benchmarks, Cross-Build, - None
2012-11-17 14:08:22asvetlovsetnosy: + asvetlov
2012-11-17 06:10:28merwoksetnosy: + merwok
2012-11-17 02:55:52ned.deilysetnosy: + vinay.sajip, ned.deily
messages: + msg175724
2012-11-15 16:55:28Marco.Amadoricreate