classification
Title: A venv created and activated from within a virtualenv uses the outer virtualenv's site-packages rather than its own.
Type: Stage: resolved
Components: Library (Lib) Versions: Python 3.6
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: Kentzo, anowlcalledjosh, cosven, uranusjr, vinay.sajip
Priority: normal Keywords:

Created on 2017-06-30 09:02 by Antony.Lee, last changed 2018-08-15 19:04 by uranusjr. This issue is now closed.

Messages (17)
msg297365 - (view) Author: Antony Lee (Antony.Lee) * Date: 2017-06-30 09:02
Python 3.6.1, virtualenv 15.1.0 (Arch Linux, distro packages)

```
export PIP_CONFIG_FILE=/dev/null  # just to be sure
# python -mvirtualenv outer-env  # using /usr/bin/python(3)
python2 -mvirtualenv outer-env  # using /usr/bin/python(3)
source outer-env/bin/activate
python -mvenv inner-env  # using outer-env's python
source inner-env/bin/activate
python -minspect -d pip
```

The last step outputs
```
Target: pip
Origin: /tmp/inner-env/lib/python3.6/site-packages/pip/__init__.py
Cached: /tmp/inner-env/lib/python3.6/site-packages/pip/__pycache__/__init__.cpython-36.pyc
Loader: <_frozen_importlib_external.SourceFileLoader object at 0x7f27c019c710>
Submodule search path: ['/tmp/inner-env/lib/python3.6/site-packages/pip']
```
i.e., the outer environment's pip leaks into the inner environment; the inner environment does not have its own pip installed.  Trying to use that pip from the inner environment (from inner-env, `python -mpip install ...`) results in the package being installed in the outer environment rather than the inner one.

Now this may all seem very academic, but for example Travis uses virtualenvs as toplevel containers for their Python projects, so this bug makes it difficult to test, say, projects that set up venvs as part of their workflow.
msg297431 - (view) Author: Antony Lee (Antony.Lee) * Date: 2017-06-30 17:10
Sorry, that was a sloppy report.  This is a better repro:

$ export PIP_CONFIG_FILE=/dev/null  # just to be sure
python -mvirtualenv outer-env  # using /usr/bin/python(3)
source outer-env/bin/activate
python -mvenv inner-env  # using outer-env's python
source inner-env/bin/activate
python -minspect -d pip

Using base prefix '/usr'
New python executable in /tmp/outer-env/bin/python
Installing setuptools, pip, wheel...done.
Target: pip
Origin: /tmp/outer-env/lib/python3.6/site-packages/pip/__init__.py
Cached: /tmp/outer-env/lib/python3.6/site-packages/pip/__pycache__/__init__.cpython-36.pyc
Loader: <_frozen_importlib_external.SourceFileLoader object at 0x7f92e00e03c8>
Submodule search path: ['/tmp/outer-env/lib/python3.6/site-packages/pip']
msg306677 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2017-11-21 19:03
Can you explain why virtualenv and venv need to be mixed in this way, other than as an academic exercise? If you use -m venv for both inner and outer venvs, then it seems to work as expected.

I'm planning to close this as "not a bug" unless you can provide a reason why this use case needs to be supported.
msg306679 - (view) Author: Antony Lee (Antony.Lee) * Date: 2017-11-21 19:28
Travis uses virtualenvs as toplevel containers for running CI.  Some projects need (or would like to) set up venvs as part of their test suites.  On example is https://github.com/airspeed-velocity/asv, which profiles a project's speed across revisions; in order to do so it sets up an environment for each revision to run the test suite.  Currently, it can either set up a virtualenv or a conda env, but it would make sense for it to be able to set up a venv too (https://github.com/airspeed-velocity/asv/pull/526).  But the unability to nest a venv into a virtualenv makes this impossible to test on Travis.
msg306691 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2017-11-21 21:51
The problem for venv is that it's tied to the running interpreter, which is (in the case you mention) the one from the outer virtualenv. Unlike virtualenv, venv does not provide a mechanism to restart itself with a different Python interpreter.

I don't know how Conda works in detail, but virtualenv makes copies of parts of the stdlib; venv, on the other hand, doesn't, but keeps a reference to its original Python environment. That is, I think, the reason for the difference in behaviour.

Instead of just using 'python' to invoke the command which runs the "-mvenv", you could get the underlying Python and create the venv using that. For example, using this script "upvenv.py":

#!/usr/bin/env python
def make_venv(venvpath):
    import os
    import subprocess
    import sysconfig

    python = os.path.join(sysconfig.get_config_var('BINDIR'), 'python3')
    cmd = [python, '-mvenv', venvpath]
    subprocess.run(cmd)

if __name__ == '__main__':
    import sys
    assert len(sys.argv) == 2
    try:
        rc = make_venv(sys.argv[1])
    except Exception as e:
        print('Failed: %s' % e)
        rc = 1
    sys.exit(rc)

I get:

vinay@ubuntu:/tmp$ cd /tmp
vinay@ubuntu:/tmp$ ve15 --version
15.1.0
vinay@ubuntu:/tmp$ ve15 -p python3 outer
Running virtualenv with interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /tmp/outer/bin/python3
Also creating executable in /tmp/outer/bin/python
Installing setuptools, pip, wheel...done.
vinay@ubuntu:/tmp$ source outer/bin/activate
(outer)  vinay@ubuntu:/tmp$ python upvenv.py inner
(outer)  vinay@ubuntu:/tmp$ source inner/bin/activate
(inner)  vinay@ubuntu:/tmp$ python -minspect -d pip
Target: pip
Origin: /tmp/inner/lib/python3.5/site-packages/pip/__init__.py
Cached: /tmp/inner/lib/python3.5/site-packages/pip/__pycache__/__init__.cpython-35.pyc
Loader: <_frozen_importlib_external.SourceFileLoader object at 0x7f6917e9e860>
Submodule search path: ['/tmp/inner/lib/python3.5/site-packages/pip']

which seems OK.

This approach can't be transplanted into venv because it doesn't have the ability to restart with a different interpreter.
msg306697 - (view) Author: Antony Lee (Antony.Lee) * Date: 2017-11-22 07:03
> venv, on the other hand, doesn't, but keeps a reference to its original Python environment. That is, I think, the reason for the difference in behaviour.

But for example, using the system Python to create a venv (no nesting), packages installed system-wide (e.g. using a linux package manager) are not visible in the venv.  Or, as you note, nesting a venv into another works fine.  I don't understand why things would be different when nesting?

In any case thanks for the workaround.
msg306699 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2017-11-22 07:22
> I don't understand why things would be different when nesting?

Specifically because venv keeps "a pointer" to the Python environment it was created from. Usually that's a system Python. If a venv ("inner") is created from a virtualenv's interpreter, the pointer points back to that environment ("outer", in this case).

Also, virtualenv can re-invoke itself with a different interpreter easily - that's business as usual. However, the venv module is part of a specific Python stdlib and doesn't reinvoke itself with a different interpreter.

To find the exact mechanisms which lead to these behaviours, you would need to examine the code of virtualenv and the venv module and perhaps do some stepping through in a debugger!
msg306700 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2017-11-22 07:27
Is it OK if I close this? It's not really a bug, nor a case that's been designed for (a stdlib module supporting specific external packages is very unusual - pip support via ensurepip is perhaps the one exception I can think of). Given there's a way to create a venv from a virtualenv's Python, as posted above, I'm not sure what more I can do.

If, on the other hand, you look into the details and find something I've missed, I'll certainly look again.
msg306702 - (view) Author: Antony Lee (Antony.Lee) * Date: 2017-11-22 07:47
I guess it's reasonable, I'll see whether we can use the workaround you proposed.  (Could a fix on virtualenv's side help?)
Thanks for the explanations.
msg306786 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2017-11-23 07:20
> I guess it's reasonable, I'll see whether we can use the workaround you proposed.

OK, please post your findings here.

> (Could a fix on virtualenv's side help?)

Probably not, as it's not behaving unexpectedly. It might be worth asking the Travis devs for an enhancement to allow the choice of -m venv rather than virtualenv to create venvs for Python >= 3.3, via an option in .travis.yml.
msg310185 - (view) Author: Tzu-ping Chung (uranusjr) * Date: 2018-01-17 15:18
Would it be possible for venv to do this automatically? Instead of using sys.executable, use

    import sysconfig
    dirname, exename = sysconfig.get_config_vars('BINDIR', 'BUILDPYTHON')

to populate values in the context?
msg310232 - (view) Author: Tzu-ping Chung (uranusjr) * Date: 2018-01-18 10:56
Plot twist: the workaround does not work on Windows.

    $ cd SOMEDIR   ; Placeholder for a valid directory.
    $ virtualenv env1
    ...
    $ env\Scripts\python.exe
    Python 3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:54:40) [MSC v.1900 64 bit (AMD64)] on win32
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import sysconfig
    >>> print(sysconfig.get_config_var('BINDIR'))
    SOMDIR\env1\Scripts

This points to the virtualenv's BINDIR, not the real Python installation's :(
msg313131 - (view) Author: cosven (cosven) * Date: 2018-03-02 12:12
It is actually a bug in virtualenv instead of venv.

> the outer environment's pip leaks into the inner environment;

This is true, but why? In short, the inner-env python binary
use `outer-env/` directory as its `prefix` direcotry,
but the `inner-env/` directory is the right `prefix` directory.

When the Python binary is executed, it attempts to determine
its `prefix` (which it stores in sys.prefix), which is then used
to find the standard library and other key files, and by the
`site` module to determine the location of the site-package directories.

However, virtualenv has its own `site` module, which is different
implemented from the site module in stdlib. It makes the inner-python
get a wrong `prefix` value.

> (Could a fix on virtualenv's side help?)
In my own opinion, if virtualenv change its implementation of site.py,
there is a change to fix this.
msg313142 - (view) Author: Tzu-ping Chung (uranusjr) * Date: 2018-03-02 17:17
@cosven you are correct! I made some additional observation while working on adding venv support to pew, in this thread: https://github.com/berdario/pew/pull/173. I’ll try to summarise below.

The root problem is indeed virtualenv’s custom site module, but even then it is possible for venv (or any tool built around it) to know where the “original” Python is, since virtualenv stores the old sys.prefix in sys.real_prefix. Unfortunately, however, this is not as useful as it seems. What we really want in this situation is sys.exec_prefix, which may or may not be the same as sys.prefix. I did manage to cobble together a hack, but it is by no means bullet-proof.

To actually deal with this problem, we will need to amend the site module provided by virtualenv, as you mentioned. But this still can’t “solve” it. Since the site module is vendored in the produced virtualenv, existing virtualenvs will stay broken even after we managed to upgrade the virtualenv installations.
msg320034 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2018-06-20 07:50
I'm closing this, as it's not a bug in the venv module.
msg323580 - (view) Author: Ilya Kulakov (Kentzo) Date: 2018-08-15 18:44
Also hit this issue while trying to run venv on Travis.

Perhaps venv should detect if it's running under virtualenv (e.g. the VIRTUAL_ENV env is present) and issue a warning?
msg323581 - (view) Author: Tzu-ping Chung (uranusjr) * Date: 2018-08-15 19:04
Unfortunately it is not a viable solution. If you run the virtualenv python without activation, e.g.

    virtualenv --python=python3.7 foo
    foo/bin/python -m venv bar

The VIRTUAL_ENV environment variable wouldn’t be set in this situation, but the created venv (bar) would still be broken.

The only viable cue I find to detect virtualenv existence is it sets sys.real_prefix to point to the prefix of the actual Python isntallation (i.e. the value of sys.prefix without virtualenv), while this variable does not exist by default. This is, however, obviously very hacky and unreliable, and not something venv should do IMO. virtualenv is the problem here, and should be responsible for fixing this situation instead.
History
Date User Action Args
2018-08-15 19:04:01uranusjrsetmessages: + msg323581
2018-08-15 18:44:12Kentzosetnosy: + Kentzo
messages: + msg323580
2018-06-20 07:50:37vinay.sajipsetstatus: open -> closed
resolution: not a bug
messages: + msg320034

stage: resolved
2018-06-19 22:43:14anowlcalledjoshsetnosy: + anowlcalledjosh
2018-03-02 17:17:46uranusjrsetmessages: + msg313142
2018-03-02 12:12:03cosvensetnosy: + cosven
messages: + msg313131
2018-01-18 10:56:09uranusjrsetmessages: + msg310232
2018-01-17 15:29:13Antony.Leesetnosy: - Antony.Lee
2018-01-17 15:18:06uranusjrsetnosy: + uranusjr
messages: + msg310185
2017-11-23 07:20:50vinay.sajipsetmessages: + msg306786
2017-11-22 07:47:26Antony.Leesetmessages: + msg306702
2017-11-22 07:27:12vinay.sajipsetmessages: + msg306700
2017-11-22 07:22:44vinay.sajipsetmessages: + msg306699
2017-11-22 07:03:39Antony.Leesetmessages: + msg306697
2017-11-21 21:51:51vinay.sajipsetmessages: + msg306691
2017-11-21 19:28:49Antony.Leesetstatus: pending -> open

messages: + msg306679
2017-11-21 19:03:57vinay.sajipsetstatus: open -> pending

messages: + msg306677
2017-10-25 22:48:05berker.peksagsetnosy: + vinay.sajip
2017-06-30 17:10:33Antony.Leesetmessages: + msg297431
2017-06-30 09:02:48Antony.Leecreate