When Envbuilder creates a new virtualenv with system_site_packages=True,
in the process of installing pip itself it actually doesn't install it inside the virtualenv because it is already there in the system.

However when you do it manually using "virtualenv" it does install pip inside the virtualenv as you would expect to.


```
(22d667c6-e383-47be-9785-ec044008b654)  ~/.local/share/fades/22d667c6-e383-47be-9785-ec044008b654  pwd
/home/gilgamezh/.local/share/fades/22d667c6-e383-47be-9785-ec044008b654
(22d667c6-e383-47be-9785-ec044008b654)  ~/.local/share/fades/22d667c6-e383-47be-9785-ec044008b654  ll
total 16
drwxr-xr-x 2 gilgamezh users 4096 Aug 16 00:05 bin
drwxr-xr-x 2 gilgamezh users 4096 Aug 16 00:05 include
drwxr-xr-x 3 gilgamezh users 4096 Aug 16 00:05 lib
lrwxrwxrwx 1 gilgamezh users    3 Aug 16 00:05 lib64 -> lib
-rw-r--r-- 1 gilgamezh users   68 Aug 16 00:05 pyvenv.cfg
(22d667c6-e383-47be-9785-ec044008b654)  ~/.local/share/fades/22d667c6-e383-47be-9785-ec044008b654  ll bin 
total 12
-rw-r--r-- 1 gilgamezh users 2251 Aug 16 00:05 activate
-rw-r--r-- 1 gilgamezh users 1367 Aug 16 00:05 activate.csh
-rw-r--r-- 1 gilgamezh users 2503 Aug 16 00:05 activate.fish
lrwxrwxrwx 1 gilgamezh users    7 Aug 16 00:05 python -> python3
lrwxrwxrwx 1 gilgamezh users   16 Aug 16 00:05 python3 -> /usr/bin/python3
(22d667c6-e383-47be-9785-ec044008b654)  ~/.local/share/fades/22d667c6-e383-47be-9785-ec044008b654  bin/python -Im ensurepip --upgrade --default-pip -v 
Ignoring indexes: https://pypi.python.org/simple
URLs to search for versions for setuptools in /usr/lib/python3.4/site-packages:
Skipping link /tmp/tmpe8rbjkcq (from -f); not a file
Skipping link file:///tmp/tmpe8rbjkcq/pip-6.0.8-py2.py3-none-any.whl; wrong project name (not setuptools)
Found link file:///tmp/tmpe8rbjkcq/setuptools-12.0.5-py2.py3-none-any.whl, version: 12.0.5
Local files found: /tmp/tmpe8rbjkcq/setuptools-12.0.5-py2.py3-none-any.whl
Installed version (18.1) is most up-to-date (past versions: 12.0.5)
Requirement already up-to-date: setuptools in /usr/lib/python3.4/site-packages
URLs to search for versions for pip in /usr/lib/python3.4/site-packages:
Found link file:///tmp/tmpe8rbjkcq/pip-6.0.8-py2.py3-none-any.whl, version: 6.0.8
Skipping link file:///tmp/tmpe8rbjkcq/setuptools-12.0.5-py2.py3-none-any.whl; wrong project name (not pip)
Local files found: /tmp/tmpe8rbjkcq/pip-6.0.8-py2.py3-none-any.whl
Installed version (7.1.0) is most up-to-date (past versions: 6.0.8)
Requirement already up-to-date: pip in /usr/lib/python3.4/site-packages
Cleaning up...
(22d667c6-e383-47be-9785-ec044008b654)  ~/.local/share/fades/22d667c6-e383-47be-9785-ec044008b654  
```

"bin/python -Im ensurepip --upgrade --default-pip -v" is from /usr/lib/python3.4/venv/__init__.py line 255

Hi, I have the same symptoms and noticed by accident that you can somewhat workaround it by running pyvenv twice:

  pyvenv /path/to/env && pyvenv --system-site-packages /path/to/env

The first provides pip, the second does not destroy it.
At least this works on my Debian 8 installation

This bug actually makes the --system-site-packages option useless, because it prevents installation of any package inside the virtualenv unless one tries the workaround given by gracinet.

I don't believe this is a pyvenv bug: pyvenv invokes

bin/python -Im ensurepip --upgrade --default-pip -v

which then terminates with

Requirement already up-to-date: pip in /usr/lib/python3.4/site-packages

as you can see from the console output. So the problem appears to be with ensurepip - all the pyvenv script does is to invoke ensurepip.

Adding dstufft to nosy.

You probably need to either pass --ignore-installed to pip when invoking it inside of venv, or you need to disable system_site_packages (internally to venv) until after pip is installed.

I agree this makes --system-site-packages a useless option unless it's fixed. We just had many beginners install pyvenv's and get very confused because of this.

Passing ensurepip a new option to force it to install within a venv I think would work. What would it take to make progress on this?

I have the same problem on a standard installation of Python 3.5 (on OS X via Homebrew). This bug is almost a year old, and it is a doozy: it utterly defeats the purpose of --system-site-packages. 

In the hope of getting some momentum going, I tried out the option suggested by dstufft: adding --ignore-installed in ensurepip's bootstrap() does seem to fix the problem.

I've never contributed a patch to Python before so I've probably screwed this up, but I'm attaching a patch that applies against latest python 3.5 on github (1b279c2). Sorry, I don't know what the equivalent Hg hash is, but this is a pretty minimal patch so it should be easy to apply.

Please ignore my previous message and patch. Although it installs a pip script into the virtual environment, the script's shebang points to the global python and not the venv python. I'm trying to figure out why this is, but the pip voodoo is way deeper than I understand. I can only guess that it has to do with sys.executable which appears to point to the normalized path, i.e. the system python instead of the symlink inside the venv.

So I tried dstufft's other suggestion: disable system site packages before bootstrapping pip and then re-enable it afterward. This actually does work in an end-to-end test on my system. So I'm attaching another patch that applies to python 3.5 (1b279c2 on github -- I don't know the equivalent mercurial hash) that is smaller than the previous patch. I don't know what the side effects are of monkeying around with system site packages like this, so apologies in advance if this is a stupid thing to do...

I'm attaching a small patch (against the current hg master branch) that I believe solves the problem, however, I think it was quite hacky.

Here is a walk-through of the logic.

1) If '--system-site-packages' is requested, then it is overrided initially in the process when creating the config file. I.e. it is hard-coded to 'false' in the pyvenv.cfg file.
2) Next ensurepip is run. This is being run in the same state as it would be if no '--system-site-packages' flag were passed, so this time it installs some packages to the venv's site-packages.
3) If necessary, the config file is regenerated without over-riding the system-site-packages flag.
4) If necessary, all packages installed in site-packages are removed and a new empty site-packages folder is regenerated.

The logic here is pretty horrid, but it seems to be functioning correctly. There are a couple possible issues here:

1) I assume that on all platforms the site-packages folder will always end up empty when the no '--system-site-packages' flag is passed.
2) I also assume that on all platforms there are no differences in the venv that is produced with or without this option except for the packages installed in the system-site-packages folder.

I don't really have any way to verify either of those assumption experimentally and I'm not familiar enough with ensurepip/pip to know that way either. I've only tested this on my debian system so it could very well not be portable.

Regardless of my misgivings of the quality of this patch, I think a solution to this problem is totally essential. The '--system-site-packages' flag is a great way to let your distro's package manager install most packages you need while allowing you then to install whatever others your distro does not provide. I use this all the time to simplify/decrease maintenance complexity. Without this fix, the --system-site-packages flag is essentially equivalent to a read-only environment and in that case, there's no point in using it at all.

Now that I look over this thread again, I realize that my patch ended up basically evolving to being extremely similar to Mark Haase's. The only thing that I don't understand though is that he says that his shebangs are somehow messed up. For me the shebangs are all fine with my patch. Possibly this is difference between his system (OSX) and mine (debian 8)? In any case, I'm afraid I may have re-engineered almost the same solution as him (and therefore that it doesn't work right), but I guess I needed to go through the process to familiarize myself with the problem.

Mark if you're still listening to this thread, maybe you could test this out and see if you're still having the same shebang problems? I can't reproduce your issue.

Hi Mark Haase,

I've gone through both of your patches and they both work for me as they should. I'm not sure why the first one isn't working for you, since it works for me. That one seems like it's solving the problem the "right" way. In any case, the second patch works for me as well so if that works for you, then at least we can confirm that it works for both your version of OSX and debian 8 which I'm running.

To any other higher powers: is Mark's second patch acceptable?

It seems simpler to do it all in EnvBuilder.create:

    def create(self, env_dir):
        """
        Create a virtual environment in a directory.

        :param env_dir: The target directory to create an environment in.

        """
        env_dir = os.path.abspath(env_dir)
        context = self.ensure_directories(env_dir)
        # See issue 24875. We need system_site_packages to be False
        # until after pip is installed.
        true_system_site_packages = self.system_site_packages
        self.system_site_packages = False
        self.create_configuration(context)
        self.setup_python(context)
        if self.with_pip:
            self._setup_pip(context)
        if not self.upgrade:
            self.setup_scripts(context)
            self.post_setup(context)
        if true_system_site_packages:
            # We had set it to False before, now
            # restore it and rewrite the configuration
            self.system_site_packages = True
            self.create_configuration(context)

Can you patch that in and see if it works for you?

Hi Vinay,

I just checked and yes that code works fine on my end (debian 8 box). Personally, I think either this or Mark's second patch would be fine.

Cheers,
Thomas

Just a sanity check ... I can't reproduce the issue with the latest versions of Python (I haven't made any changes to the venv code):

On Python 3.5:

 vinay@ubuntu:~/projects/python/3.5$ ./python
Python 3.5.3rc1+ (3.5:d9a64d00a439, Jan 13 2017, 10:26:23) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 
 vinay@ubuntu:~/projects/python/3.5$ ./python -m venv --system-site-packages /tmp/venv35
 vinay@ubuntu:~/projects/python/3.5$ /tmp/venv35/bin/pip --version
pip 9.0.1 from /tmp/venv35/lib/python3.5/site-packages (python 3.5)

Now try on 3.6:

 vinay@ubuntu:~/projects/python/3.5$ cd ../3.6
 vinay@ubuntu:~/projects/python/3.6$ ./python
Python 3.6.0+ (3.6:17ca7c4dd583, Jan 13 2017, 10:24:18) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 
 vinay@ubuntu:~/projects/python/3.6$ ./python -m venv --system-site-packages /tmp/venv36
 vinay@ubuntu:~/projects/python/3.6$ /tmp/venv36/bin/pip --version
pip 9.0.1 from /tmp/venv36/lib/python3.6/site-packages (python 3.6)

Now try on default (to-be-3.7):

 vinay@ubuntu:~/projects/python/default$ cd ../default
 vinay@ubuntu:~/projects/python/default$ ./python
Python 3.7.0a0 (default:0503024831ad, Jan 13 2017, 10:22:09) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 
 vinay@ubuntu:~/projects/python/default$ ./python -m venv --system-site-packages /tmp/venv37
 vinay@ubuntu:~/projects/python/default$ /tmp/venv37/bin/pip --version
pip 9.0.1 from /tmp/venv37/lib/python3.7/site-packages (python 3.7)

Let's try installing a package: pick a small, one-file one.

 vinay@ubuntu:~/projects/python/default$ /tmp/venv37/bin/pip install python-gnupg
Collecting python-gnupg
  Downloading python_gnupg-0.3.9-py2.py3-none-any.whl
Installing collected packages: python-gnupg
  Installing from a newer Wheel-Version (1.1)
Successfully installed python-gnupg-0.3.9
 vinay@ubuntu:~/projects/python/default$ ls -l /tmp/venv37/lib/python3.7/site-packages/
total 84
-rw-rw-r--  1 vinay vinay   126 Jan 13 11:00 easy_install.py
-rw-rw-r--  1 vinay vinay 51019 Jan 13 11:01 gnupg.py
drwxrwxr-x 11 vinay vinay  4096 Jan 13 11:00 pip
drwxrwxr-x  2 vinay vinay  4096 Jan 13 11:00 pip-9.0.1.dist-info
drwxrwxr-x  5 vinay vinay  4096 Jan 13 11:00 pkg_resources
drwxrwxr-x  2 vinay vinay  4096 Jan 13 11:01 __pycache__
drwxrwxr-x  2 vinay vinay  4096 Jan 13 11:01 python_gnupg-0.3.9.dist-info
drwxrwxr-x  5 vinay vinay  4096 Jan 13 11:00 setuptools
drwxrwxr-x  2 vinay vinay  4096 Jan 13 11:00 setuptools-28.8.0.dist-info

As you can see, gnupg.py is in the list, showing that the package was correctly installed in the virtualenv.

Hey Thomas and Vinay, thanks for looking into this. Although I discovered this problem on my MacOS laptop 6 months ago, today I also reproduced this issue on my Ubuntu desktop.

```
/home/mhaase/Downloads $ mkdir temp
/home/mhaase/Downloads $ cd temp
/home/mhaase/Downloads/temp $ pyvenv --system-site-packages pyvenv
/home/mhaase/Downloads/temp $ source pyvenv/bin/activate
(pyvenv) /home/mhaase/Downloads/temp $ pip --version
pip 9.0.1 from /usr/local/lib/python2.7/dist-packages (python 2.7)
(pyvenv) /home/mhaase/Downloads/temp $ python --version
Python 3.5.2
(pyvenv) /home/mhaase/Downloads/temp $ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.1 LTS"
(pyvenv) /home/mhaase/Downloads/temp $ uname -av
Linux prodigy 4.4.0-57-generic #78-Ubuntu SMP Fri Dec 9 23:50:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
```

This bug is really annoying if you work with scientific Python (numpy, scipy, sklearn), as those are big, stable packages that I would prefer to install once on my system and then access from multiple venvs. The standardization of linux wheels in 2016 made this slightly less problematic (don't need to compile all of numpy just to create a venv for an ipython notebook), but this is still an annoying bug. Unfortunately, I think it lacks the upvotes to get attention from the PyPA developers.

Maybe it would help to post this on the https://groups.google.com/forum/#!forum/pypa-dev mailing list?

I see why I couldn't reproduce the problem - there's no pip installed in the site-packages of the Pythons I was testing with, so the problem didn't show up.

Mark, can you apply the version of EnvBuilder.create that I posted in a message above and see if that resolves the issue for you on both OS X and Ubuntu? Thomas has confirmed that it works for him.

Hi Vinay,

You should probably upload a patch with the changes you made (however trivial) if you want that version considered. Makes it easier to comment and can then point to it for others to consider.

Cheers,
Thomas

Patch uploaded herewith.

Vinay, thanks for the patch! I just tested it on my Ubuntu 16.04.1 box with Python 3.5.2 and it works perfectly. Your patch is cleaner than mine, too.

Any idea how we get a CPython dev to look at this patch?

> Any idea how we get a CPython dev to look at this patch?

Well, I'm one of them, so ... :-)

I will commit it soon, given that Thomas Nyberg has also given his feedback. It was just waiting for your comments ...

Oh, sorry! I didn't realize you were a core dev or I would have replied sooner. Thank you for looking into this issue.

New changeset 1bf1d36463b4 by Vinay Sajip in branch '3.5':
Fixes #24875: pip can now be installed in a venv with --system-site-packages.
https://hg.python.org/cpython/rev/1bf1d36463b4

New changeset 7d4701b4210f by Vinay Sajip in branch '3.6':
Fixes #24875: Merged fix from 3.5.
https://hg.python.org/cpython/rev/7d4701b4210f

New changeset 7fd931727764 by Vinay Sajip in branch 'default':
Closes #24875: Merged fix from 3.6.
https://hg.python.org/cpython/rev/7fd931727764

New changeset fca720360f87f6dd2cd2030f7f98f689809c45d9 by Vinay Sajip in branch 'master':
Fixes #24875: pip can now be installed in a venv with --system-site-packages.
https://github.com/python/cpython/commit/fca720360f87f6dd2cd2030f7f98f689809c45d9

New changeset e3b461cf4327bc933b2421121daa5f938d929db3 by Vinay Sajip in branch 'master':
Fixes #24875: Merged fix from 3.5.
https://github.com/python/cpython/commit/e3b461cf4327bc933b2421121daa5f938d929db3

New changeset 2a2f5a741c509381ea8d0da09944aafdfc40b670 by Vinay Sajip in branch 'master':
Closes #24875: Merged fix from 3.6.
https://github.com/python/cpython/commit/2a2f5a741c509381ea8d0da09944aafdfc40b670

New changeset fca720360f87f6dd2cd2030f7f98f689809c45d9 by Vinay Sajip in branch '3.5':
Fixes #24875: pip can now be installed in a venv with --system-site-packages.
https://github.com/python/cpython/commit/fca720360f87f6dd2cd2030f7f98f689809c45d9

New changeset fca720360f87f6dd2cd2030f7f98f689809c45d9 by Vinay Sajip in branch '3.6':
Fixes #24875: pip can now be installed in a venv with --system-site-packages.
https://github.com/python/cpython/commit/fca720360f87f6dd2cd2030f7f98f689809c45d9

New changeset e3b461cf4327bc933b2421121daa5f938d929db3 by Vinay Sajip in branch '3.6':
Fixes #24875: Merged fix from 3.5.
https://github.com/python/cpython/commit/e3b461cf4327bc933b2421121daa5f938d929db3