classification
Title: python -m venv symlink dependency on how python binary is called is not documented
Type: behavior Stage: needs patch
Components: Documentation Versions: Python 3.7, Python 3.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: docs@python, r.david.murray, seliger
Priority: normal Keywords:

Created on 2017-12-29 02:48 by seliger, last changed 2017-12-29 14:15 by r.david.murray.

Messages (4)
msg309157 - (view) Author: Corey Seliger (seliger) Date: 2017-12-29 02:48
The venv module does not evenly create the necessary binaries/symlinks when building virtualenvs. If you call venv like this:

seliger@core:~/venvs$ python3 -m venv venvA

seliger@core:~/venvs$ ls -l venvA/bin/
total 32
-rw-r--r-- 1 seliger seliger 2143 Dec 28 21:29 activate
-rw-r--r-- 1 seliger seliger 1259 Dec 28 21:29 activate.csh
-rw-r--r-- 1 seliger seliger 2397 Dec 28 21:29 activate.fish
-rwxrwxr-x 1 seliger seliger  254 Dec 28 21:29 easy_install
-rwxrwxr-x 1 seliger seliger  254 Dec 28 21:29 easy_install-3.5
-rwxrwxr-x 1 seliger seliger  226 Dec 28 21:29 pip
-rwxrwxr-x 1 seliger seliger  226 Dec 28 21:29 pip3
-rwxrwxr-x 1 seliger seliger  226 Dec 28 21:29 pip3.5
lrwxrwxrwx 1 seliger seliger    7 Dec 28 21:29 python -> python3
lrwxrwxrwx 1 seliger seliger   16 Dec 28 21:29 python3 -> /usr/bin/python3

...you do not end up with a "python3.5" binary. 

However, if you call venv like this:

seliger@core:~/venvs$ python3.5 -m venv venvB

seliger@core:~/venvs$ ls -l venvB/bin
total 32
-rw-r--r-- 1 seliger seliger 2143 Dec 28 21:29 activate
-rw-r--r-- 1 seliger seliger 1259 Dec 28 21:29 activate.csh
-rw-r--r-- 1 seliger seliger 2397 Dec 28 21:29 activate.fish
-rwxrwxr-x 1 seliger seliger  256 Dec 28 21:29 easy_install
-rwxrwxr-x 1 seliger seliger  256 Dec 28 21:29 easy_install-3.5
-rwxrwxr-x 1 seliger seliger  228 Dec 28 21:29 pip
-rwxrwxr-x 1 seliger seliger  228 Dec 28 21:29 pip3
-rwxrwxr-x 1 seliger seliger  228 Dec 28 21:29 pip3.5
lrwxrwxrwx 1 seliger seliger    9 Dec 28 21:29 python -> python3.5
lrwxrwxrwx 1 seliger seliger    9 Dec 28 21:29 python3 -> python3.5
lrwxrwxrwx 1 seliger seliger   18 Dec 28 21:29 python3.5 -> /usr/bin/python3.5

...you DO get the necessary python3.5 binary. Some vendors are making it a requirement to call a specific pythonX.Y binary to ensure compatibility. One such example is the serverless framework when deploying to Amazon Web Services. Another example is the useful pyenv utility that manages full Python builds and virtualenvs. When it depends upon venv, it exhibits the same behavior. I submitted a patch workaround to force calling venv using pythonX.Y, but this really seems like an issue with the venv module itself.

The expected behavior should be that venv generates all three binaries (python, python3, and python3.5) regardless of how the python command was invoked.

I am able to reproduce this on Python 3.5 and 3.6. I could not find any other similar references in searching the bug system.
msg309158 - (view) Author: Corey Seliger (seliger) Date: 2017-12-29 02:54
Here is the write-up I posted over on the pyenv issues page:

This is apparently a behavior issue in the way venv works. Not only does this impact 3.6.x, I also confirmed that the same issue occurs with venv in general using the stock 3.5 Python that comes with Ubuntu 16.04.

The trouble is at Lib/venv/__init__.py:111. Starting at line 111:

        if sys.platform == 'darwin' and '__PYVENV_LAUNCHER__' in env:
            executable = os.environ['__PYVENV_LAUNCHER__']
        else:
            executable = sys.executable
        dirname, exename = os.path.split(os.path.abspath(executable))
        context.executable = executable

This carries down to the setup_python method starting at line 187 (Lib/venv/__init__.py:187). It takes whatever executable that is called as the source, and then creates the others based on a hard coded list (e.g. python and python3).

    def setup_python(self, context):
        """
        Set up a Python executable in the environment.
        :param context: The information for the environment creation request
                        being processed.
        """
        binpath = context.bin_path
        path = context.env_exe
        copier = self.symlink_or_copy
        copier(context.executable, path)
        dirname = context.python_dir
        if os.name != 'nt':
            if not os.path.islink(path):
                os.chmod(path, 0o755)
            for suffix in ('python', 'python3'):
                path = os.path.join(binpath, suffix)
                if not os.path.exists(path):
                    # Issue 18807: make copies if
                    # symlinks are not wanted
                    copier(context.env_exe, path, relative_symlinks_ok=True)
                    if not os.path.islink(path):
                        os.chmod(path, 0o755)
        else:
            subdir = 'DLLs'
            include = self.include_binary
            files = [f for f in os.listdir(dirname) if include(f)]
            for f in files:
                src = os.path.join(dirname, f)
                dst = os.path.join(binpath, f)
                if dst != context.env_exe:  # already done, above
                    copier(src, dst)
            dirname = os.path.join(dirname, subdir)
            if os.path.isdir(dirname):
                files = [f for f in os.listdir(dirname) if include(f)]
                for f in files:
                    src = os.path.join(dirname, f)
                    dst = os.path.join(binpath, f)
                    copier(src, dst)
            # copy init.tcl over
            for root, dirs, files in os.walk(context.python_dir):
                if 'init.tcl' in files:
                    tcldir = os.path.basename(root)
                    tcldir = os.path.join(context.env_dir, 'Lib', tcldir)
                    if not os.path.exists(tcldir):
                        os.makedirs(tcldir)
                    src = os.path.join(root, 'init.tcl')
                    dst = os.path.join(tcldir, 'init.tcl')
                    shutil.copyfile(src, dst)
                    break


If you don't call -m venv as python3.X, you will never get that binary in the bin directory.
msg309173 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2017-12-29 14:09
This is a feature that actually supports your use case, as well as the use cases of those who *don't* want to strap the python version: you get what you ask for.  If you call venv with 'python3', you get a venv that will use your new python if you upgrade your python, but if you call it with python3.x, you will get a venv that will *not* upgrade, which is exactly what you want for your use case.
msg309174 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2017-12-29 14:15
Actually, I'm going to reopen this as a doc issue because this behavior is not discussed by the docs that I can see, and it is important to know about when creating a venv.
History
Date User Action Args
2017-12-29 14:15:09r.david.murraysetstatus: closed -> open

assignee: docs@python
components: + Documentation, - Library (Lib)
title: python -m venv has incongruous behavor creating binaries -> python -m venv symlink dependency on how python binary is called is not documented
nosy: + docs@python

messages: + msg309174
resolution: not a bug ->
stage: resolved -> needs patch
2017-12-29 14:09:49r.david.murraysetstatus: open -> closed

versions: + Python 3.7, - Python 3.5
nosy: + r.david.murray

messages: + msg309173
resolution: not a bug
stage: resolved
2017-12-29 02:54:22seligersetmessages: + msg309158
2017-12-29 02:48:17seligercreate