classification
Title: Cannot create a venv on Windows when directory path contains dollar character
Type: crash Stage:
Components: Windows Versions: Python 3.8, Python 3.7, Python 3.6, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Stuart Cuthbertson, apatrushev, paul.moore, steve.dower, tim.golden, wolma, zach.ware
Priority: normal Keywords:

Created on 2018-04-01 02:25 by Stuart Cuthbertson, last changed 2018-07-25 19:23 by apatrushev.

Messages (4)
msg314755 - (view) Author: Stuart Cuthbertson (Stuart Cuthbertson) Date: 2018-04-01 02:25
I should clarify first that I haven't reproduced the following bug specifically with venv. I was asked to raise this here after raising an identical issue about virtualenv (https://github.com/pypa/virtualenv/issues/1154); a GitHub user told me this would also apply to venv. 

The bug with virtualenv is that it errors if passed a directory that contains a $ (dollar symbol). $ is a valid character for Windows directory names, filenames, and usernames. So running something simple like `python3 -m venv` (presumably) can fail in some valid Windows directories. 

The full error traceback for virtualenv is available at the above GitHub URL. A commenter in the virtualenv project (see https://github.com/pypa/virtualenv/issues/457#issuecomment-377159868) suggested that this happens because the directory path is passed as-is (with $) to distutils, and distutils is seeing the text following the $ as a placeholder and trying to replace it with a variable, which isn't found.
msg314880 - (view) Author: Wolfgang Maier (wolma) * Date: 2018-04-03 13:55
An exotic case, but it also affects Linux:

python3.7 -m venv 'at$test'
Error: Command '['/home/maier/at$test/bin/python3.7', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 2.
[maier@nb19 ~]$ mkdir 'at$test'
mkdir: cannot create directory ‘at$test’: File exists
[maier@nb19 ~]$ cd 'at$test'
[maier@nb19 at$test]$ bin/python -m ensurepip
Collecting setuptools
Collecting pip
Installing collected packages: setuptools, pip
Exception:
Traceback (most recent call last):
  File "/usr/lib64/python3.7/distutils/util.py", line 187, in subst_vars
    return re.sub(r'\$([a-zA-Z_][a-zA-Z_0-9]*)', _subst, s)
  File "/usr/lib64/python3.7/re.py", line 198, in sub
    return _compile(pattern, flags).sub(repl, string, count)
  File "/usr/lib64/python3.7/distutils/util.py", line 184, in _subst
    return os.environ[var_name]
  File "/usr/lib64/python3.7/os.py", line 678, in __getitem__
    raise KeyError(key) from None
KeyError: 'test'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/tmpppaapmyk/pip-9.0.1-py2.py3-none-any.whl/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/tmp/tmpppaapmyk/pip-9.0.1-py2.py3-none-any.whl/pip/commands/install.py", line 342, in run
    prefix=options.prefix_path,
  File "/tmp/tmpppaapmyk/pip-9.0.1-py2.py3-none-any.whl/pip/req/req_set.py", line 784, in install
    **kwargs
  File "/tmp/tmpppaapmyk/pip-9.0.1-py2.py3-none-any.whl/pip/req/req_install.py", line 851, in install
    self.move_wheel_files(self.source_dir, root=root, prefix=prefix)
  File "/tmp/tmpppaapmyk/pip-9.0.1-py2.py3-none-any.whl/pip/req/req_install.py", line 1064, in move_wheel_files
    isolated=self.isolated,
  File "/tmp/tmpppaapmyk/pip-9.0.1-py2.py3-none-any.whl/pip/wheel.py", line 247, in move_wheel_files
    prefix=prefix,
  File "/tmp/tmpppaapmyk/pip-9.0.1-py2.py3-none-any.whl/pip/locations.py", line 153, in distutils_scheme
    i.finalize_options()
  File "/usr/lib64/python3.7/distutils/command/install.py", line 307, in finalize_options
    self.expand_basedirs()
  File "/usr/lib64/python3.7/distutils/command/install.py", line 486, in expand_basedirs
    self._expand_attrs(['install_base', 'install_platbase', 'root'])
  File "/usr/lib64/python3.7/distutils/command/install.py", line 480, in _expand_attrs
    val = subst_vars(val, self.config_vars)
  File "/usr/lib64/python3.7/distutils/util.py", line 189, in subst_vars
    raise ValueError("invalid variable '$%s'" % var)
ValueError: invalid variable '$'test''

So the venv actually gets created, but it's ensurepip which chokes on the $.
msg322386 - (view) Author: Anton Patrushev (apatrushev) Date: 2018-07-25 19:00
I found when this "feature" was implemented:
gitff2d9b71547d95566416fa968872910ca9c4adb1

Part of commit message:
```
in command-line options, and in two phases at that: first, we expand
'install_base' and 'install_platbase', and then the other 'install_*'
options.  This lets us do tricky stuff like
    install --prefix='/tmp$sys_prefix'
...oooh, neat.
```

So this was intentional change in distutils.

The only suggestion I have to fix this issue without break something in the wild is to suppress exception in Lib/distutils/util.py:189 and leave variable without available substitutions as it is. To be on safe side this can be even optional and disabled by default.

I am newcomer so some guidance will be helpful.
msg322389 - (view) Author: Anton Patrushev (apatrushev) Date: 2018-07-25 19:23
The same problem is reproducible with different but obvious way on Python 2.7.
History
Date User Action Args
2018-07-25 19:23:29apatrushevsetmessages: + msg322389
versions: + Python 2.7
2018-07-25 19:00:18apatrushevsetnosy: + apatrushev
messages: + msg322386
2018-04-03 13:55:36wolmasetnosy: + wolma

messages: + msg314880
versions: + Python 3.7, Python 3.8
2018-04-01 02:25:43Stuart Cuthbertsoncreate