This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Cannot create a venv on Windows when directory path contains dollar character
Type: crash Stage: resolved
Components: Windows Versions: Python 3.11, Python 3.10, Python 3.9
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: Stuart Cuthbertson, apatrushev, iritkatriel, paul.moore, steve.dower, tim.golden, wolma, zach.ware
Priority: normal Keywords:

Created on 2018-04-01 02:25 by Stuart Cuthbertson, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Messages (6)
msg314755 - (view) Author: Stuart Cuthbertson (Stuart Cuthbertson) Date: 2018-04-01 02:25
I should clarify first that I haven't reproduced the following bug specifically with venv. I was asked to raise this here after raising an identical issue about virtualenv (https://github.com/pypa/virtualenv/issues/1154); a GitHub user told me this would also apply to venv. 

The bug with virtualenv is that it errors if passed a directory that contains a $ (dollar symbol). $ is a valid character for Windows directory names, filenames, and usernames. So running something simple like `python3 -m venv` (presumably) can fail in some valid Windows directories. 

The full error traceback for virtualenv is available at the above GitHub URL. A commenter in the virtualenv project (see https://github.com/pypa/virtualenv/issues/457#issuecomment-377159868) suggested that this happens because the directory path is passed as-is (with $) to distutils, and distutils is seeing the text following the $ as a placeholder and trying to replace it with a variable, which isn't found.
msg314880 - (view) Author: Wolfgang Maier (wolma) * Date: 2018-04-03 13:55
An exotic case, but it also affects Linux:

python3.7 -m venv 'at$test'
Error: Command '['/home/maier/at$test/bin/python3.7', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 2.
[maier@nb19 ~]$ mkdir 'at$test'
mkdir: cannot create directory ‘at$test’: File exists
[maier@nb19 ~]$ cd 'at$test'
[maier@nb19 at$test]$ bin/python -m ensurepip
Collecting setuptools
Collecting pip
Installing collected packages: setuptools, pip
Exception:
Traceback (most recent call last):
  File "/usr/lib64/python3.7/distutils/util.py", line 187, in subst_vars
    return re.sub(r'\$([a-zA-Z_][a-zA-Z_0-9]*)', _subst, s)
  File "/usr/lib64/python3.7/re.py", line 198, in sub
    return _compile(pattern, flags).sub(repl, string, count)
  File "/usr/lib64/python3.7/distutils/util.py", line 184, in _subst
    return os.environ[var_name]
  File "/usr/lib64/python3.7/os.py", line 678, in __getitem__
    raise KeyError(key) from None
KeyError: 'test'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/tmpppaapmyk/pip-9.0.1-py2.py3-none-any.whl/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/tmp/tmpppaapmyk/pip-9.0.1-py2.py3-none-any.whl/pip/commands/install.py", line 342, in run
    prefix=options.prefix_path,
  File "/tmp/tmpppaapmyk/pip-9.0.1-py2.py3-none-any.whl/pip/req/req_set.py", line 784, in install
    **kwargs
  File "/tmp/tmpppaapmyk/pip-9.0.1-py2.py3-none-any.whl/pip/req/req_install.py", line 851, in install
    self.move_wheel_files(self.source_dir, root=root, prefix=prefix)
  File "/tmp/tmpppaapmyk/pip-9.0.1-py2.py3-none-any.whl/pip/req/req_install.py", line 1064, in move_wheel_files
    isolated=self.isolated,
  File "/tmp/tmpppaapmyk/pip-9.0.1-py2.py3-none-any.whl/pip/wheel.py", line 247, in move_wheel_files
    prefix=prefix,
  File "/tmp/tmpppaapmyk/pip-9.0.1-py2.py3-none-any.whl/pip/locations.py", line 153, in distutils_scheme
    i.finalize_options()
  File "/usr/lib64/python3.7/distutils/command/install.py", line 307, in finalize_options
    self.expand_basedirs()
  File "/usr/lib64/python3.7/distutils/command/install.py", line 486, in expand_basedirs
    self._expand_attrs(['install_base', 'install_platbase', 'root'])
  File "/usr/lib64/python3.7/distutils/command/install.py", line 480, in _expand_attrs
    val = subst_vars(val, self.config_vars)
  File "/usr/lib64/python3.7/distutils/util.py", line 189, in subst_vars
    raise ValueError("invalid variable '$%s'" % var)
ValueError: invalid variable '$'test''

So the venv actually gets created, but it's ensurepip which chokes on the $.
msg322386 - (view) Author: Anton Patrushev (apatrushev) Date: 2018-07-25 19:00
I found when this "feature" was implemented:
gitff2d9b71547d95566416fa968872910ca9c4adb1

Part of commit message:
```
in command-line options, and in two phases at that: first, we expand
'install_base' and 'install_platbase', and then the other 'install_*'
options.  This lets us do tricky stuff like
    install --prefix='/tmp$sys_prefix'
...oooh, neat.
```

So this was intentional change in distutils.

The only suggestion I have to fix this issue without break something in the wild is to suppress exception in Lib/distutils/util.py:189 and leave variable without available substitutions as it is. To be on safe side this can be even optional and disabled by default.

I am newcomer so some guidance will be helpful.
msg322389 - (view) Author: Anton Patrushev (apatrushev) Date: 2018-07-25 19:23
The same problem is reproducible with different but obvious way on Python 2.7.
msg404233 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2021-10-18 22:24
Reproduced on 3.11.
msg404234 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-10-18 22:40
Affects distutils, which is deprecated and won't be getting any more fixes.

I've migrated it to https://github.com/pypa/distutils/issues/63
History
Date User Action Args
2022-04-11 14:58:59adminsetgithub: 77374
2021-10-18 22:40:17steve.dowersetstatus: open -> closed
resolution: out of date
messages: + msg404234

stage: resolved
2021-10-18 22:24:10iritkatrielsetnosy: + iritkatriel

messages: + msg404233
versions: + Python 3.9, Python 3.10, Python 3.11, - Python 2.7, Python 3.6, Python 3.7, Python 3.8
2018-07-25 19:23:29apatrushevsetmessages: + msg322389
versions: + Python 2.7
2018-07-25 19:00:18apatrushevsetnosy: + apatrushev
messages: + msg322386
2018-04-03 13:55:36wolmasetnosy: + wolma

messages: + msg314880
versions: + Python 3.7, Python 3.8
2018-04-01 02:25:43Stuart Cuthbertsoncreate