Title: Default preference not given to venv DLL's
Type: behavior Stage: resolved
Components: Library (Lib), Windows Versions: Python 3.8
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: steve.dower Nosy List: eryksun, jonathan-lp, pablogsal, paul.moore, steve.dower, tim.golden, vinay.sajip, zach.ware
Priority: normal Keywords: patch

Created on 2018-06-30 10:07 by jonathan-lp, last changed 2019-07-26 16:40 by steve.dower. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 8253 merged vinay.sajip, 2018-07-11 14:53
PR 9458 merged steve.dower, 2018-09-20 18:00
PR 10400 merged pablogsal, 2018-11-07 22:04
PR 10402 merged miss-islington, 2018-11-07 22:22
PR 10401 merged miss-islington, 2018-11-07 22:22
Messages (9)
msg320765 - (view) Author: Jonathan (jonathan-lp) Date: 2018-06-30 10:07
I don't know if this is a bug or an odd design decision or just something that hasn't been considered or maybe I'm missing something.

On Windows and I create a venv with Python 3.6.3:

> python -m venv venv

This creates a subdirectory called /venv. Inside this, there's:

This is the sqlite library - except it's not, because Python isn't using this file. If I upgrade this library by replacing it with a newer sqlite3.dll version, Python keep using the original version of the library. Turns out, Python is by default reading the DLL in the root Python install:


Now, I can change that file and sure enough my VENV (*all* VENVs) then get the newer version of SQLite, or I can delete that file and the VENV's will all use their local versions, or I can possibly play with some sys.path to try and get the VENV version loaded first.

But this raises a few questions:
1) This seems like a rather odd default - copying a file that is never used by default.
2) Surely either the correct option is to use the VENV version by default?
3) Otherwise, what's the point in copying across this DLL file into the VENV by default?
msg320932 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2018-07-03 01:56
Offhand I don't know why it copies PYD and DLL files from the DLLs directory. It's part of the standard library. The virtual environment should only need to copy or symlink the binaries in the application directory, such as python.exe, pythonw.exe, vcruntime<###>.dll, python<##>.dll, and python<#>.dll. Currently a virtual environment doesn't use the binaries that get copied or symlinked from the DLLs directory, since the system DLLs directory is in sys.path.

Also, why does it copy over init.tcl? _tkinter uses Py_GetPrefix() to to get the real prefix directory in order to find the TCL library at "tcl\tclX.y". For a Python 3.7 virtual environment, I verified using Sysinternals procmon that TCL loads the init file from "C:\Program Files\Python37\tcl\tcl8.6\init.tcl", not the virtual environment.
msg320934 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2018-07-03 05:20
The code that does the copy is the original PEP 405 implementation code (from 26 May 2012). I'm fairly sure that these files were copied over only because (at least prior to the Python 3.3 release in September 2012) they were needed to be in the venv for the venv to work correctly. Python internals may have changed subsequently, making these copies unnecessary.

I can't easily recreate a Python 3.3.0 on Windows to confirm this, but I'm pretty sure I only coded it to copy over what was actually needed at the time.
msg320935 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2018-07-03 05:54
Hmmm. I managed to find a machine to install Python3.3.0 on, and it appears that even with that copy suite removed, the tests still work. So possibly it's something that changed between 3.3.0 alpha and 3.3.0 final.

I can try removing the copy code for 3.8 but I'm not sure if any code out there is relying on the copied files being in the venv, so it may not be a good idea to backport.
msg322024 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2018-07-20 16:07
New changeset 94487d45707772723ef19e86700a40a12743baa1 by Vinay Sajip in branch 'master':
bpo-34011: Update code copying DLLs and init.tcl into venvs. (GH-8253)
msg325899 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-09-20 16:39
This change breaks a number of tests when run on a proper installation, since we still need to copy pythonXY.dll or else python.exe refuses to start.

Since I'm fixing these tests today, I'll also fix this issue.
msg325930 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-09-20 20:38
New changeset f14c28f39766855420dd58d209da4ad847f3030e by Steve Dower in branch 'master':
bpo-34011: Fixes missing venv files and other tests (GH-9458)
msg325933 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-09-20 21:31
I agree with this only applying to 3.8 - removing the copied files on earlier versions seems like an unnecessary risk to compatibility.
msg329428 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2018-11-07 17:31
This buildbot is failing when raising when creating subprocess.CalledProcessError:

test_defaults (test.test_venv.BasicTest) ... ok
test_executable (test.test_venv.BasicTest) ... ok
test_executable_symlinks (test.test_venv.BasicTest) ... skipped 'Needs symlinks'
test_isolation (test.test_venv.BasicTest) ... ok
test_overwrite_existing (test.test_venv.BasicTest) ... ok
test_prefixes (test.test_venv.BasicTest) ... ok
test_prompt (test.test_venv.BasicTest) ... ok
test_symlinking (test.test_venv.BasicTest) ... skipped 'Needs symlinks'
test_unicode_in_batch_file (test.test_venv.BasicTest) ... ERROR
test_unoverwritable_fails (test.test_venv.BasicTest) ... ok
test_upgrade (test.test_venv.BasicTest) ... ok
test_devnull (test.test_venv.EnsurePipTest) ... ok
test_explicit_no_pip (test.test_venv.EnsurePipTest) ... ok
test_no_pip_by_default (test.test_venv.EnsurePipTest) ... ok
test_with_pip (test.test_venv.EnsurePipTest) ... ok
ERROR: test_unicode_in_batch_file (test.test_venv.BasicTest)
Traceback (most recent call last):
  File "C:\\3.x.kloth-win64\build\lib\test\", line 302, in test_unicode_in_batch_file
    out, err = check_output(
  File "C:\\3.x.kloth-win64\build\lib\test\", line 37, in check_output
    raise subprocess.CalledProcessError(
TypeError: __init__() takes from 3 to 5 positional arguments but 6 were given
I will prepare a PR fixing this
Date User Action Args
2019-07-26 16:40:38steve.dowersetstatus: open -> closed
stage: patch review -> resolved
2018-11-07 22:22:49miss-islingtonsetpull_requests: + pull_request9687
2018-11-07 22:22:22miss-islingtonsetpull_requests: + pull_request9686
2018-11-07 22:04:10pablogsalsetstage: resolved -> patch review
pull_requests: + pull_request9685
2018-11-07 17:31:26pablogsalsetstatus: closed -> open
nosy: + pablogsal
messages: + msg329428

2018-09-20 21:31:41steve.dowersetstatus: open -> closed
versions: - Python 3.6, Python 3.7
messages: + msg325933

resolution: fixed
stage: patch review -> resolved
2018-09-20 20:38:42steve.dowersetmessages: + msg325930
2018-09-20 18:00:30steve.dowersetpull_requests: + pull_request8872
2018-09-20 16:39:51steve.dowersetassignee: steve.dower
messages: + msg325899
2018-07-20 16:07:40vinay.sajipsetmessages: + msg322024
2018-07-11 14:53:51vinay.sajipsetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request7785
2018-07-03 05:54:07vinay.sajipsetmessages: + msg320935
2018-07-03 05:20:46vinay.sajipsetmessages: + msg320934
2018-07-03 01:58:12eryksunsetversions: + Python 3.7, Python 3.8
2018-07-03 01:56:39eryksunsetnosy: + eryksun, paul.moore, tim.golden, vinay.sajip, zach.ware, steve.dower
messages: + msg320932

components: + Library (Lib), Windows
type: behavior
stage: needs patch
2018-06-30 10:07:43jonathan-lpcreate