classification
Title: venv.EnvBuilder environmental variable hooks
Type: enhancement Stage: patch review
Components: Library (Lib) Versions: Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: brett.cannon, donovick, vinay.sajip
Priority: normal Keywords: patch

Created on 2019-06-20 14:42 by donovick, last changed 2019-06-21 07:30 by vinay.sajip.

Pull Requests
URL Status Linked Edit
PR 14272 open donovick, 2019-06-20 15:08
Messages (13)
msg346122 - (view) Author: Caleb Donovick (donovick) * Date: 2019-06-20 14:42
Currently EnvBuilder allows for a number of customizations of virtual environments, however, it does not allow for any modifications of the activate script itself (unless one wants to rewrite it completely).  Yet, it is fairly common requirement for python packages is to set some environmental variable e.g., DJANGO_SETTINGS_MODULE=<settings>.

EnvBuilder should be able extend the set of variables which are set and restored on activate / deactivate (or more generally have activate / deactivate hooks).

Originally proposed on python-ideas https://mail.python.org/archives/list/python-ideas@python.org/thread/MJNFEFT4GBVBEETJWZUQM5SS6C34PT3K/
msg346124 - (view) Author: Caleb Donovick (donovick) * Date: 2019-06-20 15:24
I have basic version working in POSIX.
msg346140 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2019-06-20 17:12
I'm not very keen on this approach, because it needs to be implemented across all shells and maintenance in this area can be a headache because not everyone has knowledge of multiple shells.

There already are hooks for customising behaviour - you can subclass EnvBuilder and override e.g. setup_scripts to update the venv with the exact scripts you want. This puts the maintenance cost onto the people who need this functionality (it doesn't seem like a very common use case).
msg346144 - (view) Author: Caleb Donovick (donovick) * Date: 2019-06-20 17:32
Common python libraries that make use of environmental variables:
- django
- flask
- jupyter
- TensorFlow
This is just off the top my head.  While most developers are probably fine just setting the required variables at a system level that doesn't mean they should.  The whole point of virtual environments is to avoid global state.

Overriding setup scripts is hardly a trivial task.  One has to completely recreate the activate script to achieve the desired behavior.
msg346147 - (view) Author: Caleb Donovick (donovick) * Date: 2019-06-20 17:37
In regards to the added complexity by needing to be implemented across all shells, the functionality I am requesting already exists across all shells (e.g. setting PYTHONPATH) just not in a extensible way.
msg346149 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2019-06-20 17:58
> Common python libraries that make use of environmental variables

I didn't say environment variables weren't commonly used. I'm talking about the specific functionality this issue is about.

> Overriding setup scripts is hardly a trivial task. One has to completely recreate the activate script to achieve the desired behavior.

Not trivial, but not overly onerous either, e.g. just point to your own scripts subtree in the overridden method. Isn't your proposal to submit a different activate script?

> the functionality I am requesting already exists across all shells

I'm talking about the code to set, reset your custom environment variables - where is that already existent across all shells?

I'm not saying what you want is unreasonable - it's just putting it into the stdlib this way that I'm not sure about.
msg346155 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2019-06-20 18:30
I mentioned on python-ideas and opened https://bugs.python.org/issue37354 to actually make the Activate.ps1 file static for security purposes.
msg346160 - (view) Author: Caleb Donovick (donovick) * Date: 2019-06-20 19:17
> I didn't say environment variables weren't commonly used. I'm talking about the specific functionality this issue is about.

This is likely true, to most people venv is black magic.  That doesn't mean that people wouldn't use it if the functionality existed though.  In fact this the first example given by virtualenvwrapper https://virtualenvwrapper.readthedocs.io/en/latest/scripts.html#example-usage

I personally don't use virtualenvwrapper because you they use a centralized list of env's which is non-starter for me.  I maintain dozens of envs (one per project, sometimes one per branch). So one big list would be a nightmare.

> I'm talking about the code to set, reset your custom environment variables - where is that already existent across all shells?

My point is that this code is not really different than the code to set PYTHONPATH.

> I'm not saying what you want is unreasonable - it's just putting it into the stdlib this way that I'm not sure about.

Is there an architecture that would be less objectionable?

> I mentioned on python-ideas and opened https://bugs.python.org/issue37354 to actually make the Activate.ps1 file static for security purposes.

I am curious how you plan to achieve this without reducing the functionality of venv?
msg346167 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2019-06-20 21:50
> I am curious how you plan to achieve this without reducing the functionality of venv?

PowerShell has a full programming language and all details can be found in pyvenv.cfg now as of Python 3.8. So Activate.ps1 can read that file to get what it needs to set paths and the prompt which is what gets substituted into the generate Activate.ps1.
msg346169 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2019-06-20 21:53
> Is there an architecture that would be less objectionable?

One thing I would consider is the ability to configure a custom_script_path in the same way as other parameters are now. It would be used as follows: If not None, it should specify a directory, and the files in there would be copied to the target venv *after* the standard scripts are copied (possibly overwriting ones already there, such as "activate"), with the same variable substitutions that are currently done.

This approach allows for other things than just custom environment variable setting, and so it seems a more generic solution to the issue of customisability. While it involves the developers who require such functionality to maintain those scripts, it seems fair to place the onus on them, and not on stdlib maintainers.
msg346170 - (view) Author: Caleb Donovick (donovick) * Date: 2019-06-20 21:55
>PowerShell has a full programming language and all details can be found in pyvenv.cfg now as of Python 3.8. So Activate.ps1 can read that file to get what it needs to set paths and the prompt which is what gets substituted into the generate Activate.ps1.

Then that should be fairly orthogonal to this issue.  We should be able to store key, value pairs in the config and set the environmental variables accordingly.
msg346173 - (view) Author: Caleb Donovick (donovick) * Date: 2019-06-20 22:09
I have another idea, instead of baking in the semantics of the environmental variables, we could simply add the hooks __VENV_ACTIVATE_EXTRAS__ and __VENV_DEACTIVATE_EXTRAS__ to activate. By default they would be replaced by '', but would allow subclasses to extend the behavior of activate / deactivate arbitrarily. 

> One thing I would consider is the ability to configure a custom_script_path in the same way as other parameters are now. It would be used as follows: If not None, it should specify a directory, and the files in there would be copied to the target venv *after* the standard scripts are copied (possibly overwriting ones already there, such as "activate"), with the same variable substitutions that are currently done.

I'll take a stab at implementing this if the above proposal is not acceptable.
msg346187 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2019-06-21 07:30
> I'll take a stab at implementing this [custom_script_path]

I'll certainly look at a PR for this functionality.
History
Date User Action Args
2019-06-21 07:30:50vinay.sajipsetmessages: + msg346187
2019-06-20 22:09:43donovicksetmessages: + msg346173
2019-06-20 21:55:40donovicksetmessages: + msg346170
2019-06-20 21:53:32vinay.sajipsetmessages: + msg346169
2019-06-20 21:50:53brett.cannonsetmessages: + msg346167
2019-06-20 19:17:23donovicksetmessages: + msg346160
2019-06-20 18:30:15brett.cannonsetnosy: + brett.cannon
messages: + msg346155
2019-06-20 17:58:31vinay.sajipsetmessages: + msg346149
2019-06-20 17:37:17donovicksetmessages: + msg346147
2019-06-20 17:32:01donovicksetmessages: + msg346144
2019-06-20 17:12:55vinay.sajipsetmessages: + msg346140
2019-06-20 15:28:19xtreaksetnosy: + vinay.sajip

components: + Library (Lib), - Interpreter Core
versions: + Python 3.9
2019-06-20 15:24:48donovicksetmessages: + msg346124
2019-06-20 15:08:22donovicksetkeywords: + patch
stage: patch review
pull_requests: + pull_request14096
2019-06-20 14:42:40donovickcreate