classification
Title: Make it easier to use a venv with an embedded Python interpreter
Type: enhancement Stage: resolved
Components: Windows Versions: Python 3.6
process
Status: closed Resolution: duplicate
Dependencies: Superseder: Make pyvenv style virtual environments easier to configure when embedding Python
View: 22213
Assigned To: Nosy List: Dieter Weber, ncoghlan, paul.moore, pitrou, pyscripter, steve.dower, tds333, tim.golden, vstinner, zach.ware
Priority: normal Keywords:

Created on 2019-01-10 14:04 by Dieter Weber, last changed 2019-02-13 17:44 by steve.dower. This issue is now closed.

Files
File name Uploaded Description Edit
Source.cpp Dieter Weber, 2019-01-10 14:10
Messages (11)
msg333378 - (view) Author: Dieter Weber (Dieter Weber) * Date: 2019-01-10 14:04
Python virtual environments are awesome! Using venvs with an embedded Python interpreter has proven difficult, unfortunately. With conda environments it works. See appended a sample file to reproduce the behavior.

The core of the problem seems to be that a venv doesn't contain a full Python installation, and Py_Initialize() apparently doesn't support setting up the combination of venv directories and base installation correctly, i.e. setting sys.prefix and sys.base_prefix and potentially other values.

Observed behavior when trying to use a venv:

"""
Initializing... Fatal Python error: Py_Initialize: unable to load the file system codec
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00001e90 (most recent call first):
"""

Expected behavior:
Setting Py_SetPythonHome() to a venv works and sets up all paths and prefixes correctly to use the venv, just like it does for a conda environment.
msg334745 - (view) Author: PyScripter (pyscripter) Date: 2019-02-02 12:25
Similar experience.  I have found no way to get to use a venv as embedded Python in PyScripter.  Tried Py_SetPythonhome, Py_SetPythonName, combinations... Nothing worked.
msg334790 - (view) Author: Wolfgang Langner (tds333) * Date: 2019-02-03 12:46
I think this pep is related to the issue and could be a solution:
https://www.python.org/dev/peps/pep-0582/
msg334792 - (view) Author: PyScripter (pyscripter) Date: 2019-02-03 14:30
I don't see how pep-0582 is related to embedded python.

The solution is to check for the presence of pyvenv.cfg in PYTHONHOME and set up the system.path accordingly.
msg334815 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-02-04 07:33
Py_Initialize() is not supposed to set it up. If you are embedding Python, you have to be responsible for configuring the paths that it uses - pyvenv.cfg is configuration for Python interactive, not the shared lib (using Windows terminology, it's configuration for python.exe but not python36.dll).

Personally, I am 100% against letting an embedded runtime automatically pick up any settings from the environment. You don't know where they came from. That said, I'm also in favor of making it easy for embedders to imitate the interactive shell's configuration.

Victor (nosied) and others are in the long process of updating embedded interpreter configuration to be easier to use, so adding him here as feedback, but I don't believe there is going to be a specific fix for this. Consider calling Py_SetPath before Py_Initialize to configure the search paths you want to use.
msg334831 - (view) Author: PyScripter (pyscripter) Date: 2019-02-04 17:01
"Personally, I am 100% against letting an embedded runtime automatically pick up any settings from the environment. You don't know where they came from."

Why is that?  When the embedding application is using Py_SetPythonHome there is clear intent to use a specific python setup.  This is no different to using any other Python setup.  Why venvs should be treated differently that standard installations?

As to manually setting the path, why pass the burden to the embedding application.  The idea is that python, in the embedding application, should work as in the python executable.   Trying to match the complex and evolving algorithms that python uses to setup the path, is a huge and unnecessary burden on the embedding application.
msg334832 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-02-04 17:29
> The idea is that python, in the embedding application, should work as in the python executable.

I disagree that this is the idea. Python in the embedding application should work as in the embedding application, not the standalone one. The embedding application should be able to easily opt-in to "standalone-like configuration", which is also what I said above, but right now that doesn't exist (it's under development).

Now, I'd be very much in favour of being able to embed Python without having to load any encodings. That ought to get us to the point where you can call "site.main()" in your embedding app, which does most of the work of what you consider the "default" behavior. But right now you just have to provide enough search path to initialize before doing that.
msg334834 - (view) Author: PyScripter (pyscripter) Date: 2019-02-04 17:47
This issue is related (duplicate) to https://bugs.python.org/issue22213.
It appears that there is a workaround but only for Linux using Py_SetProgramName.  I wish that there was some similar way for Windows.
msg335466 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-02-13 17:13
PEP 432 will allow to give with fine control on parameters used to initialize Python. Sadly, I failed to agree with Nick Coghlan and Eric Snow on the API. The current implementation (_PyCoreConfig and _PyMainInterpreterConfig) has some flaw (don't separate clearly the early initialization and Unicode-ready state, the interpreter contains main and core config whereas some options are duplicated in both configs, etc.).
msg335467 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-02-13 17:14
See also bpo-22213.
msg335469 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-02-13 17:44
That's not a "see also" - it's just a duplicate :) I'll close this one as the other one has been around longer.
History
Date User Action Args
2019-02-13 17:44:49steve.dowersetstatus: open -> closed
superseder: Make pyvenv style virtual environments easier to configure when embedding Python
messages: + msg335469

resolution: duplicate
stage: resolved
2019-02-13 17:14:26vstinnersetmessages: + msg335467
2019-02-13 17:13:32vstinnersetmessages: + msg335466
2019-02-04 17:47:39pyscriptersetnosy: + ncoghlan, pitrou
messages: + msg334834
2019-02-04 17:29:41steve.dowersetmessages: + msg334832
2019-02-04 17:01:26pyscriptersetmessages: + msg334831
2019-02-04 07:33:01steve.dowersetnosy: + vstinner
messages: + msg334815
2019-02-03 19:18:19ned.deilysetnosy: + zach.ware, paul.moore, tim.golden, steve.dower
components: + Windows
2019-02-03 14:30:04pyscriptersetmessages: + msg334792
2019-02-03 12:46:46tds333setnosy: + tds333
messages: + msg334790
2019-02-02 12:25:33pyscriptersetnosy: + pyscripter
messages: + msg334745
2019-01-10 14:12:27Dieter Webersettitle: Making an embedded Python interpreter use a venv is difficult -> Make it easier to use a venv with an embedded Python interpreter
2019-01-10 14:10:13Dieter Webersetfiles: - Source.cpp
2019-01-10 14:10:02Dieter Webersetfiles: + Source.cpp
2019-01-10 14:04:59Dieter Webercreate