classification
Title: Add python.ini file for embedded/applocal installs
Type: enhancement Stage: resolved
Components: Interpreter Core, Windows Versions: Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: steve.dower Nosy List: eric.araujo, ncoghlan, ned.deily, paul.moore, python-dev, richard, ronaldoussoren, steve.dower, takluyver, tim.golden, zach.ware
Priority: normal Keywords: patch

Created on 2015-04-14 19:16 by steve.dower, last changed 2015-05-22 22:21 by steve.dower. This issue is now closed.

Files
File name Uploaded Description Edit
23955_1.patch steve.dower, 2015-05-03 05:57 review
Messages (18)
msg240989 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-04-14 19:16
When Python starts running we (getpathp.c) looks in the registry for its library directory. To avoid this, you can set PYTHONPATH and PYTHONHOME environment variables, but this isn't always easy if you've copied Python into your app's directory.

We should look for a python35.ini alongside python35.dll when initializing and extract values from it. Possibly this could just be an [Environment] section with key-value pairs to set before loading.

This would ensure that app-local Python installations never look for a global install in the registry. (As an aside, for the 3.5 beta we'll be shipping an app-local "installer", which will make this situation easier to end up in, but it also has value for xcopy installs.)
msg241343 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2015-04-17 16:29
Hm, could you reuse the venv config file with use-site-packages=true?
msg241422 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-04-18 16:19
I could reuse the venv config file but it would need a new property to forcibly use the 'home' value regardless of whether '$home\Lib\os.py' exists (in case the stdlib is zipped) and to make it relative to the file's path rather than the user's current directory.

How about adding an 'applocal=True' property to pyvenv.cfg that basically implies -I, PYTHONHOME=os.dirname(pyvenv_cfg), and skips the registry lookup on Windows completely (which there's currently no way to achieve)?
msg241424 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-04-18 16:27
Arguably we should be making 'home' in pyvenv.cfg relative to the pyvenv.cfg file's directory anyway... AFAIK it's always generated as an absolute path, but I see no good reason to prevent people from manually configuring a relative path here.
msg241670 - (view) Author: Thomas Kluyver (takluyver) * Date: 2015-04-20 17:44
Relative paths would be nice for Pynsist - I would prefer to create the config file at build time and then install it along with the other files. If it needed absolute paths, then the installer would have to write the config file after the user selects the installation directory. That's doable, but it's easier to write and test the build steps than the install steps.
msg242002 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2015-04-25 07:41
I'd actually like a python.ini file defined as a Python 3.5 feature in order to provide a better backporting story for the PEP 476 certificate changes in downstream long term support releases of Python 2.7.

So +1 in principle, but I'd like to see this written up as a PEP (the SSL configuration setting doesn't need to be in the PEP, but the optional existence of the config file and reading it at startup does).
msg242020 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2015-04-25 18:25
As described. this seems to be a Windows-only feature (it's replacing a registry lookup with a config file alongside the Python DLL).

So I'm not sure I understand Nick's comment - Nick, are you suggesting this should be a cross-platform feature? If so, what's the equivalent of "alongside the Python DLL", and what would the ini file be used for on Unix, where there's no registry lookup to override?
msg242038 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2015-04-26 02:35
If we're adding a static startup configuration file, then yes, I believe that should be a cross-platform feature, not something Windows specific, as it's a major change to our configuration philosophy. (I think it's a *necessary* change, but it's also a significant one)

Even if it started out as Windows only, we should have a forward looking plan for how it relates to the startup sequence changes proposed in PEP 432.
msg242042 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-04-26 04:35
I'm leaning towards clarifying/fixing the behavior of the venv config, since it's most of the way there for what I care about. Adding another option for applocal may be needed, but possibly not, in which case defining how relative paths are handled would be sufficient.

Any other startup file I'd want to define as setting environment variables only, to avoid having multiple ways to set properties. So for most cases a shell script would suffice (doesn't help people linking to libpython directly, but they can call SetPythonHome themselves).
msg242044 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2015-04-26 05:12
Treating "local Python" as a special-case of venv sounds like a good approach, then (especially as it currently seems plausible we'll end up going for environment variable based approach to the downstream PEP 476 backport)
msg242100 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-04-27 06:07
Having looked at the implementation, the easiest way to do this will be to add an "applocal = true" option to pyvenv.cfg, which would behave identically to setting %PYTHONHOME% to the directory containing the config before launch.

I wanted to support relative paths for "home = " but the path handling is so far from sufficient that it wouldn't be secure. As it is, I've fixed a lot of potential buffer overruns in getpathp.c already, but I don't want to implement true relative paths, so I'd prefer to require "home = " to be absolute and add a second property that effectively means "home = .".

Any thoughts/comments/preferences?
msg242125 - (view) Author: Thomas Kluyver (takluyver) * Date: 2015-04-27 16:47
Would that option be the only thing that needs to be set to make Python app-local? I'm not familiar with what lives in pyvenv.cfg.
msg242127 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-04-27 16:57
None of the other options really have much use in this case, since they're mostly about how to combine multiple environments rather than excluding implicit ones.

Setting PYTHONHOME to the current directory certainly seems to be enough AFAICT, at least for Windows, as that bypasses the most problematic registry lookup. Having an "applocal" option means we can go further and ignore all registry keys and environment variables.

I haven't looked into how to duplicate the behaviour on other OSs, but Modules/getpath.c looks similar enough to PC/getpathp.c that I guess it'll be similar. Adding the MacOS folks in case they have suggestions/concerns.
msg242450 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-05-03 05:57
Added a patch that:

* adds the applocal option to pyvenv.cfg
* updates the docs to describe how pyvenv.cfg (home and applocal) affect sys.path
* puts a default pyvenv.cfg in the ZIP distribution
* adds significantly more buffer overflow protection to getpathp.c
* prevents a case where sys.path may include multiple blank entries

Currently none of this applies to Linux or Mac builds, though I'm led to believe there would be some value if it did. I'm not the one who can figure all the intricacies out for those platforms though.
msg242589 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2015-05-05 08:02
Issue 23857 was the one where I thought there might be an overlap with a design consideration on Linux (related to coming up with a conventional for backporting PEP 476 as a CPython redistributor).

However, I've now suggested a different path forward there, closer to what happened with PEP 394 (which provides guidance to redistributors on symlink definitions, without making any changes to upstream CPython itself).
msg242865 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-05-10 20:48
Any further comments/bikeshedding on the applocal parameter? I still need to do some docs, but I want the basic design agreed upon first.
msg243857 - (view) Author: Roundup Robot (python-dev) Date: 2015-05-22 22:16
New changeset 620a247b4960 by Steve Dower in branch 'default':
Issue #23955: Add pyvenv.cfg option to suppress registry/environment lookup for generating sys.path.
https://hg.python.org/cpython/rev/620a247b4960
msg243859 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-05-22 22:21
The support is in and the option remains named "applocal".

Hopefully this (and the ZIP file version, which will include the preconfigured pyvenv.cfg) will help with app distributions, though I'm still open to make changes if it doesn't.
History
Date User Action Args
2015-05-22 22:21:14steve.dowersetstatus: open -> closed
type: enhancement
messages: + msg243859

resolution: fixed
stage: resolved
2015-05-22 22:16:52python-devsetnosy: + python-dev
messages: + msg243857
2015-05-10 20:48:08steve.dowersetmessages: + msg242865
2015-05-05 08:02:19ncoghlansetmessages: + msg242589
2015-05-03 05:57:46steve.dowersetfiles: + 23955_1.patch
keywords: + patch
messages: + msg242450
2015-04-27 16:57:32steve.dowersetnosy: + ronaldoussoren, ned.deily
messages: + msg242127
2015-04-27 16:47:30takluyversetmessages: + msg242125
2015-04-27 06:08:00steve.dowersetmessages: + msg242100
2015-04-26 05:12:44ncoghlansetmessages: + msg242044
2015-04-26 04:35:20steve.dowersetmessages: + msg242042
2015-04-26 02:35:20ncoghlansetmessages: + msg242038
2015-04-25 18:25:36paul.mooresetnosy: + paul.moore
messages: + msg242020
2015-04-25 07:41:31ncoghlansetmessages: + msg242002
2015-04-20 17:44:34takluyversetmessages: + msg241670
2015-04-20 16:59:38takluyversetnosy: + takluyver
2015-04-18 16:27:57steve.dowersetmessages: + msg241424
2015-04-18 16:19:44steve.dowersetmessages: + msg241422
2015-04-17 16:29:03eric.araujosetnosy: + eric.araujo
messages: + msg241343
2015-04-14 19:16:30steve.dowercreate