Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add python.ini file for embedded/applocal installs #68143

Closed
zooba opened this issue Apr 14, 2015 · 18 comments
Closed

Add python.ini file for embedded/applocal installs #68143

zooba opened this issue Apr 14, 2015 · 18 comments
Assignees
Labels
interpreter-core (Objects, Python, Grammar, and Parser dirs) OS-windows type-feature A feature request or enhancement

Comments

@zooba
Copy link
Member

zooba commented Apr 14, 2015

BPO 23955
Nosy @pfmoore, @ronaldoussoren, @ncoghlan, @tjguk, @ned-deily, @merwok, @takluyver, @zware, @zooba
Files
  • 23955_1.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/zooba'
    closed_at = <Date 2015-05-22.22:21:14.326>
    created_at = <Date 2015-04-14.19:16:30.917>
    labels = ['interpreter-core', 'type-feature', 'OS-windows']
    title = 'Add python.ini file for embedded/applocal installs'
    updated_at = <Date 2015-05-22.22:21:14.325>
    user = 'https://github.com/zooba'

    bugs.python.org fields:

    activity = <Date 2015-05-22.22:21:14.325>
    actor = 'steve.dower'
    assignee = 'steve.dower'
    closed = True
    closed_date = <Date 2015-05-22.22:21:14.326>
    closer = 'steve.dower'
    components = ['Interpreter Core', 'Windows']
    creation = <Date 2015-04-14.19:16:30.917>
    creator = 'steve.dower'
    dependencies = []
    files = ['39276']
    hgrepos = []
    issue_num = 23955
    keywords = ['patch']
    message_count = 18.0
    messages = ['240989', '241343', '241422', '241424', '241670', '242002', '242020', '242038', '242042', '242044', '242100', '242125', '242127', '242450', '242589', '242865', '243857', '243859']
    nosy_count = 11.0
    nosy_names = ['richard', 'paul.moore', 'ronaldoussoren', 'ncoghlan', 'tim.golden', 'ned.deily', 'eric.araujo', 'python-dev', 'takluyver', 'zach.ware', 'steve.dower']
    pr_nums = []
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue23955'
    versions = ['Python 3.5']

    @zooba
    Copy link
    Member Author

    zooba commented Apr 14, 2015

    When Python starts running we (getpathp.c) looks in the registry for its library directory. To avoid this, you can set PYTHONPATH and PYTHONHOME environment variables, but this isn't always easy if you've copied Python into your app's directory.

    We should look for a python35.ini alongside python35.dll when initializing and extract values from it. Possibly this could just be an [Environment] section with key-value pairs to set before loading.

    This would ensure that app-local Python installations never look for a global install in the registry. (As an aside, for the 3.5 beta we'll be shipping an app-local "installer", which will make this situation easier to end up in, but it also has value for xcopy installs.)

    @zooba zooba self-assigned this Apr 14, 2015
    @zooba zooba added interpreter-core (Objects, Python, Grammar, and Parser dirs) OS-windows labels Apr 14, 2015
    @merwok
    Copy link
    Member

    merwok commented Apr 17, 2015

    Hm, could you reuse the venv config file with use-site-packages=true?

    @zooba
    Copy link
    Member Author

    zooba commented Apr 18, 2015

    I could reuse the venv config file but it would need a new property to forcibly use the 'home' value regardless of whether '$home\Lib\os.py' exists (in case the stdlib is zipped) and to make it relative to the file's path rather than the user's current directory.

    How about adding an 'applocal=True' property to pyvenv.cfg that basically implies -I, PYTHONHOME=os.dirname(pyvenv_cfg), and skips the registry lookup on Windows completely (which there's currently no way to achieve)?

    @zooba
    Copy link
    Member Author

    zooba commented Apr 18, 2015

    Arguably we should be making 'home' in pyvenv.cfg relative to the pyvenv.cfg file's directory anyway... AFAIK it's always generated as an absolute path, but I see no good reason to prevent people from manually configuring a relative path here.

    @takluyver
    Copy link
    Mannequin

    takluyver mannequin commented Apr 20, 2015

    Relative paths would be nice for Pynsist - I would prefer to create the config file at build time and then install it along with the other files. If it needed absolute paths, then the installer would have to write the config file after the user selects the installation directory. That's doable, but it's easier to write and test the build steps than the install steps.

    @ncoghlan
    Copy link
    Contributor

    I'd actually like a python.ini file defined as a Python 3.5 feature in order to provide a better backporting story for the PEP-476 certificate changes in downstream long term support releases of Python 2.7.

    So +1 in principle, but I'd like to see this written up as a PEP (the SSL configuration setting doesn't need to be in the PEP, but the optional existence of the config file and reading it at startup does).

    @pfmoore
    Copy link
    Member

    pfmoore commented Apr 25, 2015

    As described. this seems to be a Windows-only feature (it's replacing a registry lookup with a config file alongside the Python DLL).

    So I'm not sure I understand Nick's comment - Nick, are you suggesting this should be a cross-platform feature? If so, what's the equivalent of "alongside the Python DLL", and what would the ini file be used for on Unix, where there's no registry lookup to override?

    @ncoghlan
    Copy link
    Contributor

    If we're adding a static startup configuration file, then yes, I believe that should be a cross-platform feature, not something Windows specific, as it's a major change to our configuration philosophy. (I think it's a *necessary* change, but it's also a significant one)

    Even if it started out as Windows only, we should have a forward looking plan for how it relates to the startup sequence changes proposed in PEP-432.

    @zooba
    Copy link
    Member Author

    zooba commented Apr 26, 2015

    I'm leaning towards clarifying/fixing the behavior of the venv config, since it's most of the way there for what I care about. Adding another option for applocal may be needed, but possibly not, in which case defining how relative paths are handled would be sufficient.

    Any other startup file I'd want to define as setting environment variables only, to avoid having multiple ways to set properties. So for most cases a shell script would suffice (doesn't help people linking to libpython directly, but they can call SetPythonHome themselves).

    @ncoghlan
    Copy link
    Contributor

    Treating "local Python" as a special-case of venv sounds like a good approach, then (especially as it currently seems plausible we'll end up going for environment variable based approach to the downstream PEP-476 backport)

    @zooba
    Copy link
    Member Author

    zooba commented Apr 27, 2015

    Having looked at the implementation, the easiest way to do this will be to add an "applocal = true" option to pyvenv.cfg, which would behave identically to setting %PYTHONHOME% to the directory containing the config before launch.

    I wanted to support relative paths for "home = " but the path handling is so far from sufficient that it wouldn't be secure. As it is, I've fixed a lot of potential buffer overruns in getpathp.c already, but I don't want to implement true relative paths, so I'd prefer to require "home = " to be absolute and add a second property that effectively means "home = .".

    Any thoughts/comments/preferences?

    @takluyver
    Copy link
    Mannequin

    takluyver mannequin commented Apr 27, 2015

    Would that option be the only thing that needs to be set to make Python app-local? I'm not familiar with what lives in pyvenv.cfg.

    @zooba
    Copy link
    Member Author

    zooba commented Apr 27, 2015

    None of the other options really have much use in this case, since they're mostly about how to combine multiple environments rather than excluding implicit ones.

    Setting PYTHONHOME to the current directory certainly seems to be enough AFAICT, at least for Windows, as that bypasses the most problematic registry lookup. Having an "applocal" option means we can go further and ignore all registry keys and environment variables.

    I haven't looked into how to duplicate the behaviour on other OSs, but Modules/getpath.c looks similar enough to PC/getpathp.c that I guess it'll be similar. Adding the MacOS folks in case they have suggestions/concerns.

    @zooba
    Copy link
    Member Author

    zooba commented May 3, 2015

    Added a patch that:

    • adds the applocal option to pyvenv.cfg
    • updates the docs to describe how pyvenv.cfg (home and applocal) affect sys.path
    • puts a default pyvenv.cfg in the ZIP distribution
    • adds significantly more buffer overflow protection to getpathp.c
    • prevents a case where sys.path may include multiple blank entries

    Currently none of this applies to Linux or Mac builds, though I'm led to believe there would be some value if it did. I'm not the one who can figure all the intricacies out for those platforms though.

    @ncoghlan
    Copy link
    Contributor

    ncoghlan commented May 5, 2015

    bpo-23857 was the one where I thought there might be an overlap with a design consideration on Linux (related to coming up with a conventional for backporting PEP-476 as a CPython redistributor).

    However, I've now suggested a different path forward there, closer to what happened with PEP-394 (which provides guidance to redistributors on symlink definitions, without making any changes to upstream CPython itself).

    @zooba
    Copy link
    Member Author

    zooba commented May 10, 2015

    Any further comments/bikeshedding on the applocal parameter? I still need to do some docs, but I want the basic design agreed upon first.

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented May 22, 2015

    New changeset 620a247b4960 by Steve Dower in branch 'default':
    Issue bpo-23955: Add pyvenv.cfg option to suppress registry/environment lookup for generating sys.path.
    https://hg.python.org/cpython/rev/620a247b4960

    @zooba
    Copy link
    Member Author

    zooba commented May 22, 2015

    The support is in and the option remains named "applocal".

    Hopefully this (and the ZIP file version, which will include the preconfigured pyvenv.cfg) will help with app distributions, though I'm still open to make changes if it doesn't.

    @zooba zooba closed this as completed May 22, 2015
    @zooba zooba added the type-feature A feature request or enhancement label May 22, 2015
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    interpreter-core (Objects, Python, Grammar, and Parser dirs) OS-windows type-feature A feature request or enhancement
    Projects
    None yet
    Development

    No branches or pull requests

    4 participants