Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

site.main() does not work on Python 3.6 and superior if PYTHONSTARTUP is set #74353

Closed
anjos mannequin opened this issue Apr 26, 2017 · 11 comments
Closed

site.main() does not work on Python 3.6 and superior if PYTHONSTARTUP is set #74353

anjos mannequin opened this issue Apr 26, 2017 · 11 comments
Labels
3.7 (EOL) end of life 3.8 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@anjos
Copy link
Mannequin

anjos mannequin commented Apr 26, 2017

BPO 30167
Nosy @ned-deily, @methane, @anjos, @steverweber
PRs
  • bpo-30167: site: Ignore TypeError in abs_paths() #6731
  • bpo-30167: Remove __cached__ when removing __file__ #7415
  • [3.7] bpo-30167: Prevent site.main() exception if PYTHONSTARTUP is set. (GH-6731) #7606
  • [3.6] bpo-30167: Prevent site.main() exception if PYTHONSTARTUP is set. (GH-6731) #7607
  • bpo-30167: Add test for module.__cached__ is None #7617
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2018-11-30.11:32:45.168>
created_at = <Date 2017-04-26.08:53:33.616>
labels = ['3.7', '3.8', 'type-bug', 'library']
title = 'site.main() does not work on Python 3.6 and superior if PYTHONSTARTUP is set'
updated_at = <Date 2018-11-30.11:32:45.167>
user = 'https://github.com/anjos'

    bugs.python.org fields:

    activity = <Date 2018-11-30.11:32:45.167>
actor = 'methane'
assignee = 'none'
closed = True
closed_date = <Date 2018-11-30.11:32:45.168>
closer = 'methane'
components = ['Library (Lib)']
creation = <Date 2017-04-26.08:53:33.616>
creator = 'anjos'
dependencies = []
files = []
hgrepos = []
issue_num = 30167
keywords = ['patch']
message_count = 11.0
messages = ['292325', '292327', '292329', '316295', '318725', '319258', '319260', '319261', '319262', '326797', '330677']
nosy_count = 4.0
nosy_names = ['ned.deily', 'methane', 'anjos', 'steverweber']
pr_nums = ['6731', '7415', '7606', '7607', '7617']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue30167'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']
    @anjos
    Copy link
    Mannequin Author

    anjos mannequin commented Apr 26, 2017

    Apparently, "import site; site.main()" does not seem to work anymore on Python 3.6 and superior.

    The reason is a change on the behavior of "os.path.abspath(None)". Before Python 3.6, it used to report an AttributeError which is properly caught inside "site.abs_paths" (see: https://github.com/python/cpython/blob/master/Lib/site.py#L99), making it ignore __main__, one of sys.modules, which has __file__ and __cached__ set to None.

    With Python 3.6 and superior, os.path.abspath(None) reports a TypeError, which makes calling "site.main()" raise an exception and stop.

    How to reproduce: On python 3.6 or superior, do "import site; site.main()".

    Expected behavior: Exception is properly caught and treated inside "site.abs_paths", ignoring modules in which __file__ and/or __cached__ are set to None.

    @anjos anjos mannequin added 3.7 (EOL) end of life stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error labels Apr 26, 2017
    @anjos
    Copy link
    Mannequin Author

    anjos mannequin commented Apr 26, 2017

    More information: to reproduce the problem, don't use Python's "-c" command-line option. In this case "__main__" won't be inside sys.modules which mitigates the issue.

    @anjos
    Copy link
    Mannequin Author

    anjos mannequin commented Apr 26, 2017

    After further investigation, the issue can only be reproduced iff the user sets PYTHONSTARTUP which triggers "__main__" to appear in sys.modules and the problem to occur.

    @anjos anjos mannequin changed the title site.main() does not work on Python 3.6 and superior site.main() does not work on Python 3.6 and superior with PYTHONSTARTUP is set Apr 26, 2017
    @anjos anjos mannequin changed the title site.main() does not work on Python 3.6 and superior with PYTHONSTARTUP is set site.main() does not work on Python 3.6 and superior if PYTHONSTARTUP is set Apr 26, 2017
    @steverweber
    Copy link
    Mannequin

    steverweber mannequin commented May 8, 2018

    ref to other projects that hit this issue.

    https://gitlab.idiap.ch/bob/bob.buildout/issues/15
    saltstack/salt#45080
    https://groups.google.com/forum/#!msg/comp.lang.python/tpfiHAJhl9Y/hZj1f50vBQAJ

    @methane
    Copy link
    Member

    methane commented Jun 5, 2018

    cpython/Python/pythonrun.c

    Lines 391 to 399 in f822549

    if (PyDict_SetItemString(d, "__file__", f) < 0) {
    Py_DECREF(f);
    goto done;
    }
    if (PyDict_SetItemString(d, "__cached__", Py_None) < 0) {
    Py_DECREF(f);
    goto done;
    }
    set_file_name = 1;

    __file__ and __cached__ are added here. And,

    cpython/Python/pythonrun.c

    Lines 441 to 442 in f822549

    if (set_file_name && PyDict_DelItemString(d, "__file__"))
    PyErr_Clear();

    Only __file__ is removed here. I feel __cached__ should be removed too.

    On the other hand, if None is valid value for __cached__, site.abs_paths() should ignore it.

    https://www.python.org/dev/peps/pep-3147/#file

    It is recommended that when nothing sensible can be calculated, implementations should set the __cached__ attribute to None.

    It seems it's valid, and recommended for some cases. So PR-6731 looks OK.

    @ned-deily
    Copy link
    Member

    New changeset 2487f30 by Ned Deily (Steve Weber) in branch 'master':
    bpo-30167: Prevent site.main() exception if PYTHONSTARTUP is set. (GH-6731)
    2487f30

    @ned-deily
    Copy link
    Member

    New changeset ec4343c by Ned Deily (Miss Islington (bot)) in branch '3.7':
    bpo-30167: Prevent site.main() exception if PYTHONSTARTUP is set. (GH-6731) (GH-7606)
    ec4343c

    @ned-deily
    Copy link
    Member

    New changeset 3e12158 by Ned Deily (Miss Islington (bot)) in branch '3.6':
    bpo-30167: Prevent site.main() exception if PYTHONSTARTUP is set. (GH-6731) (GH-7607)
    3e12158

    @ned-deily
    Copy link
    Member

    As I noted in the discussion on PR 6731, I think there should be a test for this so we don't break it again. But, since it seems that the problem has affected a number of users and projects and since the fix is easy and easily testable manually, I decided to merge the PR for 3.7.0rc1 and 3.6.6rc1. I am leaving the issue open for someone to supply a PR with a test case and for discussion of PR 7415. Thank you all!

    @ned-deily ned-deily added the 3.8 only security fixes label Jun 11, 2018
    @methane
    Copy link
    Member

    methane commented Oct 1, 2018

    New changeset d4c76d9 by INADA Naoki in branch 'master':
    bpo-30167: Add test for module.__cached__ is None (GH-7617)
    d4c76d9

    @methane
    Copy link
    Member

    methane commented Nov 29, 2018

    New changeset 82daa60 by INADA Naoki in branch 'master':
    bpo-30167: Remove __cached__ from __main__ when removing __file__ (GH-7415)
    82daa60

    @methane methane closed this as completed Nov 30, 2018
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    3.7 (EOL) end of life 3.8 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    2 participants
    @methane @ned-deily