Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Py_Initialize(): computing path configuration must not have side effect (PEP 432) #78351

Closed
vstinner opened this issue Jul 20, 2018 · 27 comments
Closed
Labels
3.8 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs)

Comments

@vstinner
Copy link
Member

BPO 34170
Nosy @vstinner
PRs
  • bpo-34170: Rework _PyCoreConfig_Read() to avoid side effect #8353
  • bpo-34170: _PyCoreConfig_Read() leaves Py_IsolatedFlag unchanged #8361
  • bpo-34170: Add _PyCoreConfig.isolated #8417
  • bpo-34170: Add _PyCoreConfig.bytes_warning #8447
  • bpo-34170: Add _Py_InitializeFromConfig() #8454
  • bpo-34170: Cleanup pymain_run_python() #8455
  • bpo-34170: Enhance _PyCoreConfig_Read() #8468
  • bpo-34170: Cleanup pymain_read_conf() #8476
  • bpo-34170, test_embed: write Py_Initialize() tests #8484
  • bpo-34247: actually use environment variables in Py_Initialize #8521
  • bpo-34170: Add _PyCoreConfig._frozen parameter #8591
  • bpo-34170: Py_Main() updates config on setting Py_InspectFlag #8593
  • bpo-34170: Rename _PyCoreConfig.unbuffered_stdio #8594
  • bpo-34170: _PyCoreConfig_Read() defaults to argc=0 #8595
  • bpo-34170: Add Python/coreconfig.c for _PyCoreConfig #8607
  • bpo-31650: Remove _Py_CheckHashBasedPycsMode global config var #8608
  • bpo-34170: Cleanup pymain_run_filename() #8631
  • bpo-34170: _PyCoreConfig_Read() don't replace coerce_c_locale #8658
  • bpo-34170: Fix pymain_run_file() #8660
  • bpo-34207: Fix pymain_init_cmdline_argv() #8867
  • bpo-34207: Fix pymain_read_conf() for UTF-8 Mode #8868
  • [3.7] bpo-34207: Fix pymain_read_conf() for UTF-8 Mode (GH-8868) #8870
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2018-08-23.10:44:58.712>
    created_at = <Date 2018-07-20.15:22:57.855>
    labels = ['interpreter-core', '3.8']
    title = 'Py_Initialize(): computing path configuration must not have side effect (PEP 432)'
    updated_at = <Date 2018-08-23.10:44:58.711>
    user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2018-08-23.10:44:58.711>
    actor = 'vstinner'
    assignee = 'none'
    closed = True
    closed_date = <Date 2018-08-23.10:44:58.712>
    closer = 'vstinner'
    components = ['Interpreter Core']
    creation = <Date 2018-07-20.15:22:57.855>
    creator = 'vstinner'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 34170
    keywords = ['patch']
    message_count = 27.0
    messages = ['322015', '322018', '322020', '322040', '322060', '322290', '322297', '322336', '322337', '322340', '322366', '322371', '322393', '322427', '322440', '322828', '322829', '322830', '322831', '322868', '322870', '322871', '322982', '323073', '323086', '323936', '323938']
    nosy_count = 1.0
    nosy_names = ['vstinner']
    pr_nums = ['8353', '8361', '8417', '8447', '8454', '8455', '8468', '8476', '8484', '8521', '8591', '8593', '8594', '8595', '8607', '8608', '8631', '8658', '8660', '8867', '8868', '8870']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = None
    url = 'https://bugs.python.org/issue34170'
    versions = ['Python 3.8']

    @vstinner vstinner added 3.8 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) labels Jul 20, 2018
    @vstinner
    Copy link
    Member Author

    Attached PR reworks _PyCoreConfig_Read() to leave _Py_path_config unchanged: *reading* the core configuration must not *modify* the path configuration.

    The PR adds _PyCoreConfig.dll_path field.

    The change is related to the PEP-432.

    @vstinner
    Copy link
    Member Author

    See also bpo-34008: "Do we support calling Py_Main() after Py_Initialize()?".

    @vstinner
    Copy link
    Member Author

    PR 8353 is the last piece of my long work on Py_Main() to separate code *reading* configuration and code *applying* a new configuration. My work on the new _PyPathConfig structure, and the PR should finish that work.

    Extract of the commit message:

    "A new _PyCoreConfig_SetPathConfig() function now recreates the path configuration from the core configuration. This function is now called very late in _Py_InitializeCore(), just before calling initimport()."

    @vstinner
    Copy link
    Member Author

    New changeset b1147e4 by Victor Stinner in branch 'master':
    bpo-34170: Rework _PyCoreConfig_Read() to avoid side effect (GH-8353)
    b1147e4

    @vstinner
    Copy link
    Member Author

    New changeset f2626ce by Victor Stinner in branch 'master':
    bpo-34170: _PyCoreConfig_Read() leaves Py_IsolatedFlag unchanged (GH-8361)
    f2626ce

    @vstinner
    Copy link
    Member Author

    New changeset d19d8d5 by Victor Stinner in branch 'master':
    bpo-34170: Add _PyCoreConfig.isolated (GH-8417)
    d19d8d5

    @vstinner
    Copy link
    Member Author

    Reminder for myself: I have to write unit tests :-)
    #8417 (review)

    @vstinner
    Copy link
    Member Author

    New changeset 1dc6e39 by Victor Stinner in branch 'master':
    bpo-34170: Add _Py_InitializeFromConfig() (GH-8454)
    1dc6e39

    @vstinner
    Copy link
    Member Author

    I added many fields to _PyCoreConfig which duplicate global configuration varibles: _PyConfigCore.isolated duplicates Py_IsolatedFlag.

    I started to modify Python to read the core configuration rather than global configuration flags. The problem is that sometimes, global configuration flags are updated, but not their duplicated core configuration fields.

    Example from Modules/main.c:

    static void
    pymain_repl(_PyMain *pymain, _PyCoreConfig *config, PyCompilerFlags *cf)
    {
        /* Check this environment variable at the end, to give programs the
           opportunity to set it from Python. */
        if (!Py_InspectFlag && config_get_env_var(config, "PYTHONINSPECT")) {
            Py_InspectFlag = 1;
        }
        ...
    }

    Only Py_InspectFlag is not, not core_config.inspect.

    @vstinner
    Copy link
    Member Author

    New changeset d3b1919 by Victor Stinner in branch 'master':
    bpo-34170: Cleanup pymain_run_python() (GH-8455)
    d3b1919

    @vstinner
    Copy link
    Member Author

    TODO list, open questions:

    • Should isolated=1 always imply config->ignore_environment=1 and config->user_site_directory = 0?
    • Should we rename debug to parser_debug? "debug" is a very generic term, whereas this flag is really specific to the parser
    • Remove _Py_CheckHashBasedPycsMode: replace it with config._check_hash_pycs_mode?

    @vstinner
    Copy link
    Member Author

    I created bpo-34228 "Allow PYTHONTRACEMALLOC=0 and -X tracemalloc=0 to disable explicitly tracemalloc" that I need to write unit tests on Python initiallization.

    @vstinner
    Copy link
    Member Author

    New changeset ecf411c by Victor Stinner in branch 'master':
    bpo-34170: Enhance _PyCoreConfig_Read() (GH-8468)
    ecf411c

    @vstinner
    Copy link
    Member Author

    New changeset d145775 by Victor Stinner in branch 'master':
    bpo-34170: Cleanup pymain_read_conf() (GH-8476)
    d145775

    @vstinner
    Copy link
    Member Author

    New changeset 56b29b6 by Victor Stinner in branch 'master':
    bpo-34170, test_embed: write Py_Initialize() tests (GH-8484)
    56b29b6

    @vstinner
    Copy link
    Member Author

    vstinner commented Aug 1, 2018

    New changeset b75d7e2 by Victor Stinner in branch 'master':
    bpo-34170: Add _PyCoreConfig._frozen parameter (GH-8591)
    b75d7e2

    @vstinner
    Copy link
    Member Author

    vstinner commented Aug 1, 2018

    New changeset a4d20b2 by Victor Stinner in branch 'master':
    bpo-34170: Py_Main() updates config when setting Py_InspectFlag (GH-8593)
    a4d20b2

    @vstinner
    Copy link
    Member Author

    vstinner commented Aug 1, 2018

    New changeset 9851227 by Victor Stinner in branch 'master':
    bpo-34170: Rename _PyCoreConfig.unbuffered_stdip (GH-8594)
    9851227

    @vstinner
    Copy link
    Member Author

    vstinner commented Aug 1, 2018

    New changeset ea68d83 by Victor Stinner in branch 'master':
    bpo-34170: _PyCoreConfig_Read() defaults to argc=0 (GH-8595)
    ea68d83

    @vstinner
    Copy link
    Member Author

    vstinner commented Aug 1, 2018

    New changeset 6c785c0 by Victor Stinner in branch 'master':
    bpo-34170: Add Python/coreconfig.c for _PyCoreConfig (GH-8607)
    6c785c0

    @vstinner
    Copy link
    Member Author

    vstinner commented Aug 1, 2018

    New changeset 80b762f by Victor Stinner in branch 'master':
    bpo-31650: Remove _Py_CheckHashBasedPycsMode global config var (GH-8608)
    80b762f

    @vstinner
    Copy link
    Member Author

    vstinner commented Aug 1, 2018

    """
    TODO list, open questions:

    • Should isolated=1 always imply config->ignore_environment=1 and config->user_site_directory = 0?
    • Should we rename debug to parser_debug? "debug" is a very generic term, whereas this flag is really specific to the parser
    • Remove _Py_CheckHashBasedPycsMode: replace it with config._check_hash_pycs_mode?
      """

    => DONE!

    @vstinner
    Copy link
    Member Author

    vstinner commented Aug 2, 2018

    New changeset 72ec319 by Victor Stinner in branch 'master':
    bpo-34170: Cleanup pymain_run_filename() (GH-8631)
    72ec319

    @vstinner
    Copy link
    Member Author

    vstinner commented Aug 3, 2018

    New changeset 5a953fd by Victor Stinner in branch 'master':
    bpo-34170: _PyCoreConfig_Read() don't replace coerce_c_locale (GH-8658)
    5a953fd

    @vstinner
    Copy link
    Member Author

    vstinner commented Aug 3, 2018

    New changeset d807862 by Victor Stinner in branch 'master':
    bpo-34170: Fix pymain_run_file() (GH-8660)
    d807862

    @vstinner
    Copy link
    Member Author

    New changeset 89487f5 by Victor Stinner in branch 'master':
    bpo-34207: Fix pymain_read_conf() for UTF-8 Mode (GH-8868)
    89487f5

    @vstinner
    Copy link
    Member Author

    New changeset 80a0eba by Victor Stinner in branch '3.7':
    bpo-34207: Fix pymain_read_conf() for UTF-8 Mode (GH-8868) (GH-8870)
    80a0eba

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.8 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs)
    Projects
    None yet
    Development

    No branches or pull requests

    1 participant