Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implementation of the PEP 587 #80944

Closed
vstinner opened this issue Apr 30, 2019 · 55 comments
Closed

Implementation of the PEP 587 #80944

vstinner opened this issue Apr 30, 2019 · 55 comments
Labels
3.8 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs)

Comments

@vstinner
Copy link
Member

BPO 36763
Nosy @vstinner, @miss-islington
PRs
  • bpo-36763: Fix Py_SetStandardStreamEncoding() #13028
  • bpo-36763: Add _PyCoreConfig_SetArgv #13030
  • bpo-36763: Rework _PyInitError API #13031
  • bpo-36763: Add _PyCoreConfig_SetString() #13035
  • bpo-36763: _PyCoreConfig_SetPyArgv() preinitializes Python #13037
  • bpo-36763: Make _PyCoreConfig.check_hash_pycs_mode public #13052
  • bpo-36763: Add _PyCoreConfig._config_version #13065
  • bpo-36763: Remove _PyCoreConfig._init_main #13066
  • bpo-36763: Add test for _PyCoreConfig_SetString() #13275
  • bpo-36763: InitConfigTests tests all core config #13331
  • bpo-36763: _PyInitError always use int for exitcode #13360
  • bpo-36763: Add _PyCoreConfig.parse_argv #13361
  • bpo-36763: Add _Py_InitializeMain() #13362
  • bpo-36763: Add _PyCoreConfig.configure_c_stdio #13363
  • bpo-36763: Cleanup precmdline in _PyCoreConfig_Read() #13371
  • bpo-36763: Remove _PyCoreConfig.program #13373
  • bpo-36763: Add PyMemAllocatorName #13387
  • bpo-36763: Add _PyCoreConfig_InitPythonConfig() #13388
  • bpo-36763: _Py_RunMain() doesn't call Py_Exit() anymore #13390
  • bpo-36763: Add _PyInitError functions #13395
  • bpo-36763: _Py_InitializeFromArgs() argc becomes Py_ssize_t #13396
  • bpo-36763: Use _PyCoreConfig_InitPythonConfig() #13398
  • bpo-36763: Remove _PyCoreConfig.dll_path #13402
  • bpo-36763: Fix _PyCoreConfig_DecodeLocaleErr() preinit #13432
  • bpo-36763: Fix encoding/locale tests in test_embed #13443
  • bpo-36763: Fix C locale coercion #13444
  • bpo-36763: Add _PyPreConfig._config_init #13481
  • bpo-36763: Rename Python initialization functions #13511
  • bpo-36763: Fix _PyPreConfig_InitCompatConfig() utf8_mode #13518
  • bpo-36763: Implement the PEP 587 #13592
  • bpo-36763, _testembed: enable assert() in release mode #13857
  • [3.8] bpo-36763, _testembed: enable assert() in release mode (GH-13857) #13863
  • bpo-36763: Add PyConfig_SetWideStringList() #14444
  • bpo-36763: Use PyConfig_Clear() #14445
  • [3.8] bpo-36763: Add PyConfig_SetWideStringList() (GH-14444) #14523
  • [3.8] bpo-36763: Use PyConfig_Clear() (GH-14445) #14528
  • bpo-36763: Implement PyWideStringList_Insert() of PEP 587 #15423
  • [3.8] bpo-36763: Implement PyWideStringList_Insert() of PEP 587 (GH-15423) #15427
  • bpo-36763: PyConfig_Read() handles PySys_AddXOption() #15431
  • bpo-36763, doc: Add links in the new C API init doc #15433
  • [3.8] bpo-36763, doc: Add links in the new C API init doc (GH-15433) #15434
  • [3.8] bpo-36763: PyConfig_Read() handles PySys_AddXOption() (GH-15431) #15435
  • bpo-36763: Make Py_BytesMain() public #15532
  • [3.8] bpo-36763: Make Py_BytesMain() public (GH-15532) #15535
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2019-08-26.23:01:41.769>
created_at = <Date 2019-04-30.22:38:59.950>
labels = ['interpreter-core', '3.8']
title = 'Implementation of the PEP 587'
updated_at = <Date 2019-08-26.23:17:59.610>
user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2019-08-26.23:17:59.610>
actor = 'miss-islington'
assignee = 'none'
closed = True
closed_date = <Date 2019-08-26.23:01:41.769>
closer = 'vstinner'
components = ['Interpreter Core']
creation = <Date 2019-04-30.22:38:59.950>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 36763
keywords = ['patch']
message_count = 55.0
messages = ['341167', '341168', '341172', '341177', '341195', '341244', '341299', '341307', '341311', '341314', '342508', '342527', '342647', '342650', '342654', '342659', '342682', '342697', '342718', '342738', '342766', '342770', '342773', '342782', '342784', '342899', '342907', '342912', '343248', '343254', '343273', '343631', '343648', '343649', '344808', '344810', '345527', '345528', '347047', '347049', '347050', '347051', '347053', '347058', '350311', '350312', '350313', '350314', '350315', '350316', '350317', '350328', '350580', '350582', '350583']
nosy_count = 2.0
nosy_names = ['vstinner', 'miss-islington']
pr_nums = ['13028', '13030', '13031', '13035', '13037', '13052', '13065', '13066', '13275', '13331', '13360', '13361', '13362', '13363', '13371', '13373', '13387', '13388', '13390', '13395', '13396', '13398', '13402', '13432', '13443', '13444', '13481', '13511', '13518', '13592', '13857', '13863', '14444', '14445', '14523', '14528', '15423', '15427', '15431', '15433', '15434', '15435', '15532', '15535']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue36763'
versions = ['Python 3.8']
    @vstinner
    Copy link
    Member Author

    I'm working on changes to complete the PEP-587, Python initiaization API.

    @vstinner vstinner added 3.8 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) labels Apr 30, 2019
    @vstinner
    Copy link
    Member Author

    New changeset 463b82a by Victor Stinner in branch 'master':
    bpo-36763: Fix Py_SetStandardStreamEncoding() (GH-13028)
    463b82a

    @vstinner
    Copy link
    Member Author

    vstinner commented May 1, 2019

    New changeset 5f38b84 by Victor Stinner in branch 'master':
    bpo-36763: Add _PyCoreConfig_SetArgv() (GH-13030)
    5f38b84

    @vstinner
    Copy link
    Member Author

    vstinner commented May 1, 2019

    New changeset db71975 by Victor Stinner in branch 'master':
    bpo-36763: Rework _PyInitError API (GH-13031)
    db71975

    @vstinner
    Copy link
    Member Author

    vstinner commented May 1, 2019

    New changeset 1a9f0d8 by Victor Stinner in branch 'master':
    bpo-36763: Add _PyCoreConfig_SetString() (GH-13035)
    1a9f0d8

    @vstinner
    Copy link
    Member Author

    vstinner commented May 2, 2019

    New changeset cb9fbd3 by Victor Stinner in branch 'master':
    bpo-36763: Make _PyCoreConfig.check_hash_pycs_mode public (GH-13052)
    cb9fbd3

    @vstinner
    Copy link
    Member Author

    vstinner commented May 2, 2019

    New changeset 373893c by Victor Stinner in branch 'master':
    bpo-36763: Add _PyCoreConfig._config_version (GH-13065)
    373893c

    @vstinner
    Copy link
    Member Author

    vstinner commented May 2, 2019

    New changeset 70005ac by Victor Stinner in branch 'master':
    bpo-36763: _PyCoreConfig_SetPyArgv() preinitializes Python (GH-13037)
    70005ac

    @vstinner
    Copy link
    Member Author

    vstinner commented May 2, 2019

    New changeset 4631da1 by Victor Stinner in branch 'master':
    bpo-36763: Remove _PyCoreConfig._init_main (GH-13066)
    4631da1

    @vstinner
    Copy link
    Member Author

    vstinner commented May 2, 2019

    I updated my PEP-587:
    [Python-Dev] RFC: PEP-587 "Python Initialization Configuration": 2nd version
    https://mail.python.org/pipermail/python-dev/2019-May/157290.html

    @vstinner
    Copy link
    Member Author

    New changeset 91c9987 by Victor Stinner in branch 'master':
    bpo-36763: Add test for _PyCoreConfig_SetString() (GH-13275)
    91c9987

    @vstinner
    Copy link
    Member Author

    New changeset 5eb8b07 by Victor Stinner in branch 'master':
    bpo-36763: InitConfigTests tests all core config (GH-13331)
    5eb8b07

    @vstinner
    Copy link
    Member Author

    New changeset dbacfc2 by Victor Stinner in branch 'master':
    bpo-36763: _PyInitError always use int for exitcode (GH-13360)
    dbacfc2

    @vstinner
    Copy link
    Member Author

    New changeset ae239f6 by Victor Stinner in branch 'master':
    bpo-36763: Add _PyCoreConfig.parse_argv (GH-13361)
    ae239f6

    @vstinner
    Copy link
    Member Author

    New changeset 9ef5dca by Victor Stinner in branch 'master':
    bpo-36763: Add _Py_InitializeMain() (GH-13362)
    9ef5dca

    @vstinner
    Copy link
    Member Author

    New changeset 54b43bb by Victor Stinner in branch 'master':
    bpo-36763: Add _PyCoreConfig.configure_c_stdio (GH-13363)
    54b43bb

    @vstinner
    Copy link
    Member Author

    New changeset 870b035 by Victor Stinner in branch 'master':
    bpo-36763: Cleanup precmdline in _PyCoreConfig_Read() (GH-13371)
    870b035

    @vstinner
    Copy link
    Member Author

    New changeset fed02e1 by Victor Stinner in branch 'master':
    bpo-36763: Remove _PyCoreConfig.program (GH-13373)
    fed02e1

    @vstinner
    Copy link
    Member Author

    New changeset b16b4e4 by Victor Stinner in branch 'master':
    bpo-36763: Add PyMemAllocatorName (GH-13387)
    b16b4e4

    @vstinner
    Copy link
    Member Author

    New changeset cab5d07 by Victor Stinner in branch 'master':
    bpo-36763: Add _PyCoreConfig_InitPythonConfig() (GH-13388)
    cab5d07

    @vstinner
    Copy link
    Member Author

    New changeset 1208328 by Victor Stinner in branch 'master':
    bpo-36763: _Py_RunMain() doesn't call Py_Exit() anymore (GH-13390)
    1208328

    @vstinner
    Copy link
    Member Author

    New changeset 871ff77 by Victor Stinner in branch 'master':
    bpo-36763: Add _PyInitError functions (GH-13395)
    871ff77

    @vstinner
    Copy link
    Member Author

    New changeset b594784 by Victor Stinner in branch 'master':
    bpo-36763: _Py_InitializeFromArgs() argc becomes Py_ssize_t (GH-13396)
    b594784

    @vstinner
    Copy link
    Member Author

    New changeset bab0db6 by Victor Stinner in branch 'master':
    bpo-36763: Use _PyCoreConfig_InitPythonConfig() (GH-13398)
    bab0db6

    @vstinner
    Copy link
    Member Author

    New changeset 410759f by Victor Stinner in branch 'master':
    bpo-36763: Remove _PyCoreConfig.dll_path (GH-13402)
    410759f

    @vstinner
    Copy link
    Member Author

    New changeset 6d1c467 by Victor Stinner in branch 'master':
    bpo-36763: Fix Python preinitialization (GH-13432)
    6d1c467

    @vstinner
    Copy link
    Member Author

    New changeset 425717f by Victor Stinner in branch 'master':
    bpo-36763: Fix encoding/locale tests in test_embed (GH-13443)
    425717f

    @vstinner vstinner changed the title PEP 587: Rework initialization API to prepare second version of the PEP Implementation of the PEP 587 May 20, 2019
    @vstinner
    Copy link
    Member Author

    New changeset 0f72147 by Victor Stinner in branch 'master':
    bpo-36763: Fix _PyRuntime.preconfig.coerce_c_locale (GH-13444)
    0f72147

    @vstinner
    Copy link
    Member Author

    New changeset 022be02 by Victor Stinner in branch 'master':
    bpo-36763: Add _PyPreConfig._config_init (GH-13481)
    022be02

    @vstinner
    Copy link
    Member Author

    New changeset 5edcf26 by Victor Stinner in branch 'master':
    bpo-36763: Rename private Python initialization functions (GH-13511)
    5edcf26

    @vstinner
    Copy link
    Member Author

    New changeset 20e1e25 by Victor Stinner in branch 'master':
    bpo-36763: Fix _PyPreConfig_InitCompatConfig() utf8_mode (GH-13518)
    20e1e25

    @vstinner
    Copy link
    Member Author

    New changeset 331a6a5 by Victor Stinner in branch 'master':
    bpo-36763: Implement the PEP-587 (GH-13592)
    331a6a5

    @vstinner
    Copy link
    Member Author

    I closed the following issues which are fixed by the implementation of the PEP-587:

    Copy of my message in bpo-19983: "This issue has been fixed by the PEP-587 in bpo-36763: when Py_Initialize() fails, it no longer calls abort() but only exit(1)."

    I should maybe document this change somewhere.

    In bpo-19983, I proposed a patch (init_error.patch) which changed the error message on initiallization error: "Python initialization error: %s\n" rather than "Fatal Python Error: xxx\n".

    Maybe we can change that?

    --

    I close to leave bpo-22213 open.

    bpo-32573 (missing sys.argv) is already fixed in the master branch, but I have been asked to also fix the issue in Python 3.7, so I wrote PR 13553.

    @vstinner
    Copy link
    Member Author

    TODO: I didn't implement PyWideStringList_Insert() yet.

    @vstinner
    Copy link
    Member Author

    vstinner commented Jun 6, 2019

    New changeset 013a18a by Victor Stinner in branch 'master':
    bpo-36763, _testembed: enable assert() in release mode (GH-13857)
    013a18a

    @miss-islington
    Copy link
    Contributor

    New changeset 4062841 by Miss Islington (bot) in branch '3.8':
    bpo-36763, _testembed: enable assert() in release mode (GH-13857)
    4062841

    @vstinner
    Copy link
    Member Author

    Two interesting TODO.

    sysmodule.c: get_xoptions()

        /* PEP-432 TODO: we can reach this if xoptions is NULL in the main
    *  interpreter config. When that happens, we need to properly set
     * the `xoptions` reference in the main interpreter config as well.
     *
     * For Python 3.7, we shouldn't be able to get here due to the
     * combination of how _PyMainInterpreter_ReadConfig and _PySys_EndInit
     * work, but we expect 3.8+ to make the _PyMainInterpreter_ReadConfig
     * call optional for embedding applications, thus making this
     * reachable again.
     */

    sysmodule.c: get_warnoptions()

        /* PEP-432 TODO: we can reach this if warnoptions is NULL in the main
    *  interpreter config. When that happens, we need to properly set
     * the `warnoptions` reference in the main interpreter config as well.
     *
     * For Python 3.7, we shouldn't be able to get here due to the
     * combination of how _PyMainInterpreter_ReadConfig and _PySys_EndInit
     * work, but we expect 3.8+ to make the _PyMainInterpreter_ReadConfig
     * call optional for embedding applications, thus making this
     * reachable again.
     */

    @vstinner
    Copy link
    Member Author

    _PySys_ReadPreInitOptions() code should be moved somewhere closer to PyConfig_Read().

    @vstinner
    Copy link
    Member Author

    vstinner commented Jul 1, 2019

    New changeset 36242fd by Victor Stinner in branch 'master':
    bpo-36763: Add PyConfig_SetWideStringList() (GH-14444)
    36242fd

    @vstinner
    Copy link
    Member Author

    vstinner commented Jul 1, 2019

    About the documentation: https://docs.python.org/dev/c-api/init.html doesn't link to http://docs.python.org/dev/c-api/init_config.html

    The first doc should mention somewhere the new doc.

    @miss-islington
    Copy link
    Contributor

    New changeset 96f581c by Miss Islington (bot) in branch '3.8':
    bpo-36763: Add PyConfig_SetWideStringList() (GH-14444)
    96f581c

    @vstinner
    Copy link
    Member Author

    vstinner commented Jul 1, 2019

    Another minor implementation issue: calling _PyCoreConfig_Read() twice calls _Py_SetArgcArgv() twice, the second call override the first _Py_SetArgcArgv() call. IMHO it should only be called once, at the first _PyCoreConfig_Read() call.

    @vstinner
    Copy link
    Member Author

    vstinner commented Jul 1, 2019

    New changeset 6731002 by Victor Stinner in branch 'master':
    bpo-36763: Use PyConfig_Clear() (GH-14445)
    6731002

    @miss-islington
    Copy link
    Contributor

    New changeset 4c227e6 by Miss Islington (bot) in branch '3.8':
    bpo-36763: Use PyConfig_Clear() (GH-14445)
    4c227e6

    @vstinner
    Copy link
    Member Author

    New changeset 3842f29 by Victor Stinner in branch 'master':
    bpo-36763: Implement PyWideStringList_Insert() of PEP-587 (GH-15423)
    3842f29

    @miss-islington
    Copy link
    Contributor

    New changeset a6427cb by Miss Islington (bot) in branch '3.8':
    bpo-36763: Implement PyWideStringList_Insert() of PEP-587 (GH-15423)
    a6427cb

    @vstinner
    Copy link
    Member Author

    _PySys_ReadPreInitOptions() code should be moved somewhere closer to PyConfig_Read().

    Done in PR 15431.

    @vstinner
    Copy link
    Member Author

    The first doc should mention somewhere the new doc.

    Done in PR 15433.

    @vstinner
    Copy link
    Member Author

    New changeset 1beb7c3 by Victor Stinner in branch 'master':
    bpo-36763, doc: Add links in the new C API init doc (GH-15433)
    1beb7c3

    @vstinner
    Copy link
    Member Author

    New changeset 120b707 by Victor Stinner in branch 'master':
    bpo-36763: PyConfig_Read() handles PySys_AddXOption() (GH-15431)
    120b707

    @miss-islington
    Copy link
    Contributor

    New changeset 9cbdce3 by Miss Islington (bot) in branch '3.8':
    bpo-36763, doc: Add links in the new C API init doc (GH-15433)
    9cbdce3

    @vstinner
    Copy link
    Member Author

    New changeset af84a88 by Victor Stinner in branch '3.8':
    bpo-36763: PyConfig_Read() handles PySys_AddXOption() (GH-15431) (GH-15435)
    af84a88

    @vstinner
    Copy link
    Member Author

    New changeset 9a943b4 by Victor Stinner in branch 'master':
    bpo-36763: Make Py_BytesMain() public (GH-15532)
    9a943b4

    @vstinner
    Copy link
    Member Author

    I marked the PEP-587 Status as Final. The implementation should now be complete. If there are bugs, they can be fixed in following issues :-) I close now this issue.

    @miss-islington
    Copy link
    Contributor

    New changeset 1d4285a by Miss Islington (bot) in branch '3.8':
    bpo-36763: Make Py_BytesMain() public (GH-15532)
    1d4285a

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    3.8 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs)
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    2 participants
    @vstinner @miss-islington