Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[C API] Add tests on Py_FrozenMain() #88297

Closed
vstinner opened this issue May 14, 2021 · 8 comments
Closed

[C API] Add tests on Py_FrozenMain() #88297

vstinner opened this issue May 14, 2021 · 8 comments
Labels
3.11 only security fixes topic-C-API

Comments

@vstinner
Copy link
Member

BPO 44131
Nosy @vstinner, @encukou, @corona10, @pablogsal, @miss-islington
PRs
  • bpo-44131: Test Py_FrozenMain() #26126
  • bpo-44131: Py_FrozenMain() uses PyConfig_SetBytesArgv() #26201
  • bpo-44131: Fix Makefile for test_frozenmain #26203
  • bpo-43795: Remove Py_FrozenMain from the Limited API & Stable ABI  #26241
  • [3.10] bpo-43795: Remove Py_FrozenMain from the Limited API & Stable ABI (GH-26241) #26353
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2021-05-25.12:52:42.813>
created_at = <Date 2021-05-14.12:12:49.610>
labels = ['expert-C-API', '3.11']
title = '[C API] Add tests on Py_FrozenMain()'
updated_at = <Date 2021-05-26.22:38:25.804>
user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2021-05-26.22:38:25.804>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2021-05-25.12:52:42.813>
closer = 'pablogsal'
components = ['C API']
creation = <Date 2021-05-14.12:12:49.610>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 44131
keywords = ['patch']
message_count = 8.0
messages = ['393643', '393836', '393841', '393859', '393893', '394334', '394376', '394483']
nosy_count = 5.0
nosy_names = ['vstinner', 'petr.viktorin', 'corona10', 'pablogsal', 'miss-islington']
pr_nums = ['26126', '26201', '26203', '26241', '26353']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue44131'
versions = ['Python 3.11']
    @vstinner
    Copy link
    Member Author

    I would like to change Py_FrozenMain() using the new PyConfig C API, but I would prefer to first write tests of its current behavior. Currently, Py_FrozenMain() has no test.

    Py_FrozenMain() is used by Tools/freeze/freeze.py. Old freeze.py issues:

    I never used freeze.py, but since we get bug reports, it seems like it's used by users :-)

    Note: freeze.py has no test neither.

    Py_FrozenMain() has been modified recently in bpo-44113:

    commit 7565586
    Author: Dong-hee Na <donghee.na@python.org>
    Date: Thu May 13 10:19:46 2021 +0900

    bpo-44113: Update fromzenmain not to use Py_SetProgramName (GH-26085)

    See also: [capi-sig] Py_FrozenMain and the stable ABI
    https://mail.python.org/archives/list/capi-sig@python.org/thread/5QLI3NUP3OSGLCCIBAQOTX4GEJQBWJ6F/

    See also: [Python-Dev] What's the story on Py_FrozenMain?
    https://mail.python.org/pipermail/python-dev/2013-November/130280.html

    @vstinner vstinner added 3.11 only security fixes topic-C-API labels May 14, 2021
    @vstinner
    Copy link
    Member Author

    commit eaede0d
    Author: Victor Stinner <vstinner@python.org>
    Date: Mon May 17 23:48:35 2021 +0200

    bpo-44131: Test Py_FrozenMain() (GH-26126)

* Add test_frozenmain to test_embed
* Add [Programs/test_frozenmain.py](https://github.com/python/cpython/blob/main/Programs/test_frozenmain.py)
* Add [Programs/freeze_test_frozenmain.py](https://github.com/python/cpython/blob/main/Programs/freeze_test_frozenmain.py)
* Add [Programs/test_frozenmain.h](https://github.com/python/cpython/blob/main/Programs/test_frozenmain.h)
* Add make regen-test-frozenmain
* Add test_frozenmain command to [Programs/_testembed](https://github.com/python/cpython/blob/main/Programs/_testembed)
* _testembed.c: add error(msg) function

    @vstinner
    Copy link
    Member Author

    Compilation failed on s390x RHEL7 LTO + PGO 3.x buildbot:
    https://buildbot.python.org/all/#/builders/244/builds/208

    (...)
    rm -f libpython3.11.a
    ar rcs libpython3.11.a Modules/getbuildinfo.o (...) Python/frozen.o
    lto1: error: libpython3.11.a: file too short
    (null):0: confused by earlier errors, bailing out
    lto-wrapper: gcc returned 1 exit status
    /usr/bin/ld: error: lto-wrapper failed
    collect2: error: ld returned 1 exit status

    make[3]: *** [python] Error 1
    make[3]: *** Waiting for unfinished jobs....

    gcc -pthread -flto -fuse-linker-plugin -ffat-lto-objects -flto-partition=none -g -fprofile-generate -Xlinker -export-dynamic -o python Programs/python.o libpython3.11.a -lcrypt -lpthread -ldl -lutil -lm -lm
    # Regenerate Programs/test_frozenmain.h
    # from Programs/test_frozenmain.py
    # using Programs/freeze_test_frozenmain.py
    ./python Programs/freeze_test_frozenmain.py Programs/test_frozenmain.h
    Could not find platform dependent libraries <exec_prefix>
    Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
    Programs/test_frozenmain.h written

    make[4]: Leaving directory /home/dje/cpython-buildarea/3.x.edelsohn-rhel-z.lto-pgo/build' make[3]: Leaving directory /home/dje/cpython-buildarea/3.x.edelsohn-rhel-z.lto-pgo/build'
    make[2]: *** [build_all_generate_profile] Error 2
    make[2]: Leaving directory /home/dje/cpython-buildarea/3.x.edelsohn-rhel-z.lto-pgo/build' make[1]: *** [profile-gen-stamp] Error 2 make[1]: Leaving directory /home/dje/cpython-buildarea/3.x.edelsohn-rhel-z.lto-pgo/build'
    make: *** [profile-run-stamp] Error 2
    program finished with exit code 2
    elapsedTime=62.919052

    @encukou
    Copy link
    Member

    encukou commented May 18, 2021

    The tests are now skipped on Windows. It might be better to use a feature check: hasattr(ctypes.pythonapi, 'Py_FrozenMain')

    IMO, Py_FrozenMain is quite specific: it's OK if it's exported, but also OK if it's missing. (Unless you're compiling a special build with frozen modules, which won't work at all without Py_FrozenMain.)

    @vstinner
    Copy link
    Member Author

    The tests are now skipped on Windows. It might be better to use a feature check: hasattr(ctypes.pythonapi, 'Py_FrozenMain')

    On Unix, _testembed gets the symbol since it's used. Currently, hasattr(ctypes.pythonapi, 'Py_FrozenMain') returns false if Python is built without --enable-shared, even if _testembed is able to get Py_FrozenMain().

    I hope that bpo-44133 will be solved soon.

    @pablogsal
    Copy link
    Member

    New changeset 68b9c20 by Miss Islington (bot) in branch '3.10':
    bpo-43795: Remove Py_FrozenMain from the Limited API & Stable ABI (GH-26241) (GH-26353)
    68b9c20

    @vstinner
    Copy link
    Member Author

    commit eaede0d
    Author: Victor Stinner <vstinner@python.org>
    Date: Mon May 17 23:48:35 2021 +0200

    bpo-44131: Test Py_FrozenMain() (GH-26126)

* Add test_frozenmain to test_embed
* Add [Programs/test_frozenmain.py](https://github.com/python/cpython/blob/main/Programs/test_frozenmain.py)
* Add [Programs/freeze_test_frozenmain.py](https://github.com/python/cpython/blob/main/Programs/freeze_test_frozenmain.py)
* Add [Programs/test_frozenmain.h](https://github.com/python/cpython/blob/main/Programs/test_frozenmain.h)
* Add make regen-test-frozenmain
* Add test_frozenmain command to [Programs/_testembed](https://github.com/python/cpython/blob/main/Programs/_testembed)
* _testembed.c: add error(msg) function

    commit 58a9939
    Author: Victor Stinner <vstinner@python.org>
    Date: Thu May 20 12:08:05 2021 +0200

    bpo-44131: Py_FrozenMain() uses PyConfig_SetBytesArgv() (GH-26201)

Moreover, Py_FrozenMain() relies on Py_InitializeFromConfig() to
handle the PYTHONUNBUFFERED environment variable and configure C
stdio streams like stdout (make the stream unbuffered).

    @vstinner
    Copy link
    Member Author

    I marked bpo-36940 as a duplicate of this issue.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    3.11 only security fixes topic-C-API
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    3 participants
    @vstinner @encukou @pablogsal