Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ctypes: test_frozentable fails when make regen-frozen #87538

Closed
hroncok mannequin opened this issue Mar 2, 2021 · 6 comments
Closed

ctypes: test_frozentable fails when make regen-frozen #87538

hroncok mannequin opened this issue Mar 2, 2021 · 6 comments
Assignees
Labels
3.10 only security fixes build The build process and cross-build tests Tests in the Lib/test dir topic-ctypes

Comments

@hroncok
Copy link
Mannequin

hroncok mannequin commented Mar 2, 2021

BPO 43372
Nosy @nascheme, @vstinner, @hroncok, @hrnciar
PRs
  • bpo-43372: Re-generate frozen code for __hello__. #24708
  • bpo-43372: Use BUILDPYTHON for regen-frozen. #24714
  • bpo-43372: Use _freeze_importlib for regen-frozen. #24759
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/nascheme'
    closed_at = <Date 2021-03-06.21:34:34.686>
    created_at = <Date 2021-03-02.14:40:34.110>
    labels = ['ctypes', 'build', 'tests', '3.10']
    title = 'ctypes: test_frozentable fails when make regen-frozen'
    updated_at = <Date 2021-03-09.10:41:57.922>
    user = 'https://github.com/hroncok'

    bugs.python.org fields:

    activity = <Date 2021-03-09.10:41:57.922>
    actor = 'vstinner'
    assignee = 'nascheme'
    closed = True
    closed_date = <Date 2021-03-06.21:34:34.686>
    closer = 'nascheme'
    components = ['Tests', 'ctypes']
    creation = <Date 2021-03-02.14:40:34.110>
    creator = 'hroncok'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 43372
    keywords = ['patch']
    message_count = 6.0
    messages = ['387933', '387936', '387956', '388216', '388229', '388347']
    nosy_count = 4.0
    nosy_names = ['nascheme', 'vstinner', 'hroncok', 'hrnciar']
    pr_nums = ['24708', '24714', '24759']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'compile error'
    url = 'https://bugs.python.org/issue43372'
    versions = ['Python 3.10']

    @hroncok
    Copy link
    Mannequin Author

    hroncok mannequin commented Mar 2, 2021

    The following test failure happens on Python 3.10.0a6+ when we make regen-frozen with the same Python version we test:

    ======================================================================
    FAIL: test_frozentable (ctypes.test.test_values.PythonValuesTestCase)
    ----------------------------------------------------------------------

    Traceback (most recent call last):
      File "/home/churchyard/Dokumenty/RedHat/cpython/Lib/ctypes/test/test_values.py", line 87, in test_frozentable
        self.assertEqual(items, expected, "PyImport_FrozenModules example "
    AssertionError: Lists differ: [('__hello__', 129), ('__phello__', -129), ('__phello__.spam', 129)] != [('__hello__', 125), ('__phello__', -125), ('__phello__.spam', 125)]

    First differing element 0:
    ('__hello__', 129)
    ('__hello__', 125)

    • [('__hello__', 129), ('__phello__', -129), ('__phello__.spam', 129)]
      ? ^ ^ ^

    + [('__hello__', 125), ('__phello__', -125), ('__phello__.spam', 125)]
    ? ^ ^ ^
    : PyImport_FrozenModules example in Doc/library/ctypes.rst may be out of date

    ----------------------------------------------------------------------
    Ran 494 tests in 0.466s

    FAILED (failures=1, skipped=87)

    Reproducer:

    1. Build Python from source: $ ./configure && make -j...
    2. Run ctypes tests: $ ./python -m ctypes.test
    3. Regenerate frozen: $ PYTHON_FOR_REGEN=./python make regen-frozen
    4. Build Python from source again: $ ./configure && make -j...
    5. Run ctypes tests: $ ./python -m ctypes.test

    Actual result:

    Tests in (2) pass, tests in (5) fail.

    The difference after (3) is:

    diff --git a/Python/frozen_hello.h b/Python/frozen_hello.h
    index 9c566cc81e..d58b726aa8 100644
    --- a/Python/frozen_hello.h
    +++ b/Python/frozen_hello.h
    @@ -9,5 +9,5 @@ static unsigned char M___hello__[] = {
         100,218,5,112,114,105,110,116,169,0,114,2,0,
         0,0,114,2,0,0,0,218,4,110,111,110,101,
         218,8,60,109,111,100,117,108,101,62,1,0,0,
    -    0,115,2,0,0,0,4,1,
    +    0,115,6,0,0,0,4,0,12,1,255,128,
     };

    Expected results:

    Tests pass, no diff.

    @hroncok hroncok mannequin added 3.10 only security fixes tests Tests in the Lib/test dir topic-ctypes build The build process and cross-build labels Mar 2, 2021
    @hroncok
    Copy link
    Mannequin Author

    hroncok mannequin commented Mar 2, 2021

    When I run PYTHON_FOR_REGEN=python3.10 make regen-frozen with Python 3.10.0a5, I get the same diff and the same problem.

    When I run PYTHON_FOR_REGEN=python3.9 make regen-frozen with Python 3.9.2, I get no diff and no problem (similarly with Python 3.8.8).

    @nascheme
    Copy link
    Member

    nascheme commented Mar 2, 2021

    I believe the line table format got changed but the frozen code didn't get re-generated. If you try to call co_lines() on the __hello__ code, Python crashes.

    >>> import __hello__
    Hello world!
    >>> co = __hello__.__spec__.loader.get_code('__hello__')
    >>> co.co_linetable
    b'\x04\x01'
    >>> list(co.co_lines())
    python: ../Objects/codeobject.c:1185: PyLineTable_NextAddressRange: Assertion `!at_end(range)' failed.

    My PR re-generates the code and fixes the test. Perhaps I should also add a test to exercise co_lines() on the frozen code object.

    @nascheme
    Copy link
    Member

    nascheme commented Mar 6, 2021

    New changeset 87ec26b by Neil Schemenauer in branch 'master':
    bpo-43372: Use _freeze_importlib for regen-frozen. (GH-24759)
    87ec26b

    @hroncok
    Copy link
    Mannequin Author

    hroncok mannequin commented Mar 7, 2021

    Thanks for the fixer!

    @vstinner
    Copy link
    Member

    vstinner commented Mar 9, 2021

    See also bpo-43445 "Add frozen modules to sys.stdlib_module_names".

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.10 only security fixes build The build process and cross-build tests Tests in the Lib/test dir topic-ctypes
    Projects
    None yet
    Development

    No branches or pull requests

    2 participants