classification
Title: ctypes: test_frozentable fails when make regen-frozen
Type: compile error Stage: resolved
Components: ctypes, Tests Versions: Python 3.10
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: nascheme Nosy List: hrnciar, hroncok, nascheme, vstinner
Priority: normal Keywords: patch

Created on 2021-03-02 14:40 by hroncok, last changed 2021-03-09 10:41 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 24708 closed nascheme, 2021-03-02 18:59
PR 24714 closed nascheme, 2021-03-03 02:29
PR 24759 merged nascheme, 2021-03-05 02:34
Messages (6)
msg387933 - (view) Author: Miro Hrončok (hroncok) * Date: 2021-03-02 14:40
The following test failure happens on Python 3.10.0a6+ when we make regen-frozen with the same Python version we test:

======================================================================
FAIL: test_frozentable (ctypes.test.test_values.PythonValuesTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/churchyard/Dokumenty/RedHat/cpython/Lib/ctypes/test/test_values.py", line 87, in test_frozentable
    self.assertEqual(items, expected, "PyImport_FrozenModules example "
AssertionError: Lists differ: [('__hello__', 129), ('__phello__', -129), ('__phello__.spam', 129)] != [('__hello__', 125), ('__phello__', -125), ('__phello__.spam', 125)]

First differing element 0:
('__hello__', 129)
('__hello__', 125)

- [('__hello__', 129), ('__phello__', -129), ('__phello__.spam', 129)]
?                  ^                     ^                         ^

+ [('__hello__', 125), ('__phello__', -125), ('__phello__.spam', 125)]
?                  ^                     ^                         ^
 : PyImport_FrozenModules example in Doc/library/ctypes.rst may be out of date

----------------------------------------------------------------------
Ran 494 tests in 0.466s

FAILED (failures=1, skipped=87)


Reproducer:

1. Build Python from source: $ ./configure && make -j...
2. Run ctypes tests: $ ./python -m ctypes.test
3. Regenerate frozen: $ PYTHON_FOR_REGEN=./python make regen-frozen
4. Build Python from source again: $ ./configure && make -j...
5. Run ctypes tests: $ ./python -m ctypes.test

Actual result:

Tests in (2) pass, tests in (5) fail.

The difference after (3) is:

diff --git a/Python/frozen_hello.h b/Python/frozen_hello.h
index 9c566cc81e..d58b726aa8 100644
--- a/Python/frozen_hello.h
+++ b/Python/frozen_hello.h
@@ -9,5 +9,5 @@ static unsigned char M___hello__[] = {
     100,218,5,112,114,105,110,116,169,0,114,2,0,
     0,0,114,2,0,0,0,218,4,110,111,110,101,
     218,8,60,109,111,100,117,108,101,62,1,0,0,
-    0,115,2,0,0,0,4,1,
+    0,115,6,0,0,0,4,0,12,1,255,128,
 };


Expected results:

Tests pass, no diff.
msg387936 - (view) Author: Miro Hrončok (hroncok) * Date: 2021-03-02 14:54
When I run `PYTHON_FOR_REGEN=python3.10 make regen-frozen` with Python 3.10.0a5, I get the same diff and the same problem.

When I run `PYTHON_FOR_REGEN=python3.9 make regen-frozen` with Python 3.9.2, I get no diff and no problem (similarly with Python 3.8.8).
msg387956 - (view) Author: Neil Schemenauer (nascheme) * (Python committer) Date: 2021-03-02 19:43
I believe the line table format got changed but the frozen code didn't get re-generated.  If you try to call co_lines() on the __hello__ code, Python crashes.

>>> import __hello__
Hello world!
>>> co = __hello__.__spec__.loader.get_code('__hello__')
>>> co.co_linetable
b'\x04\x01'
>>> list(co.co_lines())
python: ../Objects/codeobject.c:1185: PyLineTable_NextAddressRange: Assertion `!at_end(range)' failed.

My PR re-generates the code and fixes the test.  Perhaps I should also add a test to exercise co_lines() on the frozen code object.
msg388216 - (view) Author: Neil Schemenauer (nascheme) * (Python committer) Date: 2021-03-06 21:34
New changeset 87ec26b812e9c4095c017dc60f246eda37b83ab2 by Neil Schemenauer in branch 'master':
bpo-43372: Use _freeze_importlib for regen-frozen. (GH-24759)
https://github.com/python/cpython/commit/87ec26b812e9c4095c017dc60f246eda37b83ab2
msg388229 - (view) Author: Miro Hrončok (hroncok) * Date: 2021-03-07 09:54
Thanks for the fixer!
msg388347 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-03-09 10:41
See also bpo-43445 "Add frozen modules to sys.stdlib_module_names".
History
Date User Action Args
2021-03-09 10:41:57vstinnersetnosy: + vstinner
messages: + msg388347
2021-03-07 09:54:03hroncoksetmessages: + msg388229
2021-03-06 21:34:34naschemesetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2021-03-06 21:34:06naschemesetmessages: + msg388216
2021-03-05 02:39:10naschemesetassignee: nascheme
2021-03-05 02:34:55naschemesetpull_requests: + pull_request23531
2021-03-03 02:29:54naschemesetpull_requests: + pull_request23491
2021-03-02 19:43:33naschemesetmessages: + msg387956
2021-03-02 18:59:32naschemesetkeywords: + patch
stage: patch review
pull_requests: + pull_request23484
2021-03-02 14:54:35hroncoksetmessages: + msg387936
2021-03-02 14:48:40hroncoksetnosy: + nascheme
2021-03-02 14:40:34hroncokcreate