classification
Title: Test installers before releasing (ModuleNotFoundErrors)
Type: crash Stage: resolved
Components: Installation, Tests, Windows Versions: Python 3.10, Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: steve.dower Nosy List: honglei.jiang, lukasz.langa, miss-islington, ned.deily, pablogsal, paul.moore, steve.dower, terry.reedy, tim.golden, zach.ware
Priority: normal Keywords: patch

Created on 2020-05-24 12:47 by honglei.jiang, last changed 2020-11-13 22:47 by miss-islington. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 23271 merged steve.dower, 2020-11-13 21:54
PR 23273 merged miss-islington, 2020-11-13 22:24
PR 23274 merged miss-islington, 2020-11-13 22:25
Messages (13)
msg369798 - (view) Author: honglei jiang (honglei.jiang) * Date: 2020-05-24 12:47
https://www.python.org/ftp/python/3.9.0/python-3.9.0b1-amd64.exe

C:\Python39>python.exe -m test test_deque -v
== CPython 3.9.0b1 (tags/v3.9.0b1:97fe9cf, May 19 2020, 09:02:07) [MSC v.1924 64 bit (AMD64)]
== Windows-10-10.0.18362-SP0 little-endian
== cwd: C:\Users\jhong\AppData\Local\Temp\test_python_3376
== CPU count: 4
== encodings: locale=cp936, FS=utf-8
0:00:00 Run tests sequentially
0:00:00 [1/1] test_deque
test_add (test.test_deque.TestBasic) ... ok
test_basics (test.test_deque.TestBasic) ... ok
test_big_queue_popleft (test.test_deque.TestBasic) ... ok
test_big_queue_popright (test.test_deque.TestBasic) ... ok
test_big_stack_left (test.test_deque.TestBasic) ... ok
test_big_stack_right (test.test_deque.TestBasic) ... ok
test_clear (test.test_deque.TestBasic) ... ok
test_comparisons (test.test_deque.TestBasic) ... ok
test_container_iterator (test.test_deque.TestBasic) ... ok
test_contains (test.test_deque.TestBasic) ... ok
test_contains_count_stop_crashes (test.test_deque.TestBasic) ... ok
test_copy (test.test_deque.TestBasic) ... ok
test_copy_method (test.test_deque.TestBasic) ... ok
test_count (test.test_deque.TestBasic) ... ok
test_deepcopy (test.test_deque.TestBasic) ... ok
test_delitem (test.test_deque.TestBasic) ... ok
test_extend (test.test_deque.TestBasic) ... ok
test_extendleft (test.test_deque.TestBasic) ... ok
test_gc_doesnt_blowup (test.test_deque.TestBasic) ... ok
test_getitem (test.test_deque.TestBasic) ... ok
test_hash (test.test_deque.TestBasic) ... ok
test_iadd (test.test_deque.TestBasic) ... ok
test_imul (test.test_deque.TestBasic) ... ok
test_index (test.test_deque.TestBasic) ... ok
test_index_bug_24913 (test.test_deque.TestBasic) ... ok
test_init (test.test_deque.TestBasic) ... ok
test_insert (test.test_deque.TestBasic) ... ok
test_insert_bug_26194 (test.test_deque.TestBasic) ... ok
test_iterator_pickle (test.test_deque.TestBasic) ... ok
test_len (test.test_deque.TestBasic) ... ok
test_long_steadystate_queue_popleft (test.test_deque.TestBasic) ... ok
test_long_steadystate_queue_popright (test.test_deque.TestBasic) ... ok
test_maxlen (test.test_deque.TestBasic) ... ok
test_maxlen_attribute (test.test_deque.TestBasic) ... ok
test_maxlen_zero (test.test_deque.TestBasic) ... ok
test_mul (test.test_deque.TestBasic) ... ok
test_pickle (test.test_deque.TestBasic) ... ok
test_pickle_recursive (test.test_deque.TestBasic) ... ok
test_print (test.test_deque.TestBasic) ... ok
test_remove (test.test_deque.TestBasic) ... ok
test_repr (test.test_deque.TestBasic) ... ok
test_reverse (test.test_deque.TestBasic) ... ok
test_reversed (test.test_deque.TestBasic) ... ok
test_reversed_new (test.test_deque.TestBasic) ... ok
test_rotate (test.test_deque.TestBasic) ... ok
test_roundtrip_iter_init (test.test_deque.TestBasic) ... ok
test_setitem (test.test_deque.TestBasic) ... ok
test_sizeof (test.test_deque.TestBasic) ... ERROR
test_underflow (test.test_deque.TestBasic) ... ok
test_constructor (test.test_deque.TestVariousIteratorArgs) ... ok
test_iter_with_altered_data (test.test_deque.TestVariousIteratorArgs) ... ok
test_runtime_error_on_empty_deque (test.test_deque.TestVariousIteratorArgs) ... ok
test_basics (test.test_deque.TestSubclass) ... ok
test_bug_31608 (test.test_deque.TestSubclass) ... ok
test_copy_pickle (test.test_deque.TestSubclass) ... ok
test_pickle_recursive (test.test_deque.TestSubclass) ... ok
test_strange_subclass (test.test_deque.TestSubclass) ... ok
test_weakref (test.test_deque.TestSubclass) ... ok
test_subclass_with_kwargs (test.test_deque.TestSubclassWithKwargs) ... ok
test_addmul (test.test_deque.TestSequence) ... ok
test_bigrepeat (test.test_deque.TestSequence) ... ok
test_constructors (test.test_deque.TestSequence) ... ok
test_contains (test.test_deque.TestSequence) ... ok
test_contains_fake (test.test_deque.TestSequence) ... ok
test_contains_order (test.test_deque.TestSequence) ... ok
test_count (test.test_deque.TestSequence) ... ok
test_free_after_iterating (test.test_deque.TestSequence) ... skipped "Exhausted deque iterator doesn't free a deque"
test_getitem (test.test_deque.TestSequence) ... ok
test_getitemoverwriteiter (test.test_deque.TestSequence) ... ok
test_getslice (test.test_deque.TestSequence) ... ok
test_iadd (test.test_deque.TestSequence) ... ok
test_imul (test.test_deque.TestSequence) ... ok
test_index (test.test_deque.TestSequence) ... ok
test_len (test.test_deque.TestSequence) ... ok
test_minmax (test.test_deque.TestSequence) ... ok
test_pickle (test.test_deque.TestSequence) ... ok
test_repeat (test.test_deque.TestSequence) ... ok
test_subscript (test.test_deque.TestSequence) ... ok
test_truth (test.test_deque.TestSequence) ... ok

======================================================================
ERROR: test_sizeof (test.test_deque.TestBasic)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Python39\lib\test\test_deque.py", line 781, in test_sizeof
    check(deque(), basesize + blocksize)
  File "C:\Python39\lib\test\support\__init__.py", line 1483, in check_sizeof
    import _testinternalcapi
ModuleNotFoundError: No module named '_testinternalcapi'

----------------------------------------------------------------------

Ran 79 tests in 0.696s

FAILED (errors=1, skipped=1)
test test_deque failed
test_deque failed

== Tests result: FAILURE ==

1 test failed:
    test_deque

Total duration: 906 ms
Tests result: FAILURE

C:\Python39>
msg369912 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2020-05-25 19:33
At a guess, it looks like we're not including that module in the Windows installer.  And indeed, it looks like it's left out of Tools/msi/test/test_files.wxs if you would like to provide a PR :)

On the other hand, I'm not sure that test (and any other depending on that module) shouldn't be skipped if _testinternalcapi is missing.
msg369976 - (view) Author: honglei jiang (honglei.jiang) * Date: 2020-05-26 12:45
== Tests result: FAILURE ==

357 tests OK.

24 tests failed:
    test__locale test_array test_buffer test_capi test_deque
    test_exceptions test_flufl test_grammar test_itertools test_locale
    test_memoryio test_ordered_dict test_parser test_pickle
    test_positional_only_arg test_string_literals test_struct
    test_syntax test_sys test_traceback test_urllib test_urllib2
    test_urllib2_localnet test_xml_etree_c

43 tests skipped:
    test_asdl_parser test_check_c_globals test_clinic test_curses
    test_dbm_gnu test_dbm_ndbm test_devpoll test_epoll test_fcntl
    test_fork1 test_gdb test_grp test_ioctl test_kqueue
    test_multiprocessing_fork test_multiprocessing_forkserver test_nis
    test_openpty test_ossaudiodev test_pipes test_poll test_posix
    test_pty test_pwd test_readline test_resource test_smtpnet
    test_socketserver test_spwd test_syslog test_threadsignals
    test_timeout test_tix test_tk test_ttk_guionly test_urllib2net
    test_urllibnet test_wait3 test_wait4 test_winsound test_xmlrpc_net
    test_xxtestfuzz test_zipfile64

Zachary Ware <report@bugs.python.org> 于2020年5月26日周二 上午3:33写道:

>
> Zachary Ware <zachary.ware@gmail.com> added the comment:
>
> At a guess, it looks like we're not including that module in the Windows
> installer.  And indeed, it looks like it's left out of
> Tools/msi/test/test_files.wxs if you would like to provide a PR :)
>
> On the other hand, I'm not sure that test (and any other depending on that
> module) shouldn't be skipped if _testinternalcapi is missing.
>
> ----------
> components: +Installation, Windows
> nosy: +paul.moore, steve.dower, tim.golden, zach.ware
> versions: +Python 3.10
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue40754>
> _______________________________________
>
msg370344 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2020-05-30 00:57
I verified that running tests for installed 64-bit 3.9.0b1 on Win10 is somewhat a joke.  I got 22 failures.  Multiple tests failed on missing _testinternalcapi, and at least 1 on _xxtestfuzz.  Several tests had multiple failures and would need to be run individually and verbosely to see why.  For instance, test_itertools tries to run test.support.__init__.checksizeof 4 times, and the latter imports _testinternalcapi, resulting is 4 failures.

Release managers: I think installers should be run and the suite run with the result.  This is not the first time a file has been omitted.  One needed by IDLE was once omitted, I believe on Windows.  There is now a test that reads it, but is useless unless run.  (And I should add checks for all the other 'not .py' files.)

honglei: Thank you for both posts.  Next time, please remove the quoted message when replying by email.  Once posted on the web page, below the quoted message, the quote becomes redundant noise.
msg370346 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-30 01:08
We have a bunch of buildbots that test the installed Python. For instance, if you search for "installed" in https://buildbot.python.org/all/#/builders?tags=%2B3.x we have:

aarch64 Fedora Rawhide Clang Installed 3.x	
aarch64 Fedora Stable Clang Installed 3.x	
AMD64 Fedora Rawhide Clang Installed 3.x	
AMD64 Fedora Stable Clang Installed 3.x
PPC64LE Fedora Rawhide Clang Installed 3.x
PPC64LE Fedora Stable Clang Installed 3.x
s390x Fedora Clang Installed 3.x	
s390x Fedora Rawhide Clang Installed 3.x
x86 Gentoo Installed with X

So it seems that only Linux has "installed" builbot tests. I suppose the next step is adding some macOS and Windows ones.
msg370347 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-05-30 01:33
> So it seems that only Linux has "installed" builbot tests. I suppose the next step is adding some macOS and Windows ones.

Having more buildbots test from "installed" locations, rather than the build directory, is good.

But that's not the whole issue here. Those "installed" tests are only testing that a Python runs from an installed location on the same system it was built.  The Windows and macOS installers use various packaging methods to provide complete Python installations, including binaries, that can be installed on other systems using various OS-native install methods familiar to users of each platform. And the binaries are generally built in such a way to support running on multiple release versions of its operating system. For example, the current Python installers for macOS from python.org are usable on macOS 10.9 through 10.15. To produce the various installer packages, there are other layers of tooling involved, say, on top of "make install" in the Unixy build process for macOS; something similar is true for Windows releases.

FWIW, every macOS installer is test installed on a vanilla system, the whole standard test suite is run, and an IDLE smoke test is run to ensure there are no noticeable regressions before handing the build artifacts off to the release manager. Every installer is tested on at least one supported macOS version and usually more than one.
msg370390 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-05-30 17:02
There are non-binding CI tests run with Github Actions (post-merge) that test from a layout, and last I checked they were all passing. Nobody is really monitoring it but me though, as it isn't a buildbot. 

The Windows installer also runs pip and IDLE tests before being published, though not a full test suite.

Honestly, it's pretty quick to generate an install layout for Windows now I have the PC/layout script. We could do it for every test run if we wanted, though then a number of build-tree only tests would be skipped.
msg370408 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-05-31 01:12
> The Windows installer also runs pip and IDLE tests before being published [...]

That's good and reminds me that I forgot to mention that for the macOS installer, besides the already mentioned tests, we also download, install, and smoke test a third-party wheel from PyPI using the bundled pip and we also do a visual inspection of the release's changelog by launching through IDLE the docset bundled with each installer. If the Python binaries included with the installer are built for multiple CPU architectures (macOS universal binaries), we would also attempt to ensure that the test suite is run at least once on each architecture (as of several months ago, we no longer regularly distribute such binaries but there's a good chance we may need to do so again in the not-too-distant future, for example, if Apple introduces macOS machines based on ARM CPUs).
msg371124 - (view) Author: honglei jiang (honglei.jiang) * Date: 2020-06-09 16:44
c:\Python39>python -m test -j4
== CPython 3.9.0b2 (tags/v3.9.0b2:0b1a1c8, Jun 8 2020, 22:55:31) [MSC v.1924 64 bit (AMD64)]
== Windows-10-10.0.18362-SP0 little-endian
== cwd: C:\Users\jhong\AppData\Local\Temp\test_python_5960
== CPU count: 4
== encodings: locale=cp936, FS=utf-8
0:00:00 Run tests in parallel using 4 child processes

== Tests result: FAILURE ==

358 tests OK.

24 tests failed:
    test__locale test_array test_buffer test_capi test_deque
    test_exceptions test_flufl test_fstring test_grammar
    test_itertools test_locale test_memoryio test_ordered_dict
    test_parser test_pickle test_positional_only_arg
    test_string_literals test_struct test_syntax test_sys
    test_traceback test_urllib test_urllib2 test_xml_etree_c

43 tests skipped:
    test_asdl_parser test_check_c_globals test_clinic test_curses
    test_dbm_gnu test_dbm_ndbm test_devpoll test_epoll test_fcntl
    test_fork1 test_gdb test_grp test_ioctl test_kqueue
    test_multiprocessing_fork test_multiprocessing_forkserver test_nis
    test_openpty test_ossaudiodev test_pipes test_poll test_posix
    test_pty test_pwd test_readline test_resource test_smtpnet
    test_socketserver test_spwd test_syslog test_threadsignals
    test_timeout test_tix test_tk test_ttk_guionly test_urllib2net
    test_urllibnet test_wait3 test_wait4 test_winsound test_xmlrpc_net
    test_xxtestfuzz test_zipfile64

Total duration: 6 min 44 sec
Tests result: FAILURE
msg377093 - (view) Author: honglei jiang (honglei.jiang) * Date: 2020-09-18 07:41
Still missing _testinternalcapi module for Python3.9rc2
like https://bugs.python.org/issue36635

C:\Users\Admin>c:\python39\python.exe -m test test_deque
0:00:00 Run tests sequentially
0:00:00 [1/1] test_deque
test test_deque failed -- Traceback (most recent call last):
  File "c:\python39\lib\test\test_deque.py", line 781, in test_sizeof
    check(deque(), basesize + blocksize)
  File "c:\python39\lib\test\support\__init__.py", line 1484, in check_sizeof
    import _testinternalcapi
ModuleNotFoundError: No module named '_testinternalcapi'

test_deque failed

== Tests result: FAILURE ==

1 test failed:
    test_deque

Total duration: 1.1 sec
Tests result: FAILURE
msg380935 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-11-13 22:24
New changeset 9b6934230c35e24d8582ea8c58456fa8eab72ae2 by Steve Dower in branch 'master':
bpo-40754: Adds _testinternalcapi to Windows installer for test suite (GH-23271)
https://github.com/python/cpython/commit/9b6934230c35e24d8582ea8c58456fa8eab72ae2
msg380937 - (view) Author: miss-islington (miss-islington) Date: 2020-11-13 22:42
New changeset 8a4557240b98c322d611bfbba3ea51eac3fb841a by Miss Islington (bot) in branch '3.8':
bpo-40754: Adds _testinternalcapi to Windows installer for test suite (GH-23271)
https://github.com/python/cpython/commit/8a4557240b98c322d611bfbba3ea51eac3fb841a
msg380938 - (view) Author: miss-islington (miss-islington) Date: 2020-11-13 22:47
New changeset faadc52e755cdb316a53f3db5aa11cb97f1c4b87 by Miss Islington (bot) in branch '3.9':
bpo-40754: Adds _testinternalcapi to Windows installer for test suite (GH-23271)
https://github.com/python/cpython/commit/faadc52e755cdb316a53f3db5aa11cb97f1c4b87
History
Date User Action Args
2020-11-13 22:47:34miss-islingtonsetmessages: + msg380938
2020-11-13 22:42:25miss-islingtonsetnosy: + miss-islington
messages: + msg380937
2020-11-13 22:28:09steve.dowersetstatus: open -> closed

nosy: - miss-islington
assignee: steve.dower
resolution: fixed
stage: patch review -> resolved
2020-11-13 22:25:03miss-islingtonsetpull_requests: + pull_request22170
2020-11-13 22:24:54miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request22169
2020-11-13 22:24:44steve.dowersetmessages: + msg380935
2020-11-13 21:54:46steve.dowersetkeywords: + patch
stage: patch review
pull_requests: + pull_request22168
2020-09-18 07:41:55honglei.jiangsetmessages: + msg377093
2020-06-09 16:44:27honglei.jiangsetmessages: + msg371124
2020-05-31 01:12:46ned.deilysetmessages: + msg370408
2020-05-30 17:02:38steve.dowersetmessages: + msg370390
2020-05-30 01:33:12ned.deilysetmessages: + msg370347
2020-05-30 01:08:32pablogsalsetmessages: + msg370346
2020-05-30 00:57:03terry.reedysetnosy: + terry.reedy, ned.deily, lukasz.langa, pablogsal

messages: + msg370344
title: ModuleNotFoundError: No module named '_testinternalcapi' under Win10 -> Test installers before releasing (ModuleNotFoundErrors)
2020-05-26 12:45:23honglei.jiangsetmessages: + msg369976
2020-05-25 19:33:13zach.waresetversions: + Python 3.10
nosy: + paul.moore, tim.golden, zach.ware, steve.dower

messages: + msg369912

components: + Installation, Windows
2020-05-24 12:47:10honglei.jiangcreate