https://www.python.org/ftp/python/3.9.0/python-3.9.0b1-amd64.exe

C:\Python39>python.exe -m test test_deque -v
== CPython 3.9.0b1 (tags/v3.9.0b1:97fe9cf, May 19 2020, 09:02:07) [MSC v.1924 64 bit (AMD64)]
== Windows-10-10.0.18362-SP0 little-endian
== cwd: C:\Users\jhong\AppData\Local\Temp\test_python_3376
== CPU count: 4
== encodings: locale=cp936, FS=utf-8
0:00:00 Run tests sequentially
0:00:00 [1/1] test_deque
test_add (test.test_deque.TestBasic) ... ok
test_basics (test.test_deque.TestBasic) ... ok
test_big_queue_popleft (test.test_deque.TestBasic) ... ok
test_big_queue_popright (test.test_deque.TestBasic) ... ok
test_big_stack_left (test.test_deque.TestBasic) ... ok
test_big_stack_right (test.test_deque.TestBasic) ... ok
test_clear (test.test_deque.TestBasic) ... ok
test_comparisons (test.test_deque.TestBasic) ... ok
test_container_iterator (test.test_deque.TestBasic) ... ok
test_contains (test.test_deque.TestBasic) ... ok
test_contains_count_stop_crashes (test.test_deque.TestBasic) ... ok
test_copy (test.test_deque.TestBasic) ... ok
test_copy_method (test.test_deque.TestBasic) ... ok
test_count (test.test_deque.TestBasic) ... ok
test_deepcopy (test.test_deque.TestBasic) ... ok
test_delitem (test.test_deque.TestBasic) ... ok
test_extend (test.test_deque.TestBasic) ... ok
test_extendleft (test.test_deque.TestBasic) ... ok
test_gc_doesnt_blowup (test.test_deque.TestBasic) ... ok
test_getitem (test.test_deque.TestBasic) ... ok
test_hash (test.test_deque.TestBasic) ... ok
test_iadd (test.test_deque.TestBasic) ... ok
test_imul (test.test_deque.TestBasic) ... ok
test_index (test.test_deque.TestBasic) ... ok
test_index_bug_24913 (test.test_deque.TestBasic) ... ok
test_init (test.test_deque.TestBasic) ... ok
test_insert (test.test_deque.TestBasic) ... ok
test_insert_bug_26194 (test.test_deque.TestBasic) ... ok
test_iterator_pickle (test.test_deque.TestBasic) ... ok
test_len (test.test_deque.TestBasic) ... ok
test_long_steadystate_queue_popleft (test.test_deque.TestBasic) ... ok
test_long_steadystate_queue_popright (test.test_deque.TestBasic) ... ok
test_maxlen (test.test_deque.TestBasic) ... ok
test_maxlen_attribute (test.test_deque.TestBasic) ... ok
test_maxlen_zero (test.test_deque.TestBasic) ... ok
test_mul (test.test_deque.TestBasic) ... ok
test_pickle (test.test_deque.TestBasic) ... ok
test_pickle_recursive (test.test_deque.TestBasic) ... ok
test_print (test.test_deque.TestBasic) ... ok
test_remove (test.test_deque.TestBasic) ... ok
test_repr (test.test_deque.TestBasic) ... ok
test_reverse (test.test_deque.TestBasic) ... ok
test_reversed (test.test_deque.TestBasic) ... ok
test_reversed_new (test.test_deque.TestBasic) ... ok
test_rotate (test.test_deque.TestBasic) ... ok
test_roundtrip_iter_init (test.test_deque.TestBasic) ... ok
test_setitem (test.test_deque.TestBasic) ... ok
test_sizeof (test.test_deque.TestBasic) ... ERROR
test_underflow (test.test_deque.TestBasic) ... ok
test_constructor (test.test_deque.TestVariousIteratorArgs) ... ok
test_iter_with_altered_data (test.test_deque.TestVariousIteratorArgs) ... ok
test_runtime_error_on_empty_deque (test.test_deque.TestVariousIteratorArgs) ... ok
test_basics (test.test_deque.TestSubclass) ... ok
test_bug_31608 (test.test_deque.TestSubclass) ... ok
test_copy_pickle (test.test_deque.TestSubclass) ... ok
test_pickle_recursive (test.test_deque.TestSubclass) ... ok
test_strange_subclass (test.test_deque.TestSubclass) ... ok
test_weakref (test.test_deque.TestSubclass) ... ok
test_subclass_with_kwargs (test.test_deque.TestSubclassWithKwargs) ... ok
test_addmul (test.test_deque.TestSequence) ... ok
test_bigrepeat (test.test_deque.TestSequence) ... ok
test_constructors (test.test_deque.TestSequence) ... ok
test_contains (test.test_deque.TestSequence) ... ok
test_contains_fake (test.test_deque.TestSequence) ... ok
test_contains_order (test.test_deque.TestSequence) ... ok
test_count (test.test_deque.TestSequence) ... ok
test_free_after_iterating (test.test_deque.TestSequence) ... skipped "Exhausted deque iterator doesn't free a deque"
test_getitem (test.test_deque.TestSequence) ... ok
test_getitemoverwriteiter (test.test_deque.TestSequence) ... ok
test_getslice (test.test_deque.TestSequence) ... ok
test_iadd (test.test_deque.TestSequence) ... ok
test_imul (test.test_deque.TestSequence) ... ok
test_index (test.test_deque.TestSequence) ... ok
test_len (test.test_deque.TestSequence) ... ok
test_minmax (test.test_deque.TestSequence) ... ok
test_pickle (test.test_deque.TestSequence) ... ok
test_repeat (test.test_deque.TestSequence) ... ok
test_subscript (test.test_deque.TestSequence) ... ok
test_truth (test.test_deque.TestSequence) ... ok

======================================================================
ERROR: test_sizeof (test.test_deque.TestBasic)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Python39\lib\test\test_deque.py", line 781, in test_sizeof
    check(deque(), basesize + blocksize)
  File "C:\Python39\lib\test\support\__init__.py", line 1483, in check_sizeof
    import _testinternalcapi
ModuleNotFoundError: No module named '_testinternalcapi'

----------------------------------------------------------------------

Ran 79 tests in 0.696s

FAILED (errors=1, skipped=1)
test test_deque failed
test_deque failed

== Tests result: FAILURE ==

1 test failed:
    test_deque

Total duration: 906 ms
Tests result: FAILURE

C:\Python39>

At a guess, it looks like we're not including that module in the Windows installer.  And indeed, it looks like it's left out of Tools/msi/test/test_files.wxs if you would like to provide a PR :)

On the other hand, I'm not sure that test (and any other depending on that module) shouldn't be skipped if _testinternalcapi is missing.

== Tests result: FAILURE ==

357 tests OK.

24 tests failed:
    test__locale test_array test_buffer test_capi test_deque
    test_exceptions test_flufl test_grammar test_itertools test_locale
    test_memoryio test_ordered_dict test_parser test_pickle
    test_positional_only_arg test_string_literals test_struct
    test_syntax test_sys test_traceback test_urllib test_urllib2
    test_urllib2_localnet test_xml_etree_c

43 tests skipped:
    test_asdl_parser test_check_c_globals test_clinic test_curses
    test_dbm_gnu test_dbm_ndbm test_devpoll test_epoll test_fcntl
    test_fork1 test_gdb test_grp test_ioctl test_kqueue
    test_multiprocessing_fork test_multiprocessing_forkserver test_nis
    test_openpty test_ossaudiodev test_pipes test_poll test_posix
    test_pty test_pwd test_readline test_resource test_smtpnet
    test_socketserver test_spwd test_syslog test_threadsignals
    test_timeout test_tix test_tk test_ttk_guionly test_urllib2net
    test_urllibnet test_wait3 test_wait4 test_winsound test_xmlrpc_net
    test_xxtestfuzz test_zipfile64

Zachary Ware <report@bugs.python.org> 于2020年5月26日周二 上午3:33写道：

>
> Zachary Ware <zachary.ware@gmail.com> added the comment:
>
> At a guess, it looks like we're not including that module in the Windows
> installer.  And indeed, it looks like it's left out of
> Tools/msi/test/test_files.wxs if you would like to provide a PR :)
>
> On the other hand, I'm not sure that test (and any other depending on that
> module) shouldn't be skipped if _testinternalcapi is missing.
>
> ----------
> components: +Installation, Windows
> nosy: +paul.moore, steve.dower, tim.golden, zach.ware
> versions: +Python 3.10
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue40754>
> _______________________________________
>

I verified that running tests for installed 64-bit 3.9.0b1 on Win10 is somewhat a joke.  I got 22 failures.  Multiple tests failed on missing _testinternalcapi, and at least 1 on _xxtestfuzz.  Several tests had multiple failures and would need to be run individually and verbosely to see why.  For instance, test_itertools tries to run test.support.__init__.checksizeof 4 times, and the latter imports _testinternalcapi, resulting is 4 failures.

Release managers: I think installers should be run and the suite run with the result.  This is not the first time a file has been omitted.  One needed by IDLE was once omitted, I believe on Windows.  There is now a test that reads it, but is useless unless run.  (And I should add checks for all the other 'not .py' files.)

honglei: Thank you for both posts.  Next time, please remove the quoted message when replying by email.  Once posted on the web page, below the quoted message, the quote becomes redundant noise.

We have a bunch of buildbots that test the installed Python. For instance, if you search for "installed" in https://buildbot.python.org/all/#/builders?tags=%2B3.x we have:

aarch64 Fedora Rawhide Clang Installed 3.x	
aarch64 Fedora Stable Clang Installed 3.x	
AMD64 Fedora Rawhide Clang Installed 3.x	
AMD64 Fedora Stable Clang Installed 3.x
PPC64LE Fedora Rawhide Clang Installed 3.x
PPC64LE Fedora Stable Clang Installed 3.x
s390x Fedora Clang Installed 3.x	
s390x Fedora Rawhide Clang Installed 3.x
x86 Gentoo Installed with X

So it seems that only Linux has "installed" builbot tests. I suppose the next step is adding some macOS and Windows ones.

> So it seems that only Linux has "installed" builbot tests. I suppose the next step is adding some macOS and Windows ones.

Having more buildbots test from "installed" locations, rather than the build directory, is good.

But that's not the whole issue here. Those "installed" tests are only testing that a Python runs from an installed location on the same system it was built.  The Windows and macOS installers use various packaging methods to provide complete Python installations, including binaries, that can be installed on other systems using various OS-native install methods familiar to users of each platform. And the binaries are generally built in such a way to support running on multiple release versions of its operating system. For example, the current Python installers for macOS from python.org are usable on macOS 10.9 through 10.15. To produce the various installer packages, there are other layers of tooling involved, say, on top of "make install" in the Unixy build process for macOS; something similar is true for Windows releases.

FWIW, every macOS installer is test installed on a vanilla system, the whole standard test suite is run, and an IDLE smoke test is run to ensure there are no noticeable regressions before handing the build artifacts off to the release manager. Every installer is tested on at least one supported macOS version and usually more than one.

There are non-binding CI tests run with Github Actions (post-merge) that test from a layout, and last I checked they were all passing. Nobody is really monitoring it but me though, as it isn't a buildbot. 

The Windows installer also runs pip and IDLE tests before being published, though not a full test suite.

Honestly, it's pretty quick to generate an install layout for Windows now I have the PC/layout script. We could do it for every test run if we wanted, though then a number of build-tree only tests would be skipped.

> The Windows installer also runs pip and IDLE tests before being published [...]

That's good and reminds me that I forgot to mention that for the macOS installer, besides the already mentioned tests, we also download, install, and smoke test a third-party wheel from PyPI using the bundled pip and we also do a visual inspection of the release's changelog by launching through IDLE the docset bundled with each installer. If the Python binaries included with the installer are built for multiple CPU architectures (macOS universal binaries), we would also attempt to ensure that the test suite is run at least once on each architecture (as of several months ago, we no longer regularly distribute such binaries but there's a good chance we may need to do so again in the not-too-distant future, for example, if Apple introduces macOS machines based on ARM CPUs).

c:\Python39>python -m test -j4
== CPython 3.9.0b2 (tags/v3.9.0b2:0b1a1c8, Jun 8 2020, 22:55:31) [MSC v.1924 64 bit (AMD64)]
== Windows-10-10.0.18362-SP0 little-endian
== cwd: C:\Users\jhong\AppData\Local\Temp\test_python_5960
== CPU count: 4
== encodings: locale=cp936, FS=utf-8
0:00:00 Run tests in parallel using 4 child processes

== Tests result: FAILURE ==

358 tests OK.

24 tests failed:
    test__locale test_array test_buffer test_capi test_deque
    test_exceptions test_flufl test_fstring test_grammar
    test_itertools test_locale test_memoryio test_ordered_dict
    test_parser test_pickle test_positional_only_arg
    test_string_literals test_struct test_syntax test_sys
    test_traceback test_urllib test_urllib2 test_xml_etree_c

43 tests skipped:
    test_asdl_parser test_check_c_globals test_clinic test_curses
    test_dbm_gnu test_dbm_ndbm test_devpoll test_epoll test_fcntl
    test_fork1 test_gdb test_grp test_ioctl test_kqueue
    test_multiprocessing_fork test_multiprocessing_forkserver test_nis
    test_openpty test_ossaudiodev test_pipes test_poll test_posix
    test_pty test_pwd test_readline test_resource test_smtpnet
    test_socketserver test_spwd test_syslog test_threadsignals
    test_timeout test_tix test_tk test_ttk_guionly test_urllib2net
    test_urllibnet test_wait3 test_wait4 test_winsound test_xmlrpc_net
    test_xxtestfuzz test_zipfile64

Total duration: 6 min 44 sec
Tests result: FAILURE

Still missing _testinternalcapi module for Python3.9rc2
like https://bugs.python.org/issue36635

C:\Users\Admin>c:\python39\python.exe -m test test_deque
0:00:00 Run tests sequentially
0:00:00 [1/1] test_deque
test test_deque failed -- Traceback (most recent call last):
  File "c:\python39\lib\test\test_deque.py", line 781, in test_sizeof
    check(deque(), basesize + blocksize)
  File "c:\python39\lib\test\support\__init__.py", line 1484, in check_sizeof
    import _testinternalcapi
ModuleNotFoundError: No module named '_testinternalcapi'

test_deque failed

== Tests result: FAILURE ==

1 test failed:
    test_deque

Total duration: 1.1 sec
Tests result: FAILURE

New changeset 9b6934230c35e24d8582ea8c58456fa8eab72ae2 by Steve Dower in branch 'master':
bpo-40754: Adds _testinternalcapi to Windows installer for test suite (GH-23271)
https://github.com/python/cpython/commit/9b6934230c35e24d8582ea8c58456fa8eab72ae2

New changeset 8a4557240b98c322d611bfbba3ea51eac3fb841a by Miss Islington (bot) in branch '3.8':
bpo-40754: Adds _testinternalcapi to Windows installer for test suite (GH-23271)
https://github.com/python/cpython/commit/8a4557240b98c322d611bfbba3ea51eac3fb841a

New changeset faadc52e755cdb316a53f3db5aa11cb97f1c4b87 by Miss Islington (bot) in branch '3.9':
bpo-40754: Adds _testinternalcapi to Windows installer for test suite (GH-23271)
https://github.com/python/cpython/commit/faadc52e755cdb316a53f3db5aa11cb97f1c4b87