classification
Title: Failures and crashes when running tests by import
Type: behavior Stage:
Components: Tests Versions: Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: Mariatta, denis-osipov, docs@python, serhiy.storchaka, terry.reedy, vstinner
Priority: normal Keywords:

Created on 2017-10-11 16:55 by denis-osipov, last changed 2017-10-20 20:46 by serhiy.storchaka.

Messages (14)
msg304163 - (view) Author: Denis Osipov (denis-osipov) * Date: 2017-10-11 16:55
In Developer Guide says:

"If you don’t have easy access to a command line, you can run the test suite from a Python or IDLE shell:

>>> from test import autotest"

But I can't run test from IDLE:

Traceback (most recent call last):
  File "<pyshell#0>", line 1, in <module>
    from test import autotest
  File "D:\repos\cpython\Lib\test\autotest.py", line 5, in <module>
    main()
  File "D:\repos\cpython\Lib\test\libregrtest\main.py", line 585, in main
    Regrtest().main(tests=tests, **kwargs)
  File "D:\repos\cpython\Lib\test\libregrtest\main.py", line 510, in main
    self._main(tests, kwargs)
  File "D:\repos\cpython\Lib\test\libregrtest\main.py", line 524, in _main
    setup_tests(self.ns)
  File "D:\repos\cpython\Lib\test\libregrtest\setup.py", line 18, in setup_tests
    faulthandler.enable(all_threads=True)
io.UnsupportedOperation: fileno

If I understand it correct, this behavior is reasonable (issues 3003 and 25588).
Maybe it's worth to remove words about running from IDLE. Or in case if it's possible to run such tests add some words about it.
msg304225 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-10-12 09:51
> "If you don’t have easy access to a command line, you can run the test suite from a Python or IDLE shell:

The devguide is wrong. You should not run the Python test suite in IDLE. It doesn't work and many tests fail just because of IDLE.

Please run the test suite in a command line (like "cmd.exe" on Windows).
msg304245 - (view) Author: Denis Osipov (denis-osipov) * Date: 2017-10-12 14:41
Got it. Thank you for your help.
msg304270 - (view) Author: Mariatta (Mariatta) * (Python committer) Date: 2017-10-12 17:10
Closing this issue, I opened https://github.com/python/devguide/issues/280 so that Dev Guide can be updated.

Thanks!
msg304273 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-10-12 17:49
See issue25588. Currently running tests from IDLE doesn't work, but it worked in the past and maybe we can make it working again.
msg304295 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-10-12 23:18
Victor, what is your opinion of running 'from test import autotest' in a standard python shell window, opened from an icon, as opposed to IDLE's shell?

>  many tests fail just because of IDLE.

We cannot know what to blame on IDLE without a proper control experiment.  I updated my Win10 repository, rebuilt python 3.7, ran the tests from the command line, and all passed (thank you for making this routine).  I ran autotest in a Python console and saw a new resource warning from one of the async tests and 3 failures.  I cannot say what they were because when the test finished, the window closed (crashed), which is a metafailure.  I will rerun and try to identify the error as they occur.

Should we reopen this issue or open a new issue, to fix problems not due to IDLE, or remove the entire suggestion to use autotest, and remove autotest itself?
msg304296 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-10-12 23:42
0:00:28 [ 25/407] test_asyncio
F:\dev\3x\lib\asyncio\sslproto.py:330: ResourceWarning: unclosed transport <asyncio.sslproto._SSLProtocolTransport object at 0x066D38C0>
  source=self)

test_code_module - multiple errors

test test_importlib failed -- Traceback (most recent call last):
  File "F:\dev\3x\lib\test\test_importlib\test_locks.py", line 134, in test_all_locks
    self.bootstrap._module_locks)
AssertionError: 0 != 1 : {'test.autotest': <weakref at 0x0395D620; to '_ModuleLock' at 0x03957E70>}

test_warnings failed - multiple errors

crash after summary printed.
I am rerunning in python started from command to see if there is any message printed.
msg304298 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-10-13 00:34
The Python console exits because test.libregrtest.main.Regrtest._main calls sys.exit.  test.autotest consists of

from test.libregrtest import main
main()

The main call should be wrapped with try:...except sys.exit: pass

The output also has this variance from the command line result.

0:15:47 [319/407/2] test_subprocess
minkernel\crts\ucrt\src\appcrt\lowio\write.cpp(49) : Assertion failed: (_osfile(fh) & FOPEN)
minkernel\crts\ucrt\src\appcrt\lowio\close.cpp(49) : Assertion failed: (_osfile(fh) & FOPEN)
minkernel\crts\ucrt\src\appcrt\lowio\close.cpp(49) : Assertion failed: (_osfile(fh) & FOPEN)

The list of tests skipped seems to be the same.

By setting sys.argv before importing autotest, I got
======================================================================
FAIL: test_ps1 (test.test_code_module.TestInteractiveConsole)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "F:\dev\3x\lib\test\test_code_module.py", line 35, in test_ps1
    self.assertEqual(self.sysmod.ps1, '>>> ')
AssertionError: <MagicMock name='sys.ps1' id='71914032'> != '>>> '

======================================================================
FAIL: test_ps2 (test.test_code_module.TestInteractiveConsole)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "F:\dev\3x\lib\test\test_code_module.py", line 40, in test_ps2
    self.assertEqual(self.sysmod.ps2, '... ')
AssertionError: <MagicMock name='sys.ps2' id='72189080'> != '... '

----------------------------------------------------------------------

======================================================================
FAIL: test_missing_filename_main_with_argv (test.test_warnings.CWarnTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "F:\dev\3x\lib\test\test_warnings\__init__.py", line 446, in test_missing_filename_main_with_argv
    self.assertEqual(w[-1].filename, sys.argv[0])
AssertionError: '__main__' != ''
- __main__
+

and

======================================================================
FAIL: test_missing_filename_main_with_argv (test.test_warnings.PyWarnTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "F:\dev\3x\lib\test\test_warnings\__init__.py", line 446, in test_missing_filename_main_with_argv
    self.assertEqual(w[-1].filename, sys.argv[0])
AssertionError: '__main__' != ''
- __main__

Wneh I start python with python.bat, sys.argv = [''], not ['__main__'].
msg304300 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-10-13 01:05
When I import autotest in IDLE's shell, test_code_module fails, test_importlib and test_warnings fail, as without IDLE, and test_gc and test_logging are new failures.  (Results are easy to see because SystemExit is caught and ignored.)

test test_gc failed -- Traceback (most recent call last):
  File "F:\dev\3x\lib\test\support\__init__.py", line 1855, in wrapper
    return func(*args, **kwargs)
  File "F:\dev\3x\lib\test\test_gc.py", line 290, in test_get_count
    self.assertLess(a, 5)
AssertionError: 8 not less than 5

<a comes from gc.get_count() after gc.collect().  When I repeat outside of the test, a is 31.  I think either gc or this test is buggy.>


test test_logging failed -- Traceback (most recent call last):
  File "F:\dev\3x\lib\test\test_logging.py", line 1960, in test_warnings
    "dummy.py:42: UserWarning: Explicit\n  Dummy line\n")
AssertionError: '\nWarning (from warnings module):\n  File[57 chars]it\n' != 'dummy.py:42: UserWarning: Explicit\n  Dummy line\n'
+ dummy.py:42: UserWarning: Explicit
- 
- Warning (from warnings module):
-   File "dummy.py", line 42
-     Dummy line
? --
+   Dummy line
- UserWarning: Explicit

IDLE replaces warnings.show_warnings.  In any case, there are 3 failures in Python's shell versus 4 in IDLE's shell.
msg304326 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-10-13 08:33
After starting Python from a command line instead of an icon, importing/running autotest results in the same three failures.
msg304647 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-10-20 02:22
I am re-opening because there is a bug, a conflict between doc and code comment and actual behavior, and one which has nothing to do with IDLE.  The question is whether to fix it or to delete the invitation to invoke the bug.

The reason the devguide gives an alternative for running tests is so that someone with only installed Python and no command line could still test their installation and even contribute a patch on the tracker.

On 2000-08-23, Tim Peters add the following comment to test.autotest:
# This should be equivalent to running regrtest.py from the cmdline.
# It can be especially handy if you're in an interactive shell, e.g.,
# from test import autotest.

The autotest code was later copied into test.__main__, and both are now:
  from test.libregrtest import main
  main()
So 'from test import __main__' is the same as importing autotest, and should be the same as importing and running libregrtest

To test the equivalence in the setting where either import might be useful, I opened both python and IDLE from the 3.7.0a2 Windows start icons and imported autotest.  The test deterministicly crashes after test_marshal.  I call the stoppage a crash because wrapping the import in try: ... except BaseException has no effect.  The Python console just vanishes.  IDLE restarts.
...
0:05:51 [207/407/5] test_marshal

=============================== RESTART: Shell =========================
>>>

So the claimed equivalence in worse than just not true.  There were, I believe, 3 failures in the Python console.  The extra 2 in IDLE are due to the #25588 fix guarding against sys.stderr being None is needed in a couple of other places.

In any case, either running tests by importing should be made to work (again, presuming it once did) or deprecated and autotest removed.
msg304674 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-10-20 19:39
Victor, why do you persist in the nearly irrelevant focus on IDLE?  As I reported above, failures and crashes happen ***without*** involving IDLE.  Serhiy appears to report the same in #31794.
msg304679 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-10-20 20:40
Sorry, I didn't read you comment. I just saw IDLE, I missed that you wrote
that the bug is unrelated to IDLE. Ooops
msg304682 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-10-20 20:46
Perhaps there are issues specific to IDLE, but we should first fix issues not related to IDLE in #31794.
History
Date User Action Args
2017-10-20 20:46:32serhiy.storchakasetmessages: + msg304682
2017-10-20 20:40:43vstinnersetmessages: + msg304679
2017-10-20 19:39:56terry.reedysetmessages: + msg304674
title: Failures and crashes when running tests by import in IDLE -> Failures and crashes when running tests by import
2017-10-20 12:35:25vstinnersettitle: Failures and crashes when running tests by import. -> Failures and crashes when running tests by import in IDLE
2017-10-20 02:22:14terry.reedysetstatus: closed -> open
versions: - Python 3.8
title: regrtest: faulthandler.enable() fails with io.UnsupportedOperation: fileno when run from IDLE -> Failures and crashes when running tests by import.
messages: + msg304647

resolution: not a bug ->
stage: resolved ->
2017-10-13 08:33:08terry.reedysetmessages: + msg304326
2017-10-13 01:05:01terry.reedysetmessages: + msg304300
2017-10-13 00:34:15terry.reedysetmessages: + msg304298
2017-10-12 23:42:47terry.reedysetmessages: + msg304296
2017-10-12 23:18:26terry.reedysetnosy: + terry.reedy
messages: + msg304295
2017-10-12 17:49:05serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg304273
2017-10-12 17:10:46Mariattasetstatus: open -> closed

nosy: + Mariatta
messages: + msg304270

resolution: not a bug
stage: resolved
2017-10-12 14:41:24denis-osipovsetmessages: + msg304245
2017-10-12 09:51:16vstinnersetmessages: + msg304225
2017-10-12 09:16:23vstinnersetcomponents: + Tests, - Documentation
title: Possible error in devguide part about tests -> regrtest: faulthandler.enable() fails with io.UnsupportedOperation: fileno when run from IDLE
2017-10-11 17:18:50serhiy.storchakasetnosy: + vstinner
2017-10-11 16:55:04denis-osipovcreate