New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Run test suite from IDLE idlelib.run subprocess #69774
Comments
This issue proposes stress testing idlelib.run by running code that already reports whether results are as expected or not -- the test_suite. Since this one test will take at least as long as the test suite, and will probably need human intervention anyway, it would not be part of test_idle, but would be run separately manually, like htest. IDLE runs user code in a user process started with subprocess.POpen(executable + 'path/to/idlelib/run.py' + args + user_code_path) (with user_code_path omitted for shell restarts). When run from the editor, the intent is to imitate 'python -i user_code_path'. On python list, someone reported that running a program using pyqt5 (and qt5) hung on exit while running with python3 (on Ubuntu, no -i) worked. I have asked for more details, including whether it works with -i included. In the meanwhile, I wondered ... While the run module might not be able to run programs with certain 3rd party imports, can it at least run programs restricted to stdlib imports and well enough behaved to run with python -i. (A programs that, for instance, disables standard i/o streams would fail both in IDLE and with python.) A test would be to at least run all stdlib tests, with the same result, as run otherwise on the same system. The tests should be run in the user process itself and not in additional processes. They could be run either all in one process or each in a new run process. If successful, I will put code and instructions in a new idlelib/idle_test/rtest.py/txt. |
The premise of the issue is this: if IDLE is started with 'pythonx', then running 'filex' from a IDLE editor is equivalent to running "python x -i filex" at a command line. For instance, loading test.__main__ into an editor and running should give the same result as However,
Traceback (most recent call last):
File "F:\Python\dev\36\Lib\test\__main__.py", line 2, in <module>
main()
File "F:\Python\dev\36\lib\test\libregrtest\main.py", line 508, in main
Regrtest().main(tests=tests, **kwargs)
File "F:\Python\dev\36\lib\test\libregrtest\main.py", line 446, in main
self._main(tests, kwargs)
File "F:\Python\dev\36\lib\test\libregrtest\main.py", line 458, in _main
setup_tests(self.ns)
File "F:\Python\dev\36\lib\test\libregrtest\setup.py", line 18, in setup_tests
faulthandler.enable(all_threads=True)
io.UnsupportedOperation: fileno The pseudofile socket wrappers are not real files with a fileno. I should take a better look as sys.stdout and _stdout (and stderr) when connected to a console, |
I proposed PR 3962 to fix a few issues in regrtest when sys.stdout and sys.stderr have to file descriptor. I'm not sure that we should promote running tests with unusual sys.stdout and sys.stderr, since many tests fail in that case. That's why I chose to not add a NEWS entry to my PR. For example, test_builtin hangs until you write something in IDLE, whereas no prompt is displayed. test_faulthandler fails, and much more tests. I'm not interested to support running the Python test suite in IDLE. Maybe we should even detect that we are running in IDLE and fail with an helpful error message like : "Running the Python test suite inside IDLE is not supported. Run the test suite in a terminal." At least, write a warning in UPPERCASE if sys.stdout or sys.stderr have no file descriptor? |
I think it would be nice to make testing so flexible as possible. This would help to get rid of unintended assumptions. If some tests are failed on IDLE I would prefer to fix or skip only these tests. |
Serhiy: "If some tests are failed on IDLE I would prefer to fix or skip only these tests." Ok, go ahead. There are many failing tests :-) |
As I reported on related bpo-31761, importing autotest on Windows 10 has 3 errors with Python's shell and 4 with IDLE's, with two of those the same as the python shell failures. IDLE can be detected in at least two ways.
>>> import sys; 'idlelib.run' in sys.modules
True
>>> type(sys.stdout).__name__ == 'PseudoOutputFile'
True 'test.autotest' in sys.modules" would detect that test.autotest is being run, whether in a Python or IDLE shell. Running test.__main__ from an editor results in screwy behavior. Running just 'import test.autotest' instead gives the same result. I expected at least the latter to have the same result as when run in the shell. Wrong. First, there is an additional test failure. 0:00:12 [ 18/407] test_aifc
test test_aifc crashed -- Traceback (most recent call last):
File "F:\dev\3x\lib\test\libregrtest\runtest.py", line 163, in runtest_inner
the_module = importlib.import_module(abstest)
File "F:\dev\3x\lib\importlib\__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 680, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "F:\dev\3x\lib\test\test_aifc.py", line 9, in <module>
import aifc
File "F:\dev\3x\lib\aifc.py", line 254, in <module>
from chunk import Chunk
ImportError: cannot import name 'Chunk' from 'chunk' (F:\Python\mypy\chunk.py) Next, the tests run normallu up to Then, while running test_configparser, the user process crashes, IDLE restarts a new user process, and the test suite restarts (in the IDLE process?) from the beginning, sending output to the console, not IDLE, and running each test 5 times. This all is repeatable. The following is the initial part of the console output. == CPython 3.7.0a1+ (heads/master:8c26a34f93, Oct 14 2017, 19:37:37) [MSC v.1900 32 bit (Intel)] I said 'crash' because the run process catches SystemExit.
>>> raise SystemExit
>>>
I have ocassionally encountered unexpected restarts before, but never, that I remember, so easily repeatable. |
The README patch will finish this issue. Fixing individual failures and investigating the crash will be new issues. |
Running the file containing 'import test.autotest' directly with python gives the test_aifc failure and the same bizarre behavior after test_concurrent_futures. Testing restarts apparently in 5 parallel threads or processes. The lines of output are the same though interleaved slightly differently. The only difference is that the number in the 5 build/test_python_nnnn working directories are different. So, not an IDLE bug. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: