There are still some problems with asyncio tests hanging on FreeBSD and possibly other platforms.

While these were originally conflated with a hang on AIX, the root causes are different; just the symptoms are the same (the test hangs forever and after exactly one hour the test runner kills the process).

I'm opening a new issue to discuss the issues on platforms other than AIX.

I haven't seen any such hangs. I've seen one test_asyncio failure on FreeBSD 6.4 3.x, which is the following weirdness.

http://buildbot.python.org/all/builders/x86%20FreeBSD%206.4%203.x/builds/4116/steps/test/logs/stdio

test test_asyncio crashed -- Traceback (most recent call last):
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Lib/test/regrtest.py", line 1276, in runtest_inner
    test_runner()
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Lib/test/test_asyncio/__init__.py", line 31, in test_main
    run_unittest(suite())
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Lib/test/test_asyncio/__init__.py", line 21, in suite
    __import__(mod_name)
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Lib/test/test_asyncio/test_base_events.py", line 10, in <module>
    from asyncio import base_events
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Lib/asyncio/__init__.py", line 13, in <module>
    from .futures import *
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Lib/asyncio/futures.py", line 12, in <module>
    from . import events
ImportError: cannot import name 'events'

Sadly there's no more info about this -- does this mean events.py is corrupt? Or is one of the imports in events.py breaking? Then why doesn't it show the failing line?

The original report and build logs were attached to the original issue covering the import of asyncio: #19262

Also attaching here.

The hangs have been sporadic:
http://buildbot.python.org/all/builders/x86%20RHEL%206%203.x/builds/2916/steps/test/logs/stdio
http://buildbot.python.org/all/builders/x86%20Ubuntu%20Shared%203.x/builds/8795/steps/test/logs/stdio

Perhaps Nick can run test_subprocess_close_after_finish in a loop and tell us if the issue still happens.

Hmm, test_signal_handling_while_selecting is a different one. Perhaps koobs can investigate it :)

@antoine, would love some direction/guidance on ways to isolate.

For what it's worth, the incidence of this failure dropped sharply over the subsequent 24 hours after the initial report. Could the relaxation of the timers in 042deeeb703 have had an effect?

Thus far I've checked out default, and ran test_asyncio at -j16 -j32 and -j64 with no reproduction

Correct changeset ID: e042deeeb703

> Thus far I've checked out default, and ran test_asyncio at -j16 -j32
> and -j64 with no reproduction

you can use -F to run the test forever.

I started "strace ./python -m test -vF test_asyncio" running under screen on the RHEL6 buildbot (so it will keep running when I log out in a moment).

If I don't report back the results in the next day or two, ping the issue to remind me to log back in and see if it eventually got stuck.

> I started "strace ./python -m test -vF test_asyncio" running under screen on the RHEL6 buildbot (so it will keep running when I log out in a moment).

Trace also child processes: add -f. And add maybe also timestamps: -tt.

I created the issue #19652: test_asyncio: test_subprocess_send_signal() hangs on buildbot "AMD64 Snow Leop 3.x".

> The original report and build logs were attached to the original issue covering the import of asyncio: #19262
> Also attaching here.

Extract of the most important part:

[382/382] test_asyncio
Timeout (1:00:00)!
Thread 0x0000000805c18000:
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/threading.py", line 290 in wait
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/queue.py", line 167 in get
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/concurrent/futures/thread.py", line 63 in _worker
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/threading.py", line 869 in run
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/threading.py", line 921 in _bootstrap_inner
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/threading.py", line 889 in _bootstrap

Thread 0x0000000801407400:
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/selectors.py", line 265 in select
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/asyncio/base_events.py", line 576 in _run_once
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/asyncio/base_events.py", line 153 in run_forever
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/test/test_asyncio/test_events.py", line 450 in test_signal_handling_while_selecting
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/unittest/case.py", line 571 in run
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/unittest/case.py", line 610 in __call__
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/unittest/suite.py", line 117 in run
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/unittest/suite.py", line 79 in __call__
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/unittest/suite.py", line 117 in run
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/unittest/suite.py", line 79 in __call__
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/unittest/suite.py", line 117 in run
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/unittest/suite.py", line 79 in __call__
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/unittest/runner.py", line 168 in run
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/test/support/__init__.py", line 1661 in _run_suite
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/test/support/__init__.py", line 1695 in run_unittest
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/test/test_asyncio/__init__.py", line 26 in test_main
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/test/regrtest.py", line 1276 in runtest_inner
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/test/regrtest.py", line 965 in runtest
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/test/regrtest.py", line 532 in main
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/test/regrtest.py", line 1560 in main_in_temp_cwd
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/test/regrtest.py", line 1585 in <module>
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/runpy.py", line 73 in _run_code
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/runpy.py", line 160 in _run_module_as_main
Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/runpy.py", line 160, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/runpy.py", line 73, in _run_code
    exec(code, run_globals)
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/test/__main__.py", line 3, in <module>
    regrtest.main_in_temp_cwd()
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/test/regrtest.py", line 1560, in main_in_temp_cwd
    main()
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/test/regrtest.py", line 736, in main
    raise Exception("Child error on {}: {}".format(test, result[1]))
Exception: Child error on test_asyncio: Exit code 1
*** [buildbottest] Error code 1

I looked at last tests of "FreeBSD 6.4 3.x" between builds 4400 and 4432. The builds 4401, 4408, 4416 and 4421 hanged but because of check_interrupted_write_retry() of test_io, but none hanged because of test_asyncio.

I also checked builds 3620..3643 of "x86 RHEL 6 3.x": all succeeded.

Same on "x86 Ubuntu Shared 3.x": no hang between builds 9560 and 9568.

So I consider this issue as fixed.

It was hanging on FreeBSD 7.2 recently:

http://buildbot.python.org/all/builders/x86%20FreeBSD%207.2%203.x/builds/4917/steps/test/logs/stdio

The older FreeBSD systems have broken threading though, so we could just
disable the offending tests.  I would not waste time debugging anything
threading related before FreeBSD 9.0.

> It was hanging on FreeBSD 7.2 recently:
> http://buildbot.python.org/all/builders/x86%20FreeBSD%207.2%203.x/builds/4917/steps/test/logs/stdio

This is a new issue, test_read_pty_output() is a recent feature which came with a new test. The previous blocking test was test_signal_handling_while_selecting(). Please open a new issue.

OK, I have not seen the original issue in any recent builds either.

> It was hanging on FreeBSD 7.2 recently:
> http://buildbot.python.org/all/builders/x86%20FreeBSD%207.2%203.x/builds/4917/steps/test/logs/stdio

Oh, it still hangs on the latest version of the code. I opened #20495 to track this bug.