test_3_join_in_forked_from_thread() of test_threading failed on "x86 Ubuntu Shared 3.x" buildbot:
-----------------------------------
[201/354] test_threading
[41179 refs]
[40407 refs]
[40407 refs]
[40407 refs]
Timeout (1:00:00)!
Thread 0x404218c0:
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/subprocess.py", line 466 in _eintr_retry_call
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/subprocess.py", line 1486 in _try_wait
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/subprocess.py", line 1528 in wait
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_threading.py", line 455 in _run_and_join
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_threading.py", line 518 in test_3_join_in_forked_from_thread
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/case.py", line 387 in _executeTestPart
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/case.py", line 442 in run
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/case.py", line 494 in __call__
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py", line 105 in run
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py", line 67 in __call__
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py", line 105 in run
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py", line 67 in __call__
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support.py", line 1078 in run
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support.py", line 1166 in _run_suite
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support.py", line 1192 in run_unittest
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_threading.py", line 728 in test_main
  File "./Lib/test/regrtest.py", line 1041 in runtest_inner
  File "./Lib/test/regrtest.py", line 835 in runtest
  File "./Lib/test/regrtest.py", line 659 in main
  File "./Lib/test/regrtest.py", line 1619 in <module>
make: *** [buildbottest] Error 1
program finished with exit code 2
elapsedTime=4426.776675
http://www.python.org/dev/buildbot/all/builders/x86%20Ubuntu%20Shared%203.x/builds/3577/steps/test/logs/stdio
-----------------------------------

Code of the test:
----------------------------------
    def _run_and_join(self, script):
        script = """if 1:
            import sys, os, time, threading

            # a thread, which waits for the main program to terminate
            def joiningfunc(mainthread):
                mainthread.join()
                print('end of thread')
                # stdout is fully buffered because not a tty, we have to flush
                # before exit.
                sys.stdout.flush()
        \n""" + script

        p = subprocess.Popen([sys.executable, "-c", script], stdout=subprocess.PIPE)
        rc = p.wait() <~~~ HANG HERE ~~~~
        data = p.stdout.read().decode().replace('\r', '')
        p.stdout.close()
        self.assertEqual(data, "end of main\nend of thread\n")
        self.assertFalse(rc == 2, "interpreter was blocked")
        self.assertTrue(rc == 0, "Unexpected error")

    @unittest.skipUnless(hasattr(os, 'fork'), "needs os.fork()")
    def test_3_join_in_forked_from_thread(self):
        # Like the test above, but fork() was called from a worker thread
        # In the forked process, the main Thread object must be marked as stopped.

        # Skip platforms with known problems forking from a worker thread.
        # See http://bugs.python.org/issue3863.
        if sys.platform in ('freebsd4', 'freebsd5', 'freebsd6', 'netbsd5',
                           'os2emx'):
            raise unittest.SkipTest('due to known OS bugs on ' + sys.platform)
        script = """if 1:
            main_thread = threading.current_thread()
            def worker():
                childpid = os.fork()
                if childpid != 0:
                    os.waitpid(childpid, 0)
                    sys.exit(0)

                t = threading.Thread(target=joiningfunc,
                                     args=(main_thread,))
                print('end of main')
                t.start()
                t.join() # Should not block: main_thread is already stopped

            w = threading.Thread(target=worker)
            w.start()
            """
        self._run_and_join(script)
----------------------------------

test_2_join_in_forked_process fails on FreeBSD 6.4 buildbot.
http://www.python.org/dev/buildbot/all/builders/x86 FreeBSD 6.4 3.x/builds/1606/steps/test/logs/stdio

"""
======================================================================
FAIL: test_2_join_in_forked_process (test.test_threading.ThreadJoinOnShutdown)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Lib/test/test_threading.py", line 464, in test_2_join_in_forked_process
    self._run_and_join(script)
  File "/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Lib/test/test_threading.py", line 436, in _run_and_join
    self.assertEqual(data, "end of main\nend of thread\n")
AssertionError: '' != 'end of main\nend of thread\n'
+ end of main
+ end of thread
"""

I think it's the same problem as issue #12316: in the child process, even calling pthread_create can segfault/abort on FreeBSD6 (async-safe blahblah...).
Tests creating a thread from a fork()ed process should be skipped on FreeBSD6.
Patch attached, along with some refactoring to use the skipIf idiom.

As for test_3_join_in_forked_from_thread, well, it could be more or less the same problem. We're really doing something prohibited by POSIX, so things might break in unexpected ways. For example, calling pthread_condition_destroy from the child process can deadlock (see http://bugs.python.org/issue6721#msg136047).

Victor: to debug this kind of problem, it would be great if faulthandler could also dump tracebacks of children processes. Do you mind if I create a new issue?

> Victor: to debug this kind of problem, it would be great
> if faulthandler could also dump tracebacks of children processes.
> Do you mind if I create a new issue?

Please open a new issue.

+    @unittest.skipIf(sys.platform in ('freebsd4', 'freebsd5', 'freebsd6',
+                     'netbsd5', 'os2emx'), "due to known OS bug")

This skip gives very few information, and it is duplicated for each function. I would prefer a constant of the "broken OSes" with your following comment attached to the constant:

+    # Between fork() and exec(), only async-safe functions are allowed (issues
+    # #12316 and #11870), and fork() from a worker thread is known to trigger
+    # problems with some operating systems: skip problematic tests on platforms
+    # known to behave badly.

Or split the test case into two testcases: one using fork and skipped on broken platforms, one not using fork?

---

> As for test_3_join_in_forked_from_thread, well, it could be more
> or less the same problem. We're really doing something prohibited
> by POSIX, so things might break in unexpected ways.

If the creation of a thread after a fork is reliable on some systems, we should not deny the creation of new threads after a fork. You can replace "creation of new threads" by any other non async-safe function in my previous sentence. Therefore I agree that the good answer to this issue is to skip the test on "broken systems" (or should we call them "POSIX compliant systems?" :-)).

> This skip gives very few information, and it is duplicated for each 
> function. I would prefer a constant of the "broken OSes" with your 
> following comment attached to the constant:

Ok, I'll try to write something along those lines.

> If the creation of a thread after a fork is reliable on some systems,
> we should not deny the creation of new threads after a fork.

Well, the problem is that it is not reliable on any platform, but happens to work "most of the time" on some platforms ;-)

Patch attached.

Your patch is linux3 compliant, go ahead!

New changeset 0ed5e6ff10f8 by Victor Stinner in branch '3.2':
Issue #11870: Skip test_threading.test_2_join_in_forked_process() on platforms
http://hg.python.org/cpython/rev/0ed5e6ff10f8

New changeset f43dee86fffd by Victor Stinner in branch 'default':
(merge 3.2) Issue #11870: Skip test_threading.test_2_join_in_forked_process()
http://hg.python.org/cpython/rev/f43dee86fffd

New changeset ff36b8cadfd6 by Victor Stinner in branch '2.7':
Issue #11870: Skip test_threading.test_2_join_in_forked_process() on platforms
http://hg.python.org/cpython/rev/ff36b8cadfd6

The initial problem was test_3_join_in_forked_from_thread() and the hangs does still happen:

[324/356] test_threading
Timeout (1:00:00)!
Thread 0x404248c0:
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/subprocess.py", line 1498 in _communicate_with_poll
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/subprocess.py", line 1423 in _communicate
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/subprocess.py", line 836 in communicate
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/script_helper.py", line 32 in _assert_python
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/script_helper.py", line 50 in assert_python_ok
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_threading.py", line 434 in _run_and_join
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_threading.py", line 493 in test_3_join_in_forked_from_thread
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/case.py", line 407 in _executeTestPart
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/case.py", line 462 in run
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/case.py", line 514 in __call__
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py", line 105 in run
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py", line 67 in __call__
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py", line 105 in run
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py", line 67 in __call__
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/runner.py", line 168 in run
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support.py", line 1259 in _run_suite
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support.py", line 1285 in run_unittest
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_threading.py", line 774 in test_main
  File "./Lib/test/regrtest.py", line 1070 in runtest_inner
  File "./Lib/test/regrtest.py", line 861 in runtest
  File "./Lib/test/regrtest.py", line 669 in main
  File "./Lib/test/regrtest.py", line 1648 in <module>

http://www.python.org/dev/buildbot/all/builders/x86%20Ubuntu%20Shared%203.x/builds/4081/steps/test/logs/stdio

(neologix's patch doesn't change anything for x86 Ubuntu Shared 3.x buildbot, which is a Linux, not a FreeBSD)

I don't know why it only hangs on this Linux buildbot. It's maybe an old Linux kernel, an old GNU libc version, or something like that?

> The initial problem was test_3_join_in_forked_from_thread() and the hangs does still happen:
>

Yes, the patch was there to fix test_2_join_in_forked_from_thread.

> [324/356] test_threading
> Timeout (1:00:00)!
> Thread 0x404248c0:
>  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/subprocess.py", line 1498 in _communicate_with_poll
>  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/subprocess.py", line 1423 in _communicate
>  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/subprocess.py", line 836 in communicate
>  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/script_helper.py", line 32 in _assert_python
>  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/script_helper.py", line 50 in assert_python_ok
>  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_threading.py", line 434 in _run_and_join
>  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_threading.py", line 493 in test_3_join_in_forked_from_thread
>  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/case.py", line 407 in _executeTestPart
>  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/case.py", line 462 in run
>  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/case.py", line 514 in __call__
>  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py", line 105 in run
>  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py", line 67 in __call__
>  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py", line 105 in run
>  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py", line 67 in __call__
>  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/runner.py", line 168 in run
>  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support.py", line 1259 in _run_suite
>  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support.py", line 1285 in run_unittest
>  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_threading.py", line 774 in test_main
>  File "./Lib/test/regrtest.py", line 1070 in runtest_inner
>  File "./Lib/test/regrtest.py", line 861 in runtest
>  File "./Lib/test/regrtest.py", line 669 in main
>  File "./Lib/test/regrtest.py", line 1648 in <module>
>
> http://www.python.org/dev/buildbot/all/builders/x86%20Ubuntu%20Shared%203.x/builds/4081/steps/test/logs/stdio
>

This means that the subprocess hangs, but without a backtrace of the
child process (issue #12413), it's hard to analyse it further.
I've had a look at the code, and couldn't find anything obviously
wrong; Gregory's patches to sanitize threading's lock should have
fixed this. I also tried running this test in a loop for 48 hours but
couldn't reproduce it.
One possible explanation (but it's just a wild guess) is that with a
certain kernel/libc configuration, the lock deallocation code can
deadlock (I've seen pthread_cond_destroy() block, this could maybe
happen with sem_destroy()).

So I suggest to try to come up with a solution to #12413, which should
help analyzing this - and similar - issues.

> Gregory's patches to sanitize threading's lock should have fixed this

The subprocess hang still occurs something, it just happened:

http://www.python.org/dev/buildbot/all/builders/x86%20Ubuntu%20Shared%203.x/builds/4898/steps/test/logs/stdio

[110/363] test_threading
Timeout (1:00:00)!
Thread 0x404888c0:
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/subprocess.py", line 1513 in _communicate_with_poll
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/subprocess.py", line 1438 in _communicate
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/subprocess.py", line 850 in communicate
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/script_helper.py", line 32 in _assert_python
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/script_helper.py", line 50 in assert_python_ok
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_threading.py", line 441 in _run_and_join
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_threading.py", line 497 in test_3_join_in_forked_from_thread
...

I removed my two previous message (msg148991 and msg148992), there were unrelated to this issue: the test hangs in debug mode in the IO module because of a deadleak in IO related to the fork...

To debug this, we should probably make use of faulthandler (but not
dump_tracebacks_later, since it creates a new thread). The way to go
could be to make the parent process send a fatal signal to the child
process if the latter takes too long to complete.

Here's a patch to help nail this down.

Victor, could you try the patch attached?

New changeset 775319cebaa3 by Charles-François Natali in branch '2.7':
Issue #11870: threading: Properly reinitialize threads internal locks and
http://hg.python.org/cpython/rev/775319cebaa3

New changeset de962ec0faaa by Charles-François Natali in branch '3.2':
Issue #11870: threading: Properly reinitialize threads internal locks and
http://hg.python.org/cpython/rev/de962ec0faaa

New changeset cec0d77d01c4 by Charles-François Natali in branch 'default':
Issue #11870: threading: Properly reinitialize threads internal locks and
http://hg.python.org/cpython/rev/cec0d77d01c4

Should be fixed now.

New changeset cd54b48946ca by Stefan Krah in branch '3.3':
Issue #11870: Skip test_3_join_in_forked_from_thread() on HP-UX.
http://hg.python.org/cpython/rev/cd54b48946ca