This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author eric.snow
Recipients eric.snow, methane, mocramis
Date 2019-03-29.18:50:39
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1553885439.28.0.23680438891.issue36469@roundup.psfhosted.org>
In-reply-to
Content
At this point I think it's likely that the problem relates to how daemon threads are handled during runtime finalization.

What normally happens in the main thread of the "python3" executable is this:

1. Python runtime initializes
2. main interpreter initializes
3. the requested code is executed (in this case via PyRun_SimpleFileExFlags)
4. the runtime is finalized
   a. wait for all non-daemon threads to finish
   b. call registered atexit funcs
   c. mark the runtime as finalizing
   d. ...

From the stack trace you gave, the main thread is definitely past step 4c in the runtime finalization process.

Note the following:

* marking the runtime as finalizing is meant to cause all remaining daemon threads to exit the next time they take the GIL
* further, runtime finalization assumes that all daemon threads will have finished soon after step 4c
* not all C-API functions for acquiring the GIL actually cause the current thread to exit if the runtime is finalizing (see below)
* step 4a applies only to the main interpreter; using subinterpreters complicates the situation a little (threads get finalized with each subinterpreter later on)
* any thread created in an atexit func (4b) can cause problems

Cause thread to exit if runtime is finalizing:

* PyEval_RestoreThread()
* PyEval_EvalFrameEx()  (the eval loop)

Do not cause thread to exit if runtime is finalizing:

* PyEval_InitThreads()
* PyEval_ReInitThreads()
* PyEval_AcquireLock()
* PyEval_AcquireThread()

Regardless, from what you've reported it looks like the following is happening:

m1. main thread starts
m2. thread A (daemon) created
m3. thread B (daemon) created (by main thread or thread A)
m4. code in main thread raises SystemExit
m5. the Python runtime begins finalization (incl. steps 4a-4c above)
m6. the main thread starts cleaning up the main interpreter
m7. it starts cleaning up the main interpreter's threads
m8. it acquires the "head" lock
m9. it starts cleaning up the frame tied to one of those threads
m10. a socket object in that frame gets destroyed
m11. a warning is issued (presumably about an unclosed socket)

tB1. thread B (still running) acquires GIL
tB2. it does not release the GIL

m12. creating the warning causes a function to get called (socket.__repr__)
m13. this causes the main thread to try to acquire the GIL
m14. it blocks (waiting for thread B)

tA1. thread A (still running) finishes and starts cleaning itself up
tA2. it tries to acquire the "head" lock
tA3. it blocks (waiting for the main thread)


Notable:

* at step m7 the code assumes that all the interpreter's threads have exited at that point
* with the verbose flag to "python3", the runtime will print a warning if any thread is still running when its (finalizing) interpreter tries to clean it up
History
Date User Action Args
2019-03-29 18:50:39eric.snowsetrecipients: + eric.snow, methane, mocramis
2019-03-29 18:50:39eric.snowsetmessageid: <1553885439.28.0.23680438891.issue36469@roundup.psfhosted.org>
2019-03-29 18:50:39eric.snowlinkissue36469 messages
2019-03-29 18:50:39eric.snowcreate