Author vstinner
Recipients Johan Dahlin, db3l, emilyemorehouse, eric.snow, koobs, nascheme, ncoghlan, pmpp, serhiy.storchaka, shprotx, steve.dower, vstinner, yselivanov
Date 2019-06-03.16:40:31
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1559580031.59.0.10279239492.issue33608@roundup.psfhosted.org>
In-reply-to
Content
My bet is that multiprocessing triggers a bug in daemon threads. Multiprocessing is using and abusing daemon threads. For example, as I wrote, test_venv misuses the multiprocessing API and implicitly let Python release (or maybe not!) "resources" where "resources" can be processes and threads, likely including daemon threads.

Eric: I would love to give you access to a VM where I can reproduce the bug but... I failed to reproduce the bug today :-(

I would suggest to write a stress test for daemon threads:

* spawn daemon threads: 4x times threads than CPU to stress the system
* each thread would sleep a random number of milliseconds and then execute a few pure Python instructions
* spawns these threads, wait a random number of milliseconds, and then "exit Python"

The race condition is that daemon threads may or maybe run during Python finalization depending on the delay.

Maybe you can make the crash more likely by adding a sleep of a few *seconds* before the final exit. For example, at the exit of Py_RunMain().

Last time I saw the crash, it was a thread which uses a Python structure whereas the memory of the structure was freed (filled with a random pattern by debug hooks on memory allocators).

Ah, either use a debug build of Python, or use PYTHONMALLOC=debug, or -X dev command line option, to fill freed memory with a specific pattern, to trigger the crash.

... Good luck.
History
Date User Action Args
2019-06-03 16:40:31vstinnersetrecipients: + vstinner, nascheme, db3l, ncoghlan, pmpp, eric.snow, serhiy.storchaka, yselivanov, koobs, steve.dower, emilyemorehouse, Johan Dahlin, shprotx
2019-06-03 16:40:31vstinnersetmessageid: <1559580031.59.0.10279239492.issue33608@roundup.psfhosted.org>
2019-06-03 16:40:31vstinnerlinkissue33608 messages
2019-06-03 16:40:31vstinnercreate