This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author gregory.p.smith
Recipients bquinlan, cagney, gregory.p.smith, hroncok, hugh, josh.r, jwilk, pablogsal, pitrou, vstinner
Date 2019-04-17.20:07:40
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1555531660.35.0.284791851378.issue35866@roundup.psfhosted.org>
In-reply-to
Content
I am unable to get cf-deadlock.py to hang on my own builds of pure CPython 3.7.2+ d7cb2034bb or 3.6.8+ be77fb7a6e (versions i had in a local git clone).

which specific python builds are seeing the hang using?  Which specific platform/distro version?  "3.7.2" isn't enough, if you are using a distro supplied interpreter please try and reproduce this with a build from the CPython tree itself.  distros always apply their own patches to their own interpreters.

...

Do realize that while working on this it is fundamentally *impossible* per POSIX for os.fork() to be safely used at the Python level in a process also using pthreads.  That this _ever_ appeared to work is a pure accident of implementations of underlying libc, malloc, system libraries, and kernel behaviors.  POSIX considers it undefined behavior.  Nothing done in CPython can avoid that.  Any "fix" for these kinds of issues is merely working around the inevitable which will re-occur.

concurrent.futures.ProcessPoolExecutor uses multiprocessing for its process management.  As of 3.7 ProcessPoolExecutor accepts a mp_context parameter to specify the multiprocessing start method.  Alternatively the default appears
to be controllable as a global setting https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods.

Use the 'spawn' start method and the problem should go away as it'll no longer be misusing os.fork().  You _might_ be able to get the 'forkserver' start method to work, but only reliably if you make sure the forkserver is spawned _before_ any threads in the process (such as ProcessPoolExecutor's own queue management thread - which appears to be spawned upon the first call to .submit()).
History
Date User Action Args
2019-04-17 20:07:40gregory.p.smithsetrecipients: + gregory.p.smith, bquinlan, pitrou, vstinner, jwilk, josh.r, cagney, hroncok, pablogsal, hugh
2019-04-17 20:07:40gregory.p.smithsetmessageid: <1555531660.35.0.284791851378.issue35866@roundup.psfhosted.org>
2019-04-17 20:07:40gregory.p.smithlinkissue35866 messages
2019-04-17 20:07:40gregory.p.smithcreate