This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author rbprogrammer
Recipients davin, pitrou, rbprogrammer
Date 2018-01-21.12:55:23
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1516539327.3.0.467229070634.issue32608@psf.upfronthosting.co.za>
In-reply-to
Content
This is more of an issue with socketserver rather than multiprocessing.  I could not find socketserver experts to add to the nosy list, but I've included the multiprocessing experts for completeness.

Under specific conditions, multiprocessing queues cease to work when modules like socketserver is used, more specifically when fork() is used.  This makes sense that things can break when fork() is used since a simple forked process has things like file descriptors and threads invalidated.  The socketserver.ForkingMixIn class, which has very reasonable use-cases, causes multiprocessing queues and threads to break.  A script that replicates and simulates the issue is attached.

The attached script sets up a multiprocessing queue used for logging across all sub-processes, then uses a socketserver.ForkingTCPServer to handle new client connection.  What's happening is the fork() is invalidating the thread in the QueueHandler in each new sub-process (for each new client conncetion).  It turns the QueueHandler into a dead object.

In this scenario, the only indication the problem exists is the fact that log messages are not actually logged.  But I think it's plausible that more than just log data could be lost.

When a non-modified version of CPython, the script should run and produce a log file.  However, logs from the client connection handler (MyClientHandler) will be lost and never sent to the logging handler that performs the actual file writing.  To exercise the proposed fix to CPython, build the changes to socketserver.py and then change the script's "USE_FIX" flag to "True" and run the script.
History
Date User Action Args
2018-01-21 12:55:27rbprogrammersetrecipients: + rbprogrammer, pitrou, davin
2018-01-21 12:55:27rbprogrammersetmessageid: <1516539327.3.0.467229070634.issue32608@psf.upfronthosting.co.za>
2018-01-21 12:55:27rbprogrammerlinkissue32608 messages
2018-01-21 12:55:25rbprogrammercreate