This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author akhi singhania
Recipients akhi singhania
Date 2018-10-31.11:16:38
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1540984598.88.0.788709270274.issue35122@psf.upfronthosting.co.za>
In-reply-to
Content
I am not sure if this is an implementation bug or just a documentation bug.  When using the multiprocessing module, I have come across a scenario where the process fails to exit when it throws an unhandled exception because it is waiting for the feeder thread to join forever.  Sending SIGINT doesn't cause the process to exit either but sending SIGTERM does cause it to exit.

I have attached a simple reproducer.

When main() raises the unhandled exception, the process does not exit.  However, if the size of data that is enqueued is reduced or the child process closes the queue on exiting, then the process exits fine.  

In the scenario, when the process exits successfully, I see the following output:

**** creating queue
[DEBUG/MainProcess] created semlock with handle 140197742751744
[DEBUG/MainProcess] created semlock with handle 140197742747648
[DEBUG/MainProcess] created semlock with handle 140197742743552
[DEBUG/MainProcess] Queue._after_fork()
**** created queue
**** creating process
**** starting process
**** started process
**** starting enqueue
[DEBUG/MainProcess] Queue._start_thread()
[DEBUG/MainProcess] doing self._thread.start()
[DEBUG/Process-1] Queue._after_fork()
[INFO/Process-1] child process calling self.run()
[DEBUG/MainProcess] starting thread to feed data to pipe
[DEBUG/MainProcess] ... done self._thread.start()
**** done enqueue
**** starting sleep
**** done sleep
Traceback (most recent call last):
  File "example.py", line 58, in <module>
    main()
  File "example.py", line 54, in main
    raise Exception('foo')
Exception: foo
[INFO/MainProcess] process shutting down
[DEBUG/MainProcess] running all "atexit" finalizers with priority >= 0
[DEBUG/MainProcess] telling queue thread to quit
[DEBUG/MainProcess] running the remaining "atexit" finalizers
[DEBUG/MainProcess] joining queue thread
[DEBUG/MainProcess] feeder thread got sentinel -- exiting
[DEBUG/MainProcess] ... queue thread joined


In the scenario when the process does not exit successfully, I see the following output:

**** creating queue
[DEBUG/MainProcess] created semlock with handle 139683574689792
[DEBUG/MainProcess] created semlock with handle 139683574685696
[DEBUG/MainProcess] created semlock with handle 139683574681600
[DEBUG/MainProcess] Queue._after_fork()
**** created queue
**** creating process
**** starting process
**** started process
**** starting enqueue
[DEBUG/MainProcess] Queue._start_thread()
[DEBUG/MainProcess] doing self._thread.start()
[DEBUG/Process-1] Queue._after_fork()
[INFO/Process-1] child process calling self.run()
[DEBUG/MainProcess] starting thread to feed data to pipe
[DEBUG/MainProcess] ... done self._thread.start()
**** done enqueue
**** starting sleep
**** done sleep
Traceback (most recent call last):
  File "example.py", line 58, in <module>
    main()
  File "example.py", line 54, in main
    raise Exception('foo')
Exception: foo
[INFO/MainProcess] process shutting down
[DEBUG/MainProcess] running all "atexit" finalizers with priority >= 0
[DEBUG/MainProcess] telling queue thread to quit
[DEBUG/MainProcess] running the remaining "atexit" finalizers
[DEBUG/MainProcess] joining queue thread
<<<< Process hangs here >>>>



I found the "solution" of closing the queue in the child by trial and error and looking through the code.  The current documentation suggests that multiprocessing.Queue.close() and multiprocessing.Queue.join_thread() are "usually unnecessary for most code".  I am not sure if the attached code can be classified as normal code.  I believe that at the very least, the documentation should be updated or maybe it should be investigated if some code changes can address this.
History
Date User Action Args
2018-10-31 11:16:38akhi singhaniasetrecipients: + akhi singhania
2018-10-31 11:16:38akhi singhaniasetmessageid: <1540984598.88.0.788709270274.issue35122@psf.upfronthosting.co.za>
2018-10-31 11:16:38akhi singhanialinkissue35122 messages
2018-10-31 11:16:38akhi singhaniacreate