Author pitrou
Recipients davin, grzgrzgrz3, pitrou, tomMoral, xiang.zhang
Date 2017-06-02.16:52:00
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1496422321.02.0.740107834517.issue30414@psf.upfronthosting.co.za>
In-reply-to
Content
Thomas, thanks for the heads up.  I would suggest something like the following patch to multiprocessing.Pool:

$ git diff
diff --git a/Lib/multiprocessing/queues.py b/Lib/multiprocessing/queues.py
index 7f77837..ebbb360 100644
--- a/Lib/multiprocessing/queues.py
+++ b/Lib/multiprocessing/queues.py
@@ -260,8 +260,16 @@ class Queue(object):
                     info('error in queue thread: %s', e)
                     return
                 else:
-                    import traceback
-                    traceback.print_exc()
+                    self._on_queue_thread_error(e)
+
+    def _on_queue_thread_error(self, e):
+        """
+        Private API called when feeding data in the background thread
+        raises an exception.  For overriding by concurrent.futures.
+        """
+        import traceback
+        traceback.print_exc()
+
 
 _sentinel = object()
 


Then you can write your own Queue subclass in concurrent.futures to handle that error and clean up/restart whatever needs to be cleaned up or restarted.  What do you think?
History
Date User Action Args
2017-06-02 16:52:01pitrousetrecipients: + pitrou, davin, xiang.zhang, tomMoral, grzgrzgrz3
2017-06-02 16:52:01pitrousetmessageid: <1496422321.02.0.740107834517.issue30414@psf.upfronthosting.co.za>
2017-06-02 16:52:01pitroulinkissue30414 messages
2017-06-02 16:52:00pitroucreate