This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Calling `Multiprocessing.Queue.close()` too quickly causes intermittent failure (BrokenPipeError)
Type: Stage:
Components: Documentation, Library (Lib) Versions: Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: charmonium, docs@python, jdogzz-g5, maggyero
Priority: normal Keywords:

Created on 2019-01-28 20:28 by charmonium, last changed 2022-04-11 14:59 by admin.

Messages (1)
msg334490 - (view) Author: Samuel Grayson (charmonium) * Date: 2019-01-28 20:28
If all processes try to close the Queue immediately after someone has written to it, this causes [an error][1] (see the link for more details). Uncommenting any of the `time.sleep`s makes it work consistently again.

    import multiprocessing
    import time
    import logging
    import multiprocessing.util
    multiprocessing.util.log_to_stderr(level=logging.DEBUG)
    
    queue = multiprocessing.Queue(maxsize=10)
    
    def worker(queue):
        queue.put('abcdefghijklmnop')
    
        # "Indicate that no more data will be put on this queue by the
        # current process." --Documentation
        # time.sleep(0.01)
        queue.close()
    
    proc = multiprocessing.Process(target=worker, args=(queue,))
    proc.start()
    
    # "Indicate that no more data will be put on this queue by the current
    # process." --Documentation
    # time.sleep(0.01)
    queue.close()
    
    proc.join()

Perhaps this is because I am not understanding the documentation correctly, but in that case I would contend this is a documentation bug.

    Traceback (most recent call last):
      File "/usr/lib/python3.7/multiprocessing/queues.py", line 242, in _feed
        send_bytes(obj)
      File "/usr/lib/python3.7/multiprocessing/connection.py", line 200, in send_bytes
        self._send_bytes(m[offset:offset + size])
      File "/usr/lib/python3.7/multiprocessing/connection.py", line 404, in _send_bytes
        self._send(header + buf)
      File "/usr/lib/python3.7/multiprocessing/connection.py", line 368, in _send
        n = write(self._handle, buf)
    BrokenPipeError: [Errno 32] Broken pipe

[1]: https://stackoverflow.com/q/51680479/1078199
History
Date User Action Args
2022-04-11 14:59:10adminsetgithub: 80025
2022-03-14 23:00:30maggyerosetnosy: + maggyero
2021-12-15 20:46:45jdogzz-g5setnosy: + jdogzz-g5
2019-01-28 20:28:52charmoniumcreate