classification
Title: asyncio : ProactorEventLoop raised BlockingIOError when ThreadPoolExecutor has many workers
Type: behavior Stage: resolved
Components: asyncio Versions: Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: gvanrossum, josephgordon, kernel0, miss-islington, vstinner, yselivanov
Priority: normal Keywords: patch, patch, patch

Created on 2015-04-02 04:00 by kernel0, last changed 2019-01-15 13:59 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
example_thread_executor.py kernel0, 2015-04-02 04:00
Pull Requests
URL Status Linked Edit
PR 11566 merged vstinner, 2019-01-15 12:32
PR 11566 merged vstinner, 2019-01-15 12:32
PR 11566 merged vstinner, 2019-01-15 12:32
PR 11567 merged miss-islington, 2019-01-15 12:59
PR 11567 merged miss-islington, 2019-01-15 12:59
PR 11567 merged miss-islington, 2019-01-15 12:59
Messages (6)
msg239872 - (view) Author: SeungHyun.Hwang (kernel0) * Date: 2015-04-02 04:00
I tested on python3.4.2, windows 7

1. Only proactorEvent issued.
2. ThreadPoolExecutor has many workers (in the attached example file, worker count 20,000)
3. The loop does run_in_executor more call than worker count (in the attached example file, 40,000 calls)
4. After some random seconds, raise BlockingIOError

BlockingIOError: [WinError 10035] A non-blocking socket operation could not be completed immediately.
exception calling callback for <Future at 0x1ab89ef0 state=finished returned NoneType>
Traceback (most recent call last):
  File "c:\Python342\Lib\concurrent\futures\_base.py", line 297, in _invoke_callbacks
    callback(self)
  File "c:\Python342\Lib\asyncio\futures.py", line 410, in <lambda>
    new_future._copy_state, fut))
  File "c:\Python342\Lib\asyncio\base_events.py", line 403, in call_soon_threadsafe
    self._write_to_self()
  File "c:\Python342\Lib\asyncio\proactor_events.py", line 449, in _write_to_self
    self._csock.send(b'\0')


I guess that proactor's _write_to_self method misses exception handle.

proactor_events.py
def _write_to_self(self):
        self._csock.send(b'\0')


selector_events.py
def _write_to_self(self):
        # This may be called from a different thread, possibly after
        # _close_self_pipe() has been called or even while it is
        # running.  Guard for self._csock being None or closed.  When
        # a socket is closed, send() raises OSError (with errno set to
        # EBADF, but let's not rely on the exact error code).
        csock = self._csock
        if csock is not None:
            try:
                csock.send(b'\0')
            except OSError:
                if self._debug:
                    logger.debug("Fail to write a null byte into the "
                                 "self-pipe socket",
                                 exc_info=True)


Ps: It's my first publication. Hope you understand my poor comment..
msg256843 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2015-12-22 16:46
Why are you using 20000 threads?

--Guido (mobile)
On Dec 22, 2015 02:03, "Joseph Gordon" <report@bugs.python.org> wrote:

>
> Changes by Joseph Gordon <j.gordon.matthew@gmail.com>:
>
>
> ----------
> nosy: +josephgordon
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue23846>
> _______________________________________
>
msg256881 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2015-12-22 23:18
> Why are you using 20000 threads?

That's a good question.

In any case it looks like self-pipe sock's buffer was overflown because call_soon_threadsafe was called too many times, and loop._read_from_self couldn't empty the buffer promptly.  Then, at some point, _write_to_self failed with an IOError.

It looks like this was fixed for the selector loop, but not for proactor:

selector_events.py:

    def _write_to_self(self):
        csock = self._csock
        if csock is not None:
            try:
                csock.send(b'\0')
            except OSError:
                if self._debug:
                    logger.debug("Fail to write a null byte into the "
                                 "self-pipe socket",
                                 exc_info=True)

proactor_events.py:

    def _write_to_self(self):
        self._csock.send(b'\0')
msg333700 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-01-15 12:58
New changeset c9f872b0bdce5888f1879fa74e098bf4a05430c5 by Victor Stinner in branch 'master':
bpo-23846: Fix ProactorEventLoop._write_to_self() (GH-11566)
https://github.com/python/cpython/commit/c9f872b0bdce5888f1879fa74e098bf4a05430c5
msg333703 - (view) Author: miss-islington (miss-islington) Date: 2019-01-15 13:17
New changeset c9f26714d511a338ba2fdd926e3dc62636f31815 by Miss Islington (bot) in branch '3.7':
bpo-23846: Fix ProactorEventLoop._write_to_self() (GH-11566)
https://github.com/python/cpython/commit/c9f26714d511a338ba2fdd926e3dc62636f31815
msg333704 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-01-15 13:59
> In any case it looks like self-pipe sock's buffer was overflown because call_soon_threadsafe was called too many times, and loop._read_from_self couldn't empty the buffer promptly.  Then, at some point, _write_to_self failed with an IOError.

I fixed the issue. Thanks for your bug report ;-)
History
Date User Action Args
2019-01-15 13:59:09vstinnersetstatus: open -> closed
messages: + msg333704

keywords: patch, patch, patch
resolution: fixed
stage: patch review -> resolved
2019-01-15 13:17:13miss-islingtonsetnosy: + miss-islington
messages: + msg333703
2019-01-15 12:59:26miss-islingtonsetpull_requests: + pull_request11232
2019-01-15 12:59:15miss-islingtonsetpull_requests: + pull_request11231
2019-01-15 12:59:03miss-islingtonsetpull_requests: + pull_request11230
2019-01-15 12:58:43vstinnersetmessages: + msg333700
2019-01-15 12:32:42vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request11229
2019-01-15 12:32:33vstinnersetkeywords: + patch
stage: (no value)
pull_requests: + pull_request11228
2019-01-15 12:32:25vstinnersetkeywords: + patch
stage: (no value)
pull_requests: + pull_request11227
2015-12-22 23:18:47yselivanovsetmessages: + msg256881
2015-12-22 16:46:57gvanrossumsetmessages: + msg256843
2015-12-22 09:03:43josephgordonsetnosy: + josephgordon
2015-04-02 05:31:52kernel0settype: crash -> behavior
2015-04-02 04:00:18kernel0create