Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

asyncio : ProactorEventLoop raised BlockingIOError when ThreadPoolExecutor has many workers #68034

Closed
kernel0 mannequin opened this issue Apr 2, 2015 · 6 comments
Closed
Labels
topic-asyncio type-bug An unexpected behavior, bug, or error

Comments

@kernel0
Copy link
Mannequin

kernel0 mannequin commented Apr 2, 2015

BPO 23846
Nosy @gvanrossum, @vstinner, @1st1, @miss-islington
PRs
  • bpo-23846: Fix ProactorEventLoop._write_to_self() #11566
  • bpo-23846: Fix ProactorEventLoop._write_to_self() #11566
  • bpo-23846: Fix ProactorEventLoop._write_to_self() #11566
  • [3.7] bpo-23846: Fix ProactorEventLoop._write_to_self() (GH-11566) #11567
  • [3.7] bpo-23846: Fix ProactorEventLoop._write_to_self() (GH-11566) #11567
  • [3.7] bpo-23846: Fix ProactorEventLoop._write_to_self() (GH-11566) #11567
  • Files
  • example_thread_executor.py
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2019-01-15.13:59:09.776>
    created_at = <Date 2015-04-02.04:00:18.506>
    labels = ['type-bug', 'expert-asyncio']
    title = 'asyncio : ProactorEventLoop raised BlockingIOError when ThreadPoolExecutor has many workers'
    updated_at = <Date 2019-01-15.13:59:09.775>
    user = 'https://bugs.python.org/kernel0'

    bugs.python.org fields:

    activity = <Date 2019-01-15.13:59:09.775>
    actor = 'vstinner'
    assignee = 'none'
    closed = True
    closed_date = <Date 2019-01-15.13:59:09.776>
    closer = 'vstinner'
    components = ['asyncio']
    creation = <Date 2015-04-02.04:00:18.506>
    creator = 'kernel0'
    dependencies = []
    files = ['38801']
    hgrepos = []
    issue_num = 23846
    keywords = ['patch', 'patch', 'patch']
    message_count = 6.0
    messages = ['239872', '256843', '256881', '333700', '333703', '333704']
    nosy_count = 6.0
    nosy_names = ['gvanrossum', 'vstinner', 'yselivanov', 'kernel0', 'josephgordon', 'miss-islington']
    pr_nums = ['11566', '11566', '11566', '11567', '11567', '11567']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue23846'
    versions = ['Python 3.4']

    @kernel0
    Copy link
    Mannequin Author

    kernel0 mannequin commented Apr 2, 2015

    I tested on python3.4.2, windows 7

    1. Only proactorEvent issued.
    2. ThreadPoolExecutor has many workers (in the attached example file, worker count 20,000)
    3. The loop does run_in_executor more call than worker count (in the attached example file, 40,000 calls)
    4. After some random seconds, raise BlockingIOError
    BlockingIOError: [WinError 10035] A non-blocking socket operation could not be completed immediately.
    exception calling callback for <Future at 0x1ab89ef0 state=finished returned NoneType>
    Traceback (most recent call last):
      File "c:\Python342\Lib\concurrent\futures\_base.py", line 297, in _invoke_callbacks
        callback(self)
      File "c:\Python342\Lib\asyncio\futures.py", line 410, in <lambda>
        new_future._copy_state, fut))
      File "c:\Python342\Lib\asyncio\base_events.py", line 403, in call_soon_threadsafe
        self._write_to_self()
      File "c:\Python342\Lib\asyncio\proactor_events.py", line 449, in _write_to_self
        self._csock.send(b'\0')

    I guess that proactor's _write_to_self method misses exception handle.

    proactor_events.py
    def _write_to_self(self):
    self._csock.send(b'\0')

    selector_events.py
    def _write_to_self(self):
    # This may be called from a different thread, possibly after
    # _close_self_pipe() has been called or even while it is
    # running. Guard for self._csock being None or closed. When
    # a socket is closed, send() raises OSError (with errno set to
    # EBADF, but let's not rely on the exact error code).
    csock = self._csock
    if csock is not None:
    try:
    csock.send(b'\0')
    except OSError:
    if self._debug:
    logger.debug("Fail to write a null byte into the "
    "self-pipe socket",
    exc_info=True)

    Ps: It's my first publication. Hope you understand my poor comment..

    @kernel0 kernel0 mannequin added type-crash A hard crash of the interpreter, possibly with a core dump topic-asyncio type-bug An unexpected behavior, bug, or error and removed type-crash A hard crash of the interpreter, possibly with a core dump labels Apr 2, 2015
    @gvanrossum
    Copy link
    Member

    Why are you using 20000 threads?

    --Guido (mobile)
    On Dec 22, 2015 02:03, "Joseph Gordon" <report@bugs.python.org> wrote:

    Changes by Joseph Gordon <j.gordon.matthew@gmail.com>:

    ----------
    nosy: +josephgordon


    Python tracker <report@bugs.python.org>
    <http://bugs.python.org/issue23846\>


    @1st1
    Copy link
    Member

    1st1 commented Dec 22, 2015

    Why are you using 20000 threads?

    That's a good question.

    In any case it looks like self-pipe sock's buffer was overflown because call_soon_threadsafe was called too many times, and loop._read_from_self couldn't empty the buffer promptly. Then, at some point, _write_to_self failed with an IOError.

    It looks like this was fixed for the selector loop, but not for proactor:

    selector_events.py:

        def _write_to_self(self):
            csock = self._csock
            if csock is not None:
                try:
                    csock.send(b'\0')
                except OSError:
                    if self._debug:
                        logger.debug("Fail to write a null byte into the "
                                     "self-pipe socket",
                                     exc_info=True)

    proactor_events.py:

        def _write_to_self(self):
            self._csock.send(b'\0')

    @vstinner
    Copy link
    Member

    New changeset c9f872b by Victor Stinner in branch 'master':
    bpo-23846: Fix ProactorEventLoop._write_to_self() (GH-11566)
    c9f872b

    @miss-islington
    Copy link
    Contributor

    New changeset c9f2671 by Miss Islington (bot) in branch '3.7':
    bpo-23846: Fix ProactorEventLoop._write_to_self() (GH-11566)
    c9f2671

    @vstinner
    Copy link
    Member

    In any case it looks like self-pipe sock's buffer was overflown because call_soon_threadsafe was called too many times, and loop._read_from_self couldn't empty the buffer promptly. Then, at some point, _write_to_self failed with an IOError.

    I fixed the issue. Thanks for your bug report ;-)

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    topic-asyncio type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    4 participants