Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

asyncio subprocess AttributeError: 'NoneType' object has no attribute '_add_reader' / '_remove_reader' #82200

Closed
decaz mannequin opened this issue Sep 3, 2019 · 8 comments
Closed

asyncio subprocess AttributeError: 'NoneType' object has no attribute '_add_reader' / '_remove_reader' #82200

decaz mannequin opened this issue Sep 3, 2019 · 8 comments
Assignees
@asvetlov
Labels
3.7 (EOL) end of life 3.8 only security fixes 3.9 only security fixes topic-asyncio type-bug An unexpected behavior, bug, or error

Comments

@decaz
Copy link
Mannequin

decaz mannequin commented Sep 3, 2019

BPO 38019
Nosy @asvetlov, @1st1, @decaz, @miss-islington
PRs
  • bpo-38019: correctly handle pause/resume reading of closed asyncio unix pipe #16472
  • [3.8] bpo-38019: correctly handle pause/resume reading of closed asyncio unix pipe (GH-16472) #16473
  • [3.7] bpo-38019: correctly handle pause/resume reading of closed asyncio unix pipe (GH-16472) #16474
    		•
    Files
  • f1_and_f2.zip
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/asvetlov'
closed_at = <Date 2019-09-29.13:21:00.810>
created_at = <Date 2019-09-03.16:55:09.357>
labels = ['3.8', 'type-bug', '3.7', '3.9', 'expert-asyncio']
title = "asyncio subprocess AttributeError: 'NoneType' object has no attribute '_add_reader' / '_remove_reader'"
updated_at = <Date 2019-09-30.09:21:10.830>
user = 'https://github.com/decaz'

    bugs.python.org fields:

    activity = <Date 2019-09-30.09:21:10.830>
actor = 'decaz'
assignee = 'asvetlov'
closed = True
closed_date = <Date 2019-09-29.13:21:00.810>
closer = 'asvetlov'
components = ['asyncio']
creation = <Date 2019-09-03.16:55:09.357>
creator = 'decaz'
dependencies = []
files = ['48589']
hgrepos = []
issue_num = 38019
keywords = ['patch']
message_count = 8.0
messages = ['351090', '352489', '352490', '352779', '353496', '353497', '353498', '353559']
nosy_count = 4.0
nosy_names = ['asvetlov', 'yselivanov', 'decaz', 'miss-islington']
pr_nums = ['16472', '16473', '16474']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue38019'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9']
    @decaz
    Copy link
    Mannequin Author

    decaz mannequin commented Sep 3, 2019

    I'm trying to use dwdiff (https://os.ghalkes.nl/dwdiff.html) to compare two text files. While running subprocess I'm getting the following errors:

    Traceback (most recent call last):
  File "test_dwdiff.py", line 17, in <module>
    asyncio.run(main())
  File "/home/decaz/.pyenv/versions/3.7.4/lib/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/home/decaz/.pyenv/versions/3.7.4/lib/python3.7/asyncio/base_events.py", line 579, in run_until_complete
    return future.result()
  File "test_dwdiff.py", line 14, in main
    await asyncio.gather(*(dwdiff() for __ in range(25)))
  File "test_dwdiff.py", line 10, in dwdiff
    await process.communicate()
  File "/home/decaz/.pyenv/versions/3.7.4/lib/python3.7/asyncio/subprocess.py", line 187, in communicate
    loop=self._loop)
  File "/home/decaz/.pyenv/versions/3.7.4/lib/python3.7/asyncio/subprocess.py", line 166, in _read_stream
    output = await stream.read()
  File "/home/decaz/.pyenv/versions/3.7.4/lib/python3.7/asyncio/streams.py", line 633, in read
    block = await self.read(self._limit)
  File "/home/decaz/.pyenv/versions/3.7.4/lib/python3.7/asyncio/streams.py", line 646, in read
    self._maybe_resume_transport()
  File "/home/decaz/.pyenv/versions/3.7.4/lib/python3.7/asyncio/streams.py", line 417, in _maybe_resume_transport
    self._transport.resume_reading()
  File "/home/decaz/.pyenv/versions/3.7.4/lib/python3.7/asyncio/unix_events.py", line 498, in resume_reading
    self._loop._add_reader(self._fileno, self._read_ready)
AttributeError: 'NoneType' object has no attribute '_add_reader'

    and

    Exception in callback SubprocessStreamProtocol.pipe_data_received(1, b'\xd0\x9f\xd...b8\xd1\x8f.\n')
handle: <Handle SubprocessStreamProtocol.pipe_data_received(1, b'\xd0\x9f\xd...b8\xd1\x8f.\n')>
Traceback (most recent call last):
  File "/home/decaz/.pyenv/versions/3.7.4/lib/python3.7/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "/home/decaz/.pyenv/versions/3.7.4/lib/python3.7/asyncio/subprocess.py", line 71, in pipe_data_received
    reader.feed_data(data)
  File "/home/decaz/.pyenv/versions/3.7.4/lib/python3.7/asyncio/streams.py", line 440, in feed_data
    self._transport.pause_reading()
  File "/home/decaz/.pyenv/versions/3.7.4/lib/python3.7/asyncio/unix_events.py", line 495, in pause_reading
    self._loop._remove_reader(self._fileno)
AttributeError: 'NoneType' object has no attribute '_remove_reader'

    The code is the following:

    import asyncio

async def dwdiff():
    process = await asyncio.create_subprocess_exec(
        '/usr/local/bin/dwdiff', 'f1.txt', 'f2.txt',
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.PIPE,
    )
    await process.communicate()

async def main():
    await asyncio.gather(*(dwdiff() for __ in range(25)))

asyncio.run(main())

    Two text files f1.txt and f2.txt are attached as archive.

    Perhaps it relates to https://bugs.python.org/issue29704

    @decaz decaz mannequin added 3.7 (EOL) end of life topic-asyncio type-bug An unexpected behavior, bug, or error labels Sep 3, 2019
    @decaz decaz mannequin added 3.8 only security fixes 3.9 only security fixes labels Sep 13, 2019
    @decaz decaz mannequin changed the title AttributeError: 'NoneType' object has no attribute '_add_reader' / '_remove_reader' asyncio subprocess AttributeError: 'NoneType' object has no attribute '_add_reader' / '_remove_reader' Sep 13, 2019
    @asvetlov
    Copy link
    Contributor

    Thanks. The correct fix is 'do nothing if stream._transport is None'.
    We have only two weeks before 3.8rc, thus I'll prepare a fix myself for the sake of fast merging.

    @asvetlov asvetlov self-assigned this Sep 15, 2019
    @asvetlov
    Copy link
    Contributor

    This fix cannot land on 3.6, the version is in security-fix only mode.

    @decaz
    Copy link
    Mannequin Author

    decaz mannequin commented Sep 19, 2019

    Andrew, it would be nice to have it fixed within 3.8. Hope you'll find time for this.

    @asvetlov
    Copy link
    Contributor

    New changeset 58498bc by Andrew Svetlov in branch 'master':
    bpo-38019: correctly handle pause/resume reading of closed asyncio unix pipe (GH-16472)
    58498bc

    @miss-islington
    Copy link
    Contributor

    New changeset 1c3e469 by Miss Islington (bot) in branch '3.7':
    bpo-38019: correctly handle pause/resume reading of closed asyncio unix pipe (GH-16472)
    1c3e469

    @miss-islington
    Copy link
    Contributor

    New changeset 19cd595 by Miss Islington (bot) in branch '3.8':
    bpo-38019: correctly handle pause/resume reading of closed asyncio unix pipe (GH-16472)
    19cd595

    @decaz
    Copy link
    Mannequin Author

    decaz mannequin commented Sep 30, 2019

    Andrew, thank you!

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees

    @asvetlov asvetlov

    Labels
    3.7 (EOL) end of life 3.8 only security fixes 3.9 only security fixes topic-asyncio type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    2 participants
    @asvetlov @miss-islington