classification
Title: asyncio subprocess AttributeError: 'NoneType' object has no attribute '_add_reader' / '_remove_reader'
Type: behavior Stage: resolved
Components: asyncio Versions: Python 3.9, Python 3.8, Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: asvetlov Nosy List: asvetlov, decaz, miss-islington, yselivanov
Priority: normal Keywords: patch

Created on 2019-09-03 16:55 by decaz, last changed 2019-09-30 09:21 by decaz. This issue is now closed.

Files
File name Uploaded Description Edit
f1_and_f2.zip decaz, 2019-09-03 16:55
Pull Requests
URL Status Linked Edit
PR 16472 merged asvetlov, 2019-09-29 10:51
PR 16473 merged miss-islington, 2019-09-29 12:00
PR 16474 merged miss-islington, 2019-09-29 12:00
Messages (8)
msg351090 - (view) Author: Marat Sharafutdinov (decaz) * Date: 2019-09-03 16:55
I'm trying to use dwdiff (https://os.ghalkes.nl/dwdiff.html) to compare two text files. While running subprocess I'm getting the following errors:

```
Traceback (most recent call last):
  File "test_dwdiff.py", line 17, in <module>
    asyncio.run(main())
  File "/home/decaz/.pyenv/versions/3.7.4/lib/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/home/decaz/.pyenv/versions/3.7.4/lib/python3.7/asyncio/base_events.py", line 579, in run_until_complete
    return future.result()
  File "test_dwdiff.py", line 14, in main
    await asyncio.gather(*(dwdiff() for __ in range(25)))
  File "test_dwdiff.py", line 10, in dwdiff
    await process.communicate()
  File "/home/decaz/.pyenv/versions/3.7.4/lib/python3.7/asyncio/subprocess.py", line 187, in communicate
    loop=self._loop)
  File "/home/decaz/.pyenv/versions/3.7.4/lib/python3.7/asyncio/subprocess.py", line 166, in _read_stream
    output = await stream.read()
  File "/home/decaz/.pyenv/versions/3.7.4/lib/python3.7/asyncio/streams.py", line 633, in read
    block = await self.read(self._limit)
  File "/home/decaz/.pyenv/versions/3.7.4/lib/python3.7/asyncio/streams.py", line 646, in read
    self._maybe_resume_transport()
  File "/home/decaz/.pyenv/versions/3.7.4/lib/python3.7/asyncio/streams.py", line 417, in _maybe_resume_transport
    self._transport.resume_reading()
  File "/home/decaz/.pyenv/versions/3.7.4/lib/python3.7/asyncio/unix_events.py", line 498, in resume_reading
    self._loop._add_reader(self._fileno, self._read_ready)
AttributeError: 'NoneType' object has no attribute '_add_reader'
```

and

```
Exception in callback SubprocessStreamProtocol.pipe_data_received(1, b'\xd0\x9f\xd...b8\xd1\x8f.\n')
handle: <Handle SubprocessStreamProtocol.pipe_data_received(1, b'\xd0\x9f\xd...b8\xd1\x8f.\n')>
Traceback (most recent call last):
  File "/home/decaz/.pyenv/versions/3.7.4/lib/python3.7/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "/home/decaz/.pyenv/versions/3.7.4/lib/python3.7/asyncio/subprocess.py", line 71, in pipe_data_received
    reader.feed_data(data)
  File "/home/decaz/.pyenv/versions/3.7.4/lib/python3.7/asyncio/streams.py", line 440, in feed_data
    self._transport.pause_reading()
  File "/home/decaz/.pyenv/versions/3.7.4/lib/python3.7/asyncio/unix_events.py", line 495, in pause_reading
    self._loop._remove_reader(self._fileno)
AttributeError: 'NoneType' object has no attribute '_remove_reader'
```

The code is the following:
```
import asyncio

async def dwdiff():
    process = await asyncio.create_subprocess_exec(
        '/usr/local/bin/dwdiff', 'f1.txt', 'f2.txt',
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.PIPE,
    )
    await process.communicate()

async def main():
    await asyncio.gather(*(dwdiff() for __ in range(25)))

asyncio.run(main())
```

Two text files f1.txt and f2.txt are attached as archive.

Perhaps it relates to https://bugs.python.org/issue29704
msg352489 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2019-09-15 17:28
Thanks. The correct fix is 'do nothing if stream._transport is None'.
We have only two weeks before 3.8rc, thus I'll prepare a fix myself for the sake of fast merging.
msg352490 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2019-09-15 17:30
This fix cannot land on 3.6, the version is in security-fix only mode.
msg352779 - (view) Author: Marat Sharafutdinov (decaz) * Date: 2019-09-19 09:46
Andrew, it would be nice to have it fixed within 3.8. Hope you'll find time for this.
msg353496 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2019-09-29 12:00
New changeset 58498bc7178608b1ab031994ca09c43889ce3e76 by Andrew Svetlov in branch 'master':
bpo-38019: correctly handle pause/resume reading of closed asyncio unix pipe (GH-16472)
https://github.com/python/cpython/commit/58498bc7178608b1ab031994ca09c43889ce3e76
msg353497 - (view) Author: miss-islington (miss-islington) Date: 2019-09-29 12:19
New changeset 1c3e4691bb41fa070d6f4836cb03005123e53e4b by Miss Islington (bot) in branch '3.7':
bpo-38019: correctly handle pause/resume reading of closed asyncio unix pipe (GH-16472)
https://github.com/python/cpython/commit/1c3e4691bb41fa070d6f4836cb03005123e53e4b
msg353498 - (view) Author: miss-islington (miss-islington) Date: 2019-09-29 12:20
New changeset 19cd5951ec4480c7cfcbcc379b36902312cfb8b8 by Miss Islington (bot) in branch '3.8':
bpo-38019: correctly handle pause/resume reading of closed asyncio unix pipe (GH-16472)
https://github.com/python/cpython/commit/19cd5951ec4480c7cfcbcc379b36902312cfb8b8
msg353559 - (view) Author: Marat Sharafutdinov (decaz) * Date: 2019-09-30 09:21
Andrew, thank you!
History
Date User Action Args
2019-09-30 09:21:10decazsetmessages: + msg353559
2019-09-29 13:21:00asvetlovsetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2019-09-29 12:20:18miss-islingtonsetmessages: + msg353498
2019-09-29 12:19:14miss-islingtonsetnosy: + miss-islington
messages: + msg353497
2019-09-29 12:00:56miss-islingtonsetpull_requests: + pull_request16058
2019-09-29 12:00:49miss-islingtonsetpull_requests: + pull_request16057
2019-09-29 12:00:38asvetlovsetmessages: + msg353496
2019-09-29 10:51:18asvetlovsetkeywords: + patch
stage: patch review
pull_requests: + pull_request16056
2019-09-19 09:46:43decazsetmessages: + msg352779
2019-09-15 17:30:40asvetlovsetmessages: + msg352490
versions: - Python 3.6
2019-09-15 17:28:56asvetlovsetassignee: asvetlov
messages: + msg352489
2019-09-13 14:26:16decazsetversions: + Python 3.6, Python 3.8, Python 3.9
title: AttributeError: 'NoneType' object has no attribute '_add_reader' / '_remove_reader' -> asyncio subprocess AttributeError: 'NoneType' object has no attribute '_add_reader' / '_remove_reader'
2019-09-03 16:55:09decazcreate