classification
Title: Python 3.8 improperly warns about closing properly closed streams
Type: behavior Stage: resolved
Components: asyncio Versions: Python 3.9, Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Ron Frederick, asvetlov, cjrh, falsetru, lukasz.langa, miss-islington, thehesiod, yselivanov
Priority: release blocker Keywords: patch

Created on 2019-10-19 18:40 by Ron Frederick, last changed 2019-12-07 11:40 by asvetlov. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 17474 merged asvetlov, 2019-12-05 14:14
PR 17492 merged miss-islington, 2019-12-07 11:22
Messages (6)
msg354953 - (view) Author: Ron Frederick (Ron Frederick) * Date: 2019-10-19 18:40
In testing AsyncSSH against Python 3.8, I noticed a large number of the following errors, even though I was properly closing streams before the objects holding them were garbage-collected.

    An open stream object is being garbage collected; call "stream.close()" explicitly.

After some investigation, the problem appears to be that closing a stream is not good enough to prevent the error. The check in asyncio doesn't properly handle the case where the stream is closing, but has not fully closed. Here's a simple test program that demonstrates this:

import asyncio

async def tcp_client():
    reader, writer = await asyncio.open_connection('127.0.0.1', 22)

    writer.close()

asyncio.run(tcp_client())

It's possible to avoid this message by awaiting on writer.wait_closed(), but wait_closed() doesn't exist until Python 3.7, making it very difficult to write portable code and still avoid this message.
msg355481 - (view) Author: Ron Frederick (Ron Frederick) * Date: 2019-10-27 16:38
I think the following change might address this problem:

***************
*** 233,239 ****
  
      def _on_reader_gc(self, wr):
          transport = self._transport
!         if transport is not None:
              # connection_made was called
              context = {
                  'message': ('An open stream object is being garbage '
--- 233,239 ----
  
      def _on_reader_gc(self, wr):
          transport = self._transport
!         if transport is not None and not transport.is_closing():
              # connection_made was called
              context = {
                  'message': ('An open stream object is being garbage '
msg355482 - (view) Author: Ron Frederick (Ron Frederick) * Date: 2019-10-27 16:41
Sorry, I should have said that the change below was in the file asyncio/streams.py.
msg355485 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2019-10-27 17:26
Yes, I've experienced this bug. We need to fix this in 3.8.1.
msg357968 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2019-12-07 11:22
New changeset 7ddcd0caa4c2e6b43265df144f59c5aa508a94f2 by Andrew Svetlov in branch 'master':
bpo-38529: Fix asyncio stream warning (GH-17474)
https://github.com/python/cpython/commit/7ddcd0caa4c2e6b43265df144f59c5aa508a94f2
msg357970 - (view) Author: miss-islington (miss-islington) Date: 2019-12-07 11:40
New changeset 7fde4f446a3dcfed780a38fbfcd7c0b4d9d73b93 by Miss Islington (bot) in branch '3.8':
bpo-38529: Fix asyncio stream warning (GH-17474)
https://github.com/python/cpython/commit/7fde4f446a3dcfed780a38fbfcd7c0b4d9d73b93
History
Date User Action Args
2019-12-07 11:40:47asvetlovsetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2019-12-07 11:40:00miss-islingtonsetnosy: + miss-islington
messages: + msg357970
2019-12-07 11:22:23miss-islingtonsetpull_requests: + pull_request16970
2019-12-07 11:22:04asvetlovsetmessages: + msg357968
2019-12-05 14:32:29asvetlovsetversions: + Python 3.9
2019-12-05 14:14:49asvetlovsetkeywords: + patch
stage: patch review
pull_requests: + pull_request16956
2019-11-14 02:04:53thehesiodsetnosy: + thehesiod
2019-10-27 17:26:42yselivanovsetpriority: normal -> release blocker
nosy: + lukasz.langa
messages: + msg355485

2019-10-27 16:41:05Ron Fredericksetmessages: + msg355482
2019-10-27 16:38:07Ron Fredericksetmessages: + msg355481
2019-10-25 14:49:19falsetrusetnosy: + falsetru
2019-10-20 00:25:32cjrhsetnosy: + cjrh
2019-10-19 18:40:28Ron Frederickcreate