This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: asyncio open_connection fails when a socket is explicitly closed
Type: Stage: resolved
Components: asyncio, Windows Versions: Python 3.11, Python 3.10, Python 3.9
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: adhika.setyap, asvetlov, danielen1337, eamanu, mhils, miss-islington, paul.moore, steve.dower, tim.golden, yselivanov, zach.ware
Priority: normal Keywords:

Created on 2021-02-18 13:46 by danielen1337, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 31892 merged mhils, 2022-03-15 07:51
PR 31905 merged miss-islington, 2022-03-15 14:59
PR 31906 merged miss-islington, 2022-03-15 14:59
Messages (12)
msg387228 - (view) Author: Daniel Engel (danielen1337) Date: 2021-02-18 13:46
Python 3.9.0 (tags/v3.9.0:9cf6752, Oct  5 2020, 15:34:40) [MSC v.1927 64 bit (AMD64)]
Type 'copyright', 'credits' or 'license' for more information
IPython 7.18.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import socket
   ...: s1, s2 = socket.socketpair()
   ...: import asyncio
   ...: async def test():
   ...:     r1, w1 = await asyncio.open_connection(sock=s1)
   ...:     r2, w2 = await asyncio.open_connection(sock=s2)
   ...:     s1.close()
Exception in callback _ProactorBasePipeTransport._call_connection_lost(ConnectionAbo...e, 1236, None))
handle: <Handle _ProactorBasePipeTransport._call_connection_lost(ConnectionAbo...e, 1236, None))>
Traceback (most recent call last):
  File "c:\python39\lib\asyncio\", line 80, in _run, *self._args)
  File "c:\python39\lib\asyncio\", line 162, in _call_connection_lost
OSError: [WinError 10038] An operation was attempted on something that is not a socket
msg387235 - (view) Author: Emmanuel Arias (eamanu) * Date: 2021-02-18 14:37
I cannot reproduce on a Debian machine. Seems to be a Windows component issue?
msg387238 - (view) Author: Daniel Engel (danielen1337) Date: 2021-02-18 15:07
It reproduced on a windows machine
msg414620 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2022-03-06 17:37
Please consider passing 'sock' argument as the ownership transfer.

You should not perform any action on 'sock' object directly anymore.
This is true for all asyncio API.
msg415071 - (view) Author: Maximilian Hils (mhils) * Date: 2022-03-13 18:47
We are hitting the same traceback with mitmproxy on Windows without ever passing a socket object to open_connection. In other words, this can be triggered without performing any action on 'sock' due to some race conditions in the Windows network stack. I unfortunately don't have a better repro for that race other than what Daniel provided.
msg415128 - (view) Author: Maximilian Hils (mhils) * Date: 2022-03-14 10:54
asvetlov: Sorry if I articulated myself badly, but I do think this is a valid bug. It's unfortunately hard to provide a better repro (I tried), but we are hitting this regularly when mitmproxy is accepting connections under heavy load. We're just calling `asyncio.start_server(handler, "", 8080)` in mitmproxy and never interact with the underlying socket object.

Here are some observations that are true for all crashes:

- The socket fileno is -1 when it crashes.
- `_call_connection_lost` is called by `_ProactorBasePipeTransport.close`, which is called by `_ProactorBasePipeTransport.__del__` [1]
- There are no previous calls to `_call_connection_lost`.
- Windows only, loopback connections in our case.
- Wireshark shows that client and server are first happily exchanging packets. At some point the client sends a FIN, which the Python server ACKs immediately. A few seconds later the Python server sends a FIN back.

An obvious fix without understanding the root cause would be to check fileno in I'm not too familar with proactor to assess if that is a good idea. Sorry for not being able to provide more details.

msg415205 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2022-03-14 22:37
Maximilian, thanks for the investigation.

A check for 'fileno != -1' seems correct to me.
Would you prepare a pull request?
msg415250 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2022-03-15 14:59
New changeset 70155412f1543f100d4aa309b8691cbcabd3e0e1 by Maximilian Hils in branch 'main':
bpo-43253: Don't call shutdown() for invalid socket handles (GH-31892)
msg415254 - (view) Author: miss-islington (miss-islington) Date: 2022-03-15 15:22
New changeset 88c243fd8d5a43282ef06bd0872e3b88c68bb856 by Miss Islington (bot) in branch '3.10':
bpo-43253: Don't call shutdown() for invalid socket handles (GH-31892)
msg415255 - (view) Author: miss-islington (miss-islington) Date: 2022-03-15 15:23
New changeset 64a68c39cb508b016e5a4486ebb4052f6e65fca0 by Miss Islington (bot) in branch '3.9':
bpo-43253: Don't call shutdown() for invalid socket handles (GH-31892)
msg415257 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2022-03-15 15:49
msg415338 - (view) Author: Maximilian Hils (mhils) * Date: 2022-03-16 12:19
Thank you for the super quick turnaround.
My heart goes out to you and everyone else in Ukraine.
Date User Action Args
2022-04-11 14:59:41adminsetgithub: 87419
2022-03-16 12:19:30mhilssetmessages: + msg415338
2022-03-15 15:49:39asvetlovsetstatus: closed
resolution: fixed
messages: + msg415257
2022-03-15 15:23:52miss-islingtonsetmessages: + msg415255
2022-03-15 15:22:09miss-islingtonsetmessages: + msg415254
2022-03-15 14:59:30miss-islingtonsetpull_requests: + pull_request30000
2022-03-15 14:59:26miss-islingtonsetnosy: + miss-islington

pull_requests: + pull_request29999
2022-03-15 14:59:12asvetlovsetmessages: + msg415250
2022-03-15 14:57:55asvetlovsetversions: + Python 3.11
2022-03-15 07:51:17mhilssetpull_requests: + pull_request29990
2022-03-14 22:37:34asvetlovsetmessages: + msg415205
2022-03-14 10:54:47mhilssetmessages: + msg415128
2022-03-13 19:10:34asvetlovsetstatus: closed -> (no value)
resolution: not a bug -> (no value)
2022-03-13 18:48:23mhilssetversions: + Python 3.10
2022-03-13 18:47:44mhilssetnosy: + mhils
messages: + msg415071
2022-03-06 17:37:09asvetlovsetstatus: open -> closed
resolution: not a bug
messages: + msg414620

stage: resolved
2022-03-06 13:01:10adhika.setyapsetnosy: + adhika.setyap
2021-02-18 16:06:00eamanusetnosy: + paul.moore, tim.golden, zach.ware, steve.dower
components: + Windows
2021-02-18 15:07:47danielen1337setmessages: + msg387238
2021-02-18 14:37:11eamanusetnosy: + eamanu
messages: + msg387235
2021-02-18 13:46:25danielen1337create