classification
Title: ProactorEventLoop doesn't support stdin/stdout nor files with connect_read_pipe/connect_write_pipe
Type: behavior Stage:
Components: asyncio, Windows Versions: Python 3.5
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Gabriel Mesquita Cangussu, mjpieters, paul.moore, steve.dower, terry.reedy, tim.golden, yselivanov, zach.ware
Priority: normal Keywords:

Created on 2016-04-23 03:45 by Gabriel Mesquita Cangussu, last changed 2018-09-11 23:03 by gvanrossum.

Messages (6)
msg264043 - (view) Author: Gabriel Mesquita Cangussu (Gabriel Mesquita Cangussu) Date: 2016-04-23 03:45
The documentation of asyncio specifies that the methods connect_read_pipe and connect_write_pipe are available on Windows with the ProactorEventLoop. The documentation then says that those methods accept file-like objects on the pipe parameter. However, the methods doesn't seem to work with stdio or any disk file under Windows. The following example catches this problem:

import asyncio
import sys

class MyProtocol(asyncio.Protocol):
    def connection_made(self, transport):
        print('connection established')
        
    def data_received(self, data):
        print('received: {!r}'.format(data.decode()))
        
    def connection_lost(self, exc):
        print('lost connection')

if sys.platform.startswith('win32'):
    loop = asyncio.ProactorEventLoop()
else:
    loop = asyncio.SelectorEventLoop()
coro = loop.connect_read_pipe(MyProtocol, sys.stdin)
loop.run_until_complete(coro)
loop.run_forever()
loop.close()


This code when executed on Ubuntu have the desired behavior, but under Windows 10 it gives OSError: [WinError 6] The handle is invalid. The complete output is this:

c:\Users\Gabriel\Documents\Python Scripts>python async_pipe.py
connection established
Fatal read error on pipe transport
protocol: <__main__.MyProtocol object at 0x000001970EB2FAC8>
transport: <_ProactorReadPipeTransport fd=0>
Traceback (most recent call last):
  File "C:\Program Files\Python35\lib\asyncio\proactor_events.py", line 195, in _loop_reading
    self._read_fut = self._loop._proactor.recv(self._sock, 4096)
  File "C:\Program Files\Python35\lib\asyncio\windows_events.py", line 425, in recv
    self._register_with_iocp(conn)
  File "C:\Program Files\Python35\lib\asyncio\windows_events.py", line 606, in _register_with_iocp
    _overlapped.CreateIoCompletionPort(obj.fileno(), self._iocp, 0, 0)
OSError: [WinError 6] Identificador inválido
lost connection


I think that the documentation should state that there is no support for disk files and stdio with the methods in question and also state what exactly they support (an example would be nice). And, of course, better support for watching file descriptors on Windows on future Python releases would be nice.

Thank you,
Gabriel
msg264044 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2016-04-23 04:57
> File "C:\Program Files\Python35\lib\asyncio\windows_events.py", line 606, in _register_with_iocp
>   _overlapped.CreateIoCompletionPort(obj.fileno(), self._iocp, 0, 0)

I don't think there's any case where this is going to work on an actual file - CreateIoCompletionPort needs a handle and not a file descriptor.

Presumably this line of code normally gets used with something else that defines fileno()?
msg264516 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-04-29 19:24
The surprise to me, being on Windows, is that the pipe connection methods sometimes work with non-pipes.  The limitations of Windows event loops are given in https://docs.python.org/3/library/asyncio-eventloops.html#windows.  The pipe connection functions are discussed in https://docs.python.org/3/library/asyncio-eventloop.html#connect-pipes.  Both say that the methods do not work with Windows' SelectorEventLoop.

My understanding is that this is because Windows' select() call does not work with pipes -- meaning honest-to-goodness OS pipes.  So I understood "*pipe* is file-like object." more as a un-surprising statement of fact than as a permissive "'pipe' can be any file-like object and not necessarily a pipe".

If 'pipe' were intended to mean 'file-like', then why use 'pipe'?  But I can see how a current unix user would understand that sentence the other way.  Perhaps the sentence should read "For SelectorEventLoops (not on Windows), *pipe* can also be any file-like object with the appropriate methods." -- assuming that this is true on all non-Windows systems.

Isn't there some other way to asynchronously read/file files, as opposed to sockets and pipes, on Windows?
msg264523 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2016-04-29 21:26
Pipes and file handles are equivalent in Windows, but socket handles are their own namespace and have their own functions. Some Python code will switch between them automatically to make socket functions work with file descriptors, but generally I'd expect stream pipes (as opposed to, IIRC, datagram pipes) to be compatible with files but not sockets.

Not entirely sure how that plays into this issue, but it's a bit more background.
msg264599 - (view) Author: Gabriel Mesquita Cangussu (Gabriel Mesquita Cangussu) Date: 2016-05-01 15:36
Terry,

About your question, "Isn't there some other way to asynchronously read/file files, as opposed to sockets and pipes, on Windows?", that is exactly the problem I was trying to overcome, specifically for reading the stdin.

On my research I found both this stackoverflow questions with no valid answers:

http://stackoverflow.com/questions/31510190/aysncio-cannot-read-stdin-on-windows

http://stackoverflow.com/questions/31511563/file-to-socket-adapter-in-python

And I understood that 'pipe' wouldn't mean a 'file', but why can't I use 'sys.stdin' with 'ProactorEventLoop.connect_read_pipe()', can't 'sys.stdin' be a pipe on Windows like it can on Posix?
msg325001 - (view) Author: Martijn Pieters (mjpieters) * Date: 2018-09-11 11:24
I'm trying to figure out why Windows won't let us do this. I think the reason is that sys.std(in|out) filehandles are not opened as pipes, and do not have the required OVERLAPPED flag set (see the CreateIoCompletionPort documentation at https://docs.microsoft.com/en-us/windows/desktop/fileio/createiocompletionport; it's that function that is used to handle pipes (via IocpProactor.recv -> IocpProactor._register_with_iocp -> overlapped.CreateIoCompletionPort).

The solution then would be to create a pipe for a stdio filehandle with the flag set.

And that's where my Windows-fu ends, and where I lack the VM and motivation to go try that out.
History
Date User Action Args
2018-09-11 23:03:47gvanrossumsetnosy: - gvanrossum
2018-09-11 22:31:32vstinnersetnosy: - vstinner
2018-09-11 11:24:08mjpieterssetnosy: + mjpieters
messages: + msg325001
2016-05-01 15:36:39Gabriel Mesquita Cangussusetmessages: + msg264599
2016-04-29 21:26:47steve.dowersetmessages: + msg264523
2016-04-29 19:24:17terry.reedysetnosy: + terry.reedy
messages: + msg264516
2016-04-23 04:57:24steve.dowersetmessages: + msg264044
2016-04-23 03:45:50Gabriel Mesquita Cangussucreate