ProactorEventLoop doesn't support stdin/stdout nor files with connect_read_pipe/connect_write_pipe

[GabrielMesquitaCangussu]

BPO	26832
Nosy	@terryjreedy, @pfmoore, @mjpieters, @tjguk, @zware, @1st1, @zooba, @minrk

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2016-04-23.03:45:50.331>
labels = ['3.8', '3.9', '3.7', 'expert-asyncio', 'type-bug', '3.10', '3.11', 'OS-windows']
title = "ProactorEventLoop doesn't support stdin/stdout nor files with connect_read_pipe/connect_write_pipe"
updated_at = <Date 2022-02-23.10:03:17.188>
user = 'https://bugs.python.org/GabrielMesquitaCangussu'

bugs.python.org fields:

activity = <Date 2022-02-23.10:03:17.188>
actor = 'minrk'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Windows', 'asyncio']
creation = <Date 2016-04-23.03:45:50.331>
creator = 'Gabriel Mesquita Cangussu'
dependencies = []
files = []
hgrepos = []
issue_num = 26832
keywords = []
message_count = 7.0
messages = ['264043', '264044', '264516', '264523', '264599', '325001', '413781']
nosy_count = 9.0
nosy_names = ['terry.reedy', 'paul.moore', 'mjpieters', 'tim.golden', 'zach.ware', 'yselivanov', 'steve.dower', 'minrk', 'Gabriel Mesquita Cangussu']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue26832'
versions = ['Python 3.5', 'Python 3.7', 'Python 3.8', 'Python 3.9', 'Python 3.10', 'Python 3.11']

Linked PRs

• gh-71019: document which objects connect_read_pipe/connect_write_pipe accept and add tests #153660

[GabrielMesquitaCangussu]

The documentation of asyncio specifies that the methods connect_read_pipe and connect_write_pipe are available on Windows with the ProactorEventLoop. The documentation then says that those methods accept file-like objects on the pipe parameter. However, the methods doesn't seem to work with stdio or any disk file under Windows. The following example catches this problem:

import asyncio
import sys

class MyProtocol(asyncio.Protocol):
    def connection_made(self, transport):
        print('connection established')
        
    def data_received(self, data):
        print('received: {!r}'.format(data.decode()))
        
    def connection_lost(self, exc):
        print('lost connection')

if sys.platform.startswith('win32'):
    loop = asyncio.ProactorEventLoop()
else:
    loop = asyncio.SelectorEventLoop()
coro = loop.connect_read_pipe(MyProtocol, sys.stdin)
loop.run_until_complete(coro)
loop.run_forever()
loop.close()

This code when executed on Ubuntu have the desired behavior, but under Windows 10 it gives OSError: [WinError 6] The handle is invalid. The complete output is this:

c:\Users\Gabriel\Documents\Python Scripts>python async_pipe.py
connection established
Fatal read error on pipe transport
protocol: <__main__.MyProtocol object at 0x000001970EB2FAC8>
transport: <_ProactorReadPipeTransport fd=0>
Traceback (most recent call last):
  File "C:\Program Files\Python35\lib\asyncio\proactor_events.py", line 195, in _loop_reading
    self._read_fut = self._loop._proactor.recv(self._sock, 4096)
  File "C:\Program Files\Python35\lib\asyncio\windows_events.py", line 425, in recv
    self._register_with_iocp(conn)
  File "C:\Program Files\Python35\lib\asyncio\windows_events.py", line 606, in _register_with_iocp
    _overlapped.CreateIoCompletionPort(obj.fileno(), self._iocp, 0, 0)
OSError: [WinError 6] Identificador inválido
lost connection

I think that the documentation should state that there is no support for disk files and stdio with the methods in question and also state what exactly they support (an example would be nice). And, of course, better support for watching file descriptors on Windows on future Python releases would be nice.

Thank you,
Gabriel

[zooba]

File "C:\Program Files\Python35\lib\asyncio\windows_events.py", line 606, in _register_with_iocp
_overlapped.CreateIoCompletionPort(obj.fileno(), self._iocp, 0, 0)

I don't think there's any case where this is going to work on an actual file - CreateIoCompletionPort needs a handle and not a file descriptor.

Presumably this line of code normally gets used with something else that defines fileno()?

[terryjreedy]

The surprise to me, being on Windows, is that the pipe connection methods sometimes work with non-pipes. The limitations of Windows event loops are given in https://docs.python.org/3/library/asyncio-eventloops.html#windows. The pipe connection functions are discussed in https://docs.python.org/3/library/asyncio-eventloop.html#connect-pipes. Both say that the methods do not work with Windows' SelectorEventLoop.

My understanding is that this is because Windows' select() call does not work with pipes -- meaning honest-to-goodness OS pipes. So I understood "*pipe* is file-like object." more as a un-surprising statement of fact than as a permissive "'pipe' can be any file-like object and not necessarily a pipe".

If 'pipe' were intended to mean 'file-like', then why use 'pipe'? But I can see how a current unix user would understand that sentence the other way. Perhaps the sentence should read "For SelectorEventLoops (not on Windows), *pipe* can also be any file-like object with the appropriate methods." -- assuming that this is true on all non-Windows systems.

Isn't there some other way to asynchronously read/file files, as opposed to sockets and pipes, on Windows?

[zooba]

Pipes and file handles are equivalent in Windows, but socket handles are their own namespace and have their own functions. Some Python code will switch between them automatically to make socket functions work with file descriptors, but generally I'd expect stream pipes (as opposed to, IIRC, datagram pipes) to be compatible with files but not sockets.

Not entirely sure how that plays into this issue, but it's a bit more background.

[GabrielMesquitaCangussu]

Terry,

About your question, "Isn't there some other way to asynchronously read/file files, as opposed to sockets and pipes, on Windows?", that is exactly the problem I was trying to overcome, specifically for reading the stdin.

On my research I found both this stackoverflow questions with no valid answers:

http://stackoverflow.com/questions/31510190/aysncio-cannot-read-stdin-on-windows

http://stackoverflow.com/questions/31511563/file-to-socket-adapter-in-python

And I understood that 'pipe' wouldn't mean a 'file', but why can't I use 'sys.stdin' with 'ProactorEventLoop.connect_read_pipe()', can't 'sys.stdin' be a pipe on Windows like it can on Posix?

[mjpieters]

I'm trying to figure out why Windows won't let us do this. I think the reason is that sys.std(in|out) filehandles are not opened as pipes, and do not have the required OVERLAPPED flag set (see the CreateIoCompletionPort documentation at https://docs.microsoft.com/en-us/windows/desktop/fileio/createiocompletionport; it's that function that is used to handle pipes (via IocpProactor.recv -> IocpProactor._register_with_iocp -> overlapped.CreateIoCompletionPort).

The solution then would be to create a pipe for a stdio filehandle with the flag set.

And that's where my Windows-fu ends, and where I lack the VM and motivation to go try that out.

[minrk]

It appears that connect_read_pipe also doesn't accept pipes returned by os.pipe. If that's the case, what does ProactorEventLoop.connect_read_pipe accept? I haven't been able to find any examples of connect_read_pipe that work on Windows, and every connect_read_pipe call in the cpython test suite appears to be skipped on win32. Should it still be raising NotImplementedError on ProactorEventLoop?

I think the error handling could be better (I only get logged errors, nothing I can catch/handle). It seems like connect_read_pipe itself should raise when it fails to register the pipe with IOCP. If that's not feasible, connection_lost/transport.close should probably be triggered, but it isn't with Python 3.9, at least.

Example that works on posix, but seems to fail with non-catchable errors with ProactorEventLoop:

import asyncio
import os
import sys

class PipeProtocol(asyncio.Protocol):
    def __init__(self):
        self.finished = asyncio.Future()

    def connection_made(self, transport):
        print("connection made", file=sys.stderr)
        self.transport = transport

    def connection_lost(self, exc):
        print("connection lost", exc, file=sys.stderr)
        self.finished.set_result(None)

    def data_received(self, data):
        print("data received", data, file=sys.stderr)
        self.handler(data)

    def eof_received(self):
        print("eof received", file=sys.stderr)
        self.finished.set_result(None)

async def test():
    r, w = os.pipe()
    rf = os.fdopen(r, 'r')
    x, p = await asyncio.get_running_loop().connect_read_pipe(PipeProtocol, rf)
    await asyncio.sleep(1)
    print("writing")
    os.write(w, b'asdf')
    await asyncio.sleep(2)
    print("closing")
    os.close(w)
    await asyncio.wait([p.finished], timeout=3)
    x.close()

if __name__ == "__main__":
    asyncio.run(test())