classification
Title: No clean way to get notified when a Transport's write buffer empties out
Type: enhancement Stage:
Components: asyncio Versions: Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: asvetlov, vitaly.krug, yselivanov
Priority: normal Keywords:

Created on 2018-03-22 06:40 by vitaly.krug, last changed 2018-03-22 18:58 by yselivanov.

Messages (10)
msg314236 - (view) Author: Vitaly Kruglikov (vitaly.krug) Date: 2018-03-22 06:40
There doesn't appear to be an ordained mechanism for getting notified when a Transport's (or WriteTransport's) write buffer drains to zero (i.e., all output data has been transferred to socket). I don't want to hijack `set_write_buffer_limits()` for this purpose, because that would preclude me from using it for its intended purpose.

I see that transport in selector_events.py has a private method `_make_empty_waiter()`, which is along the lines of what I need, but it's private and is used by `BaseSelectorEventLoop._sendfile_native()`.

Just like `BaseSelectorEventLoop._sendfile_native()`, my app needs equivalent functionality in order to be able to run the loop (`run_until_complete()`) until the transport's write buffer empties out.
msg314271 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2018-03-22 16:42
We'll likely add 'write_buffer_drained' callback method to `asyncio.Protocol` in 3.8.  In the meanwhile, the only option would be using `_make_empty_waiter` in 3.7, or set_write_buffer_limits(0, 0).

What's your use case, by the way?
msg314274 - (view) Author: Vitaly Kruglikov (vitaly.krug) Date: 2018-03-22 17:50
Thank you for following up. My use case is this:

In the Pika AMQP client package, we have a blocking AMQP connection adapter - `BlockingConnection` - wrapped around an asynchronous AMQP connection adapter. Presently, BlockingConnection is wrapped around the asynchronous `SelectConnection` which has a home-grown proprietary IOLoop. I would like to switch BlockingConnection to use an asyncio-based adapter when running on Python3.

SelectConnection's proprietary IOLoop provides a single-stepping run function (poll with a timeout as normally determined by pending callbacks, timers, etc., process all ready events/timers/callbacks, and return). When BlockingConnection needs to send something to the AMQP broker and/or wait for an expected reply, it sends the request (which typically gets queued up in a write buffer) and then runs the proprietary IOLoop's single-stepping method in a loop (thus blocking efficiently on I/O); each time after the single-stepping IOLoop method returns, BlockingConnection checks whether the conditions of interest have been satisfied (e.g., the write buffer size being zero and AMQP Channel.OpenOk has been received).

So, another way that asyncio could help, and certainly simplest for me and my use case, is by providing a single-stepping function in the event loop implementations, such as the single-stepping method that I described at the top of the previous paragraph. E.g., `events.AbstractEventLoop.run_one_step()`.
msg314278 - (view) Author: Vitaly Kruglikov (vitaly.krug) Date: 2018-03-22 17:57
... or `events.AbstractEventLoop.run_one_iteration()`.
msg314279 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2018-03-22 17:59
Yeah, I think your best option would be to use `set_write_buffer_limits(0, 0)`.  You don't need asyncio flow control anyways, as AMQP protocol is unlikely to generate any pressure on IO buffers.
msg314280 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2018-03-22 18:00
> 'events.AbstractEventLoop.run_one_step()'

This is highly unlikely to ever happen.  It's hard to define what one iteration of an event loop is, and it would be even harder to get that agreement for all frameworks/event loops that are compatible with or based on asyncio.
msg314282 - (view) Author: Vitaly Kruglikov (vitaly.krug) Date: 2018-03-22 18:06
> 'events.AbstractEventLoop.run_one_step()'

> This is highly unlikely to ever happen.

Sure, I can see how that could be a problem with other underlying implementations, such as Twisted reactor. Just wishful thinking :).
msg314283 - (view) Author: Vitaly Kruglikov (vitaly.krug) Date: 2018-03-22 18:16
I'll have to settle for `set_write_buffer_limits(0, 0)` for my blocking wrapper case.

I think that 'write_buffer_drained' callback is a good idea, though.
msg314284 - (view) Author: Vitaly Kruglikov (vitaly.krug) Date: 2018-03-22 18:26
Yet another paradigm is the likes of `GSource` in gnome's glib. GSource "tasks" added to the event loop are polled by the event loop for readiness before the event loop blocks on select/epoll/etc.. The ones that are ready are removed from the loop and their callbacks are dispatched.

I suppose that it would also be difficult to get buy-in for a feature like this from the different frameworks?
msg314288 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2018-03-22 18:58
> I suppose that it would also be difficult to get buy-in for a feature like this from the different frameworks?

Maybe :)  Ideally, asyncio programs should not depend on how exactly tasks are scheduled.
History
Date User Action Args
2018-03-22 18:58:50yselivanovsetmessages: + msg314288
2018-03-22 18:26:05vitaly.krugsetmessages: + msg314284
2018-03-22 18:16:21vitaly.krugsetmessages: + msg314283
2018-03-22 18:06:20vitaly.krugsetmessages: + msg314282
2018-03-22 18:00:46yselivanovsetmessages: + msg314280
2018-03-22 17:59:29yselivanovsetmessages: + msg314279
2018-03-22 17:57:50vitaly.krugsetmessages: + msg314278
2018-03-22 17:50:57vitaly.krugsetmessages: + msg314274
2018-03-22 16:42:57yselivanovsetmessages: + msg314271
versions: - Python 3.5, Python 3.6, Python 3.7
2018-03-22 06:40:11vitaly.krugcreate