classification
Title: Add @contextlib.asynccontextmanager
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Jelle Zijlstra, giampaolo.rodola, ncoghlan, rhettinger, yselivanov
Priority: normal Keywords:

Created on 2017-02-28 23:21 by Jelle Zijlstra, last changed 2017-06-12 10:40 by yselivanov. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 360 merged Jelle Zijlstra, 2017-03-01 00:19
Messages (13)
msg288729 - (view) Author: Jelle Zijlstra (Jelle Zijlstra) * Date: 2017-02-28 23:21
An async equivalent of @contextmanager would be an obvious use case for async generators and the async with statement. In my own code, I have several async context objects that could have been written more concisely if @asynccontextmanager was available.

I'll be working on a PR to implement this.
msg288732 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-03-01 04:19
The general idea seems reasonable to me, and I've added some specific code level feedback on the PR.

A related question to this one would be whether or not it may make sense to develop a more async-friendly variant of contextlib.ExitStack (I don't currently write enough async code to design that myself, but I'm open to the idea of adding something along those lines).
msg288733 - (view) Author: Jelle Zijlstra (Jelle Zijlstra) * Date: 2017-03-01 04:30
Thanks, I'll address your PR comments.

issue 29302 is asking for AsyncExitStack, but that can be an independent change. I haven't personally felt the need for AsyncExitStack.
msg288734 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2017-03-01 04:35
Nick, do you think this belongs in contextlib to be side-by-side with @contextmanager or would it be better to group all async tools in their own module.   

It seems to me that people are going to ask for an async version of everything, effectively shadowing the core grammar and the library.  Before we go very far down that road, I think we should decide where it all should go.
msg288736 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-03-01 05:11
It's probably worth a python-dev discussion, but I personally draw the line at "Does this need to import 'asyncio'?". If it does, then that's a clear dependency inversion, and the functionality doesn't belong in a relatively low level module like contextlib.

If it doesn't, then I think the potentially tricky part of this kind of code is the way it interacts with the execution stack and the context management machinery, so it makes sense for it to live in contextlib and have a design and review process that's closely aligned with that for the corresponding synchronous APIs.

That said, one of my review comments on the PR was that the new test cases should be split out to their own file to avoiding making the existing tests depend on asyncio, which I'd consider a point in favour of adding an `asyncio.contextlib` module instead of adding these APIs directly to contextlib.
msg288738 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-03-01 05:45
python-dev thread: https://mail.python.org/pipermail/python-dev/2017-March/147501.html

In formulating my question for the list, it occurred to me that while asynccontextmanager doesn't need to depend on asyncio, the AsyncExitStack proposed in issue 29302 likely will, which pushes me towards favouring the `asyncio.contextlib.asynccontextmanager` approach.
msg288759 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2017-03-01 15:37
> In formulating my question for the list, it occurred to me that while asynccontextmanager doesn't need to depend on asyncio, the AsyncExitStack proposed in issue 29302 likely will, which pushes me towards favouring the `asyncio.contextlib.asynccontextmanager` approach.

As I said in [1], I think asynccontextmanager and AsyncExitStack should be framework agnostic and thus stay in the top level contextlib package.

IMO AsyncExitStack should not be dependent on asyncio, and if it is it means we would need to tweak its design to make it not to.  I'll take a look at the other issue to see if I can help.

[1] https://mail.python.org/pipermail/python-dev/2017-March/147505.html
msg288777 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2017-03-01 21:48
At third alternative is to create an asynctools module to collect together tools that work with the async keyword but aren't dependent on asyncio.
msg288781 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-03-02 01:44
In the specific case of contextlib, most of the APIs should be able to transparently support async/await without negatively impacting their synchronous behaviour, so after the python-dev discussion, I think one module with separate sync and async test suites is a good way to go for contextlib specifically: https://mail.python.org/pipermail/python-dev/2017-March/147520.html

However, as noted in that message, I *don't* think we can conclude that's going to be the right answer for the standard library in general - for many modules, a separate a<module> or aio<module> published via PyPI and potentially considered for stdlib inclusion later is going to make more sense.
msg288850 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-03-03 04:00
Something that occurred to me as being a bit tricky to handle here is the backport to contextlib2: that maintains compatibility with 2.6+, so it would need to split any code using "async def" and "await" out to a separate file that only gets imported on 3.5+ (and similarly only run the corresponding test cases on 3.5+).

A potentially simpler alternative to that would be to create a new "backports.contextlib" package that only supports 3.5+ and explicitly restrict contextlib2 itself to code that runs in the common subset of Python 2 & 3.
msg288851 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2017-03-03 04:02
> so it would need to split any code using "async def" and "await" out to a separate file that only gets imported on 3.5+ (and similarly only run the corresponding test cases on 3.5+).

This seems doable.  I'd only vote to keep contextlib.py as one file in CPython though.
msg289335 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-03-10 06:04
For anyone interested in the question of backports, I moved that discussion over to the contextlib2 tracker: https://github.com/jazzband/contextlib2/issues/12
msg292647 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2017-05-01 01:26
New changeset 2e624690bd74071358566300b7ef0bc45f444a30 by Yury Selivanov (Jelle Zijlstra) in branch 'master':
bpo-29679: Implement @contextlib.asynccontextmanager (#360)
https://github.com/python/cpython/commit/2e624690bd74071358566300b7ef0bc45f444a30
History
Date User Action Args
2017-06-12 10:40:58yselivanovsetstatus: open -> closed
type: enhancement
resolution: fixed
stage: resolved
2017-05-01 01:26:00yselivanovsetmessages: + msg292647
2017-03-10 06:04:47ncoghlansetmessages: + msg289335
2017-03-03 04:02:46yselivanovsetmessages: + msg288851
2017-03-03 04:00:08ncoghlansetmessages: + msg288850
2017-03-02 01:44:39ncoghlansetmessages: + msg288781
2017-03-01 21:48:33rhettingersetmessages: + msg288777
2017-03-01 20:41:40hayposetnosy: - haypo
2017-03-01 17:43:32gvanrossumsetnosy: - gvanrossum
2017-03-01 15:37:51yselivanovsetmessages: + msg288759
2017-03-01 05:45:23ncoghlansetmessages: + msg288738
2017-03-01 05:11:29ncoghlansetmessages: + msg288736
2017-03-01 04:35:40rhettingersetnosy: + rhettinger
messages: + msg288734
2017-03-01 04:30:32Jelle Zijlstrasetmessages: + msg288733
2017-03-01 04:19:01ncoghlansetmessages: + msg288732
2017-03-01 04:16:37ncoghlansetnosy: + gvanrossum, ncoghlan, haypo, giampaolo.rodola, yselivanov
2017-03-01 00:19:07Jelle Zijlstrasetpull_requests: + pull_request307
2017-02-28 23:21:09Jelle Zijlstracreate