classification
Title: Fix types.coroutine to accept objects from Cython
Type: Stage: resolved
Components: Library (Lib) Versions: Python 3.6, Python 3.5
process
Status: closed Resolution: fixed
Dependencies: 24315 Superseder:
Assigned To: yselivanov Nosy List: gvanrossum, ncoghlan, python-dev, scoder, yselivanov
Priority: normal Keywords: patch

Created on 2015-05-28 16:31 by yselivanov, last changed 2015-05-29 21:11 by yselivanov. This issue is now closed.

Files
File name Uploaded Description Edit
coroutine.patch yselivanov, 2015-05-28 16:31 review
types_coroutine.patch yselivanov, 2015-05-29 17:05 review
types_coroutine.patch yselivanov, 2015-05-29 18:56 review
types_coroutine.patch yselivanov, 2015-05-29 20:02 review
types_coroutine.patch scoder, 2015-05-29 20:03 review
types_coroutine.patch yselivanov, 2015-05-29 20:07 review
Messages (14)
msg244315 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2015-05-28 16:31
Stefan,

This patch should solve the problem with types.coroutine accepting only pure python generator functions.

The approach is, however, slightly different from what you've proposed.  Instead of having a wrapper class (delegating .throw, .send etc to a wrapped object), we now simply check if the returned value of the wrapped function is an instance of collections.abc.Coroutine.  Issue 24315 enables duck typing for coroutines, so if a cython-based coroutine implements all coroutine abstract methods, it will automatically pass types.coroutine.
msg244373 - (view) Author: Roundup Robot (python-dev) Date: 2015-05-29 13:06
New changeset 7356f71fb0a4 by Yury Selivanov in branch '3.5':
Issue 24316: Fix types.coroutine() to accept objects from Cython
https://hg.python.org/cpython/rev/7356f71fb0a4

New changeset 748c55375225 by Yury Selivanov in branch 'default':
Issue 24316: Fix types.coroutine() to accept objects from Cython
https://hg.python.org/cpython/rev/748c55375225
msg244386 - (view) Author: Stefan Behnel (scoder) * Date: 2015-05-29 15:41
I just noticed that I hadn't used the real "types.coroutine" in my Py3.5 tests when reporting back in issue 24017. When I pass a Cython generator through it, I get

"""
Traceback (most recent call last):
  File "tests/run/test_coroutines_pep492.pyx", line 245, in test_coroutines_pep492.CoroutineTest.test_func_5 (test_coroutines_pep492.c:13445)
    for el in bar():
  File "/opt/python3.5/lib/python3.5/types.py", line 197, in wrapped
    'non-coroutine: {!r}'.format(coro))
TypeError: callable wrapped with types.coroutine() returned non-coroutine: <generator object at 0x7f178c458898>
"""

This is actually obvious, given that the sole purpose of the decorator is to turn something that is a Generator and *not* a Coroutine into something that is a Coroutine, as a means for the user to say "but I know better". So checking for the return value being a Coroutine is wrong. Instead, it should check that it's a Generator and if it's not an Awaitable, wrap it as a self-returning Awaitable. That's more or less what my proposed implementation in issue 24017 did:

  class types_coroutine(object):
    def __init__(self, gen):
        self._gen = gen

    class as_coroutine(object):
        def __init__(self, gen):
            self._gen = gen
            self.send = gen.send
            self.throw = gen.throw
            self.close = gen.close

        def __await__(self):
            return self._gen

    def __call__(self, *args, **kwargs):
        return self.as_coroutine(self._gen(*args, **kwargs))
msg244391 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2015-05-29 17:05
> I just noticed that I hadn't used the real "types.coroutine" in my Py3.5 tests when reporting back in issue 24017.

Please test thoroughly the attached patch.
msg244393 - (view) Author: Stefan Behnel (scoder) * Date: 2015-05-29 17:26
One failing test in "test_coroutines": test_func_5. The reason is that the GeneratorWrapper is not iterable (and there is no reason it shouldn't be, given that it wraps a Generator). That was my fault, I had already added an __iter__ method but didn't copy it in my previous message. Adding it as follows fixes the test for me:

    def __iter__(self):
        return self.__wrapped__

Alternatively, "__iter__ = __await__" would do the same.
msg244394 - (view) Author: Stefan Behnel (scoder) * Date: 2015-05-29 17:30
BTW, it's not only for compiled generators but also for normal Python functions that construct Python generators internally and return them, or that delegate the generator creation in some way. With this change, it's enough to decorate the constructor function and not each of the potential generators that it returns.
msg244399 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2015-05-29 18:56
Please test the attached patch.

> BTW, it's not only for compiled generators but also for normal Python functions that construct Python generators internally and return them

You're right, that's why I used "primarily" word in that comment ;) 

types.coroutine() is only used by asyncio.coroutine() so far, and returning generator objects from "factory" functions isn't a very common pattern in asyncio.
msg244406 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2015-05-29 20:02
Updated patch. Wrapper now proxies gi_code, gi_running and gi_frame
msg244408 - (view) Author: Stefan Behnel (scoder) * Date: 2015-05-29 20:03
Ok, now the problem with *this* patch is that __iter__ and __await__ are special methods that are being looked up on the type, not the instance. Similarly __next__, I think, as it also has its own (type) slot. But I think you're right that __next__ is also needed here.

I'm attaching a patch that works for me.
msg244409 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2015-05-29 20:07
> I'm attaching a patch that works for me.

Looks like we were working in parallel ;)  I've incorporated your changes.  Please look at the new patch (hopefully this one is final)
msg244412 - (view) Author: Stefan Behnel (scoder) * Date: 2015-05-29 20:11
Your latest patch works for me.
msg244413 - (view) Author: Roundup Robot (python-dev) Date: 2015-05-29 20:19
New changeset 8080b53342e8 by Yury Selivanov in branch '3.5':
Issue 24316: Wrap gen objects returned from callables in types.coroutine
https://hg.python.org/cpython/rev/8080b53342e8

New changeset c0434ef75177 by Yury Selivanov in branch 'default':
Issue 24316: Wrap gen objects returned from callables in types.coroutine
https://hg.python.org/cpython/rev/c0434ef75177
msg244414 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2015-05-29 20:21
Committed. Thanks, Stefan!
msg244419 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2015-05-29 21:11
Stefan, please take a look at this issue #24325 too.
History
Date User Action Args
2015-05-29 21:11:28yselivanovsetmessages: + msg244419
2015-05-29 20:21:09yselivanovsetstatus: open -> closed
resolution: fixed
messages: + msg244414
2015-05-29 20:19:48python-devsetmessages: + msg244413
2015-05-29 20:11:57scodersetmessages: + msg244412
2015-05-29 20:07:29yselivanovsetfiles: + types_coroutine.patch

messages: + msg244409
2015-05-29 20:03:24scodersetfiles: + types_coroutine.patch

messages: + msg244408
2015-05-29 20:02:54yselivanovsetfiles: + types_coroutine.patch

messages: + msg244406
2015-05-29 18:56:41yselivanovsetfiles: + types_coroutine.patch

messages: + msg244399
2015-05-29 18:54:13yselivanovsetstatus: closed -> open
resolution: fixed -> (no value)
2015-05-29 17:30:47scodersetmessages: + msg244394
2015-05-29 17:26:16scodersetmessages: + msg244393
2015-05-29 17:05:40yselivanovsetfiles: + types_coroutine.patch

messages: + msg244391
2015-05-29 15:41:41scodersetmessages: + msg244386
2015-05-29 13:07:03yselivanovsetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2015-05-29 13:06:37python-devsetnosy: + python-dev
messages: + msg244373
2015-05-28 16:42:29yselivanovlinkissue24017 dependencies
2015-05-28 16:32:02yselivanovsetdependencies: + collections.abc: Coroutine should be derived from Awaitable
2015-05-28 16:31:52yselivanovcreate