classification
Title: Add asyncio (tulip, PEP 3156) to stdlib
Type: enhancement Stage: patch review
Components: Versions: Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: David.Edelsohn, benjamin.peterson, georg.brandl, giampaolo.rodola, gvanrossum, koobs, larry, ned.deily, pitrou, python-dev, sbt, serhiy.storchaka, skrah, vstinner
Priority: release blocker Keywords: needs review

Created on 2013-10-14 21:18 by gvanrossum, last changed 2013-10-19 05:00 by gvanrossum. This issue is now closed.

Files
File name Uploaded Description Edit
asyncio2.patch gvanrossum, 2013-10-14 22:16 review
asyncio3.patch gvanrossum, 2013-10-14 22:50 review
asyncio4.patch gvanrossum, 2013-10-15 03:28 review
asyncio5.patch gvanrossum, 2013-10-15 16:15 review
asyncio6.patch gvanrossum, 2013-10-15 23:53 review
asyncio7.patch gvanrossum, 2013-10-16 19:12 review
asyncio8.patch gvanrossum, 2013-10-17 19:41 review
koobs-freebsd9-py3x-build178.log koobs, 2013-10-18 09:08
koobs-freebsd9-py3x-build180.log koobs, 2013-10-18 09:09
koobs-freebsd9-py3x-build182.log koobs, 2013-10-18 09:09
Messages (45)
msg199953 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2013-10-14 21:18
I'll add the tests next.  The plan is to get this in the next (last) alpha release.
msg199954 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-10-14 21:21
It will probably need some messaging to integrate the Windows overlapped stuff?
msg199955 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-10-14 21:21
(massaging, probably)
msg199957 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2013-10-14 21:28
If you need help with docs, let me know.
msg199959 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2013-10-14 22:16
Here's an updated patch that doesn't botch the selectors imports.
msg199962 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2013-10-14 22:23
What I need help with most right now is a suggestion of where the unittests should go. There are too many of them to all put in one file. Some options:

asyncio/test/*_test.py
asyncio/test/test_*.py
test/test_asyncio/<either of the above>

What's the best current practice?

(I also want to add some hacks so that the files can actually be identical in the stdlib and in the separate 3rd party repo, otherwise keeping the two in sync will be too much work.)
msg199963 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-10-14 22:33
> asyncio/test/*_test.py
> asyncio/test/test_*.py
> test/test_asyncio/<either of the above>

Personally I have a preference for test/test_asyncio/test_*.py, because
putting all tests in test/ makes them easier to find.

However, other packages such as unittest have their dedicated test
packages (unittest/test/...).

As for test_*.py vs. *_test.py, test_*.py is definitely the norm in the
source tree.
msg199965 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2013-10-14 22:50
OK, here's a new patch that includes tests.

To run the tests, I use:

./python.exe Lib/test/regrtest.py  --fromfile Lib/test/test_asyncio/tests.txt --verbose

There are a total of 4 individual test failures, all having to do with SSL:

ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:589)

Did anything change in the ssl module recently?
msg199973 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2013-10-15 03:28
Found the cause of the ssl test failure -- the location of the ssl test key and cert are different.  Here's a new patch with a quick fix (#4), but I think the correct solution is to either have the certificates inline in the source and write them to a temp file, or move them into the main asyncio library, or move the test_utils.py module -- since the test files are referenced from test_utils.py they should be in the same directory.  Preferences?

There are still some test failures, the Windows tests are being run everywhere.  I'll look into how to do that later.
msg199981 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-10-15 08:04
> Found the cause of the ssl test failure -- the location of the ssl
> test key and cert are different.  Here's a new patch with a quick
> fix (#4), but I think the correct solution is to either have the
> certificates inline in the source and write them to a temp file, or
> move them into the main asyncio library, or move the test_utils.py
> module -- since the test files are referenced from test_utils.py
> they should be in the same directory.  Preferences?

You could simply reuse Lib/test/keycert.pem (when in stdlib mode).
msg200006 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2013-10-15 16:15
Turns out there were other uses of the sample key/cert pair. The easiest solution is to just have the code try both locations if necessary. Here's a new patch to review.
msg200028 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2013-10-15 23:36
Here's a partial patch for Windows. (Mostly for myself; I need to integrate this into the main patch.)
msg200029 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2013-10-15 23:53
Here's a full new patch, with Windows project/solution changes included, and updated from the latest Tulip asyncio branch.
msg200030 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2013-10-16 00:03
PS. There's some garbage at the start of pcbuild.sln (perhaps a BOM mark?). I'm not going to re-upload the patch for now, but please note this.
msg200072 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2013-10-16 19:12
New patch, mostly SSL hardening IIRC.
msg200073 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2013-10-16 19:14
I could use some help for two issues with the tests:

(1) How do I stop regrtest.py from running the windows tests?  (These import _winapi.)
2 tests failed:
    test_asyncio.test_windows_events test_asyncio.test_windows_utils


(2) I get this message -- what does it mean and should I care?
2 tests altered the execution environment:
    test_asyncio.test_base_events test_asyncio.test_futures
msg200075 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-10-16 19:30
1) The test case decorator perhaps:

@unittest.skipIf(sys.platform == "win32", "Does not apply to Windows")
msg200077 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-10-16 19:35
2) See class saved_test_environment in regrtest.py.  I wouldn't worry too much about it initially but it should be looked at and tidied up before release.
msg200078 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2013-10-16 19:39
On 16/10/2013 8:14pm, Guido van Rossum wrote:
> (2) I get this message -- what does it mean and should I care?
> 2 tests altered the execution environment:
>      test_asyncio.test_base_events test_asyncio.test_futures

Perhaps threads from the ThreadExecutor are still alive when those tests 
finish.
msg200079 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2013-10-16 19:40
I'd have to decorate a lot of tests. Is there a way to fix this at the module or at least class level? (I'd be willing to move the imports around.)
msg200080 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2013-10-16 19:58
I think at module level you can do

     if sys.platform != 'win32':
         raise unittest.SkipTest('Windows only')
msg200081 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2013-10-16 20:23
Yup, that works!  Not uploading a new patch right now but this is in the tulip repo now.
msg200082 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2013-10-16 20:25
You can skip classes with skipIf as a class decorator.
msg200156 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2013-10-17 19:41
Another day, another patch.

I'd rather like to commit this (and then iterate as needed), it makes my workflow for porting to Windows a little easier.

Larry???
msg200159 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2013-10-17 19:59
Does it break anything?  (Besides, possibly, itself?)
msg200161 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2013-10-17 20:05
It doesn't break anything AFAICT. (It's a pure addition except for one new
extension module on Windows).

However I just discovered that apparently regrtest doesn't automatically
run tests in a subdirectory of the test package. I'm guessing I'll need to
add some magic to test_asyncio/__init__.py. Any hints?

On Thu, Oct 17, 2013 at 12:59 PM, Larry Hastings <report@bugs.python.org>wrote:

>
> Larry Hastings added the comment:
>
> Does it break anything?  (Besides, possibly, itself?)
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue19262>
> _______________________________________
>
msg200163 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2013-10-17 20:08
If its breakage is restricted to itself (and its tests) then you have my blessing to check it in.  (Sorry, can't help you with the test expertise.)
msg200167 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2013-10-17 20:45
OK, no more giant patches. It's checked in. I've also solved the regrtest issues by adding some code in __init__.py and __main__.py.

I expect to iterate a bit over the next few days.
msg200179 - (view) Author: Roundup Robot (python-dev) Date: 2013-10-18 01:09
New changeset 30f33e6a04c1 by Ned Deily in branch 'default':
Issue #19262: Install asyncio and test_asyncio directories.
http://hg.python.org/cpython/rev/30f33e6a04c1
msg200188 - (view) Author: Kubilay Kocak (koobs) Date: 2013-10-18 07:38
There are 5 unique test failures that have come up in the koobs-freebsd* buildbots post the test_asyncio import. Would we prefer to create a meta issue to track them, or put them here?
msg200231 - (view) Author: Kubilay Kocak (koobs) Date: 2013-10-18 09:08
Summary of 4 test failures below, will attach the complete buildbot logs for detail.

======================================================================
FAIL: test_call_later (test.test_asyncio.test_events.SelectEventLoopTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/test/test_asyncio/test_events.py", line 241, in test_call_later
    self.assertTrue(0.09 <= t1-t0 <= 0.12, t1-t0)
AssertionError: False is not true : 0.12008954107295722

======================================================================
FAIL: test_signal_handling_args (test.test_asyncio.test_events.SelectEventLoopTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/test/test_asyncio/test_events.py", line 468, in test_signal_handling_args
    self.assertEqual(caught, 1)
AssertionError: 0 != 1

======================================================================
FAIL: test_create_server_ssl (test.test_asyncio.test_events.KqueueEventLoopTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/test/test_asyncio/test_events.py", line 626, in test_create_server_ssl
    self.assertEqual(3, proto.nbytes)
AssertionError: 3 != 0


======================================================================
Timeout (1:00:00)!
<snip>
Exception: Child error on test_asyncio: Exit code 1
----------------------------------------------------------------------
msg200235 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-10-18 09:49
> ======================================================================
> FAIL: test_call_later
> (test.test_asyncio.test_events.SelectEventLoopTests)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File
>   "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/test/test_asyncio/test_events.py",
>   line 241, in test_call_later
>     self.assertTrue(0.09 <= t1-t0 <= 0.12, t1-t0)
> AssertionError: False is not true : 0.12008954107295722

This one is a classical timing issue. The test is too optimistic:
many buildbots can be quite slow or loaded. Other timing tests in the
stdlib allow for much more slack.

(e.g. call_later with 0.5 and check that the resulting delay
is between 0.4 and 1.0)
msg200261 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2013-10-18 14:17
@Koobs: I'll look into these, but in the future it's better to report bugs "upstream" for now, i.e. at http://code.google.com/p/tulip/ -- they will get my immediate attention.

@Antoine: while most of the timing-related tests use a simulated clock, there are still some that must use real time. I'll fix them as needed.
msg200268 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2013-10-18 15:02
I fixed the easy one (the expected delay in test_call_later). I could use some hands with the rest -- I suspect there are similar race conditions.

I'm tracking this now in http://code.google.com/p/tulip/issues/detail?id=75
msg200283 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2013-10-18 16:49
The --without-threads buildbot fails, so I guess all tests need to be
skipped in that case:

http://buildbot.python.org/all/builders/AMD64%20Fedora%20without%20threads%203.x/builds/5333/steps/test/logs/stdio
msg200288 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2013-10-18 17:22
Maybe adding something that returns [] from suite() if therea re no threads in test/test_asyncio/__init__.py would help?  I don't have time to test this, but go ahead and commit something if it's a release blocker.

Even better would or course be to fix asyncio to actually work if there are no threads -- the only issue is what to do about run_in_executor(), I guess it will have to run the function in-line...
msg200307 - (view) Author: David Edelsohn (David.Edelsohn) * Date: 2013-10-18 21:29
AIX buildbot is experiencing a similar failure:

[276/382/4] test_asyncio
Timeout (1:00:00)!
Thread 0x00000001:
  File "/home/shager/cpython-buildarea/3.x.edelsohn-aix-ppc64/build/Lib/selectors.py", line 265 in select
  File "/home/shager/cpython-buildarea/3.x.edelsohn-aix-ppc64/build/Lib/asyncio/base_events.py", line 576 in _run_once
  File "/home/shager/cpython-buildarea/3.x.edelsohn-aix-ppc64/build/Lib/asyncio/base_events.py", line 153 in run_forever
  File "/home/shager/cpython-buildarea/3.x.edelsohn-aix-ppc64/build/Lib/asyncio/base_events.py", line 172 in run_until_complete
  File "/home/shager/cpython-buildarea/3.x.edelsohn-aix-ppc64/build/Lib/test/test_asyncio/test_events.py", line 1012 in test_subprocess_interactive
msg200309 - (view) Author: Kubilay Kocak (koobs) Date: 2013-10-18 21:34
@Guido, another expected delay test failure:

======================================================================
FAIL: test_run_until_complete (test.test_asyncio.test_events.KqueueEventLoopTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd9/build/Lib/test/test_asyncio/test_events.py", line 218, in test_run_until_complete
    self.assertTrue(0.08 <= t1-t0 <= 0.12, t1-t0)
AssertionError: False is not true : 0.12018337799236178

Note: Also referenced "upstream" :]
msg200317 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2013-10-18 22:23
Relaxed a bunch of timeouts. No news on the --without-threads case, that
will have to happen post-alpha-4 I expect.
msg200319 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2013-10-18 22:31
I've landed a bunch of stuff, and now I am pretty happy. It's also soon going to be weekend, which means family time, so I really hope everyone else is also happy. :-)

A summary of what changed since the initial asyncio checkin:

- Rename the logger to plain "logger".
- Rename Transport.pause/resume to pause_reading/pause_writing.
- Important race condition fix.
- Write flow control for asyncio! (And asyncio/streams.py overhaul.)

And in the tests:

- Make the tests run on Windows.
- Relax various test timeouts to reduce flakiness on slow buildbots.
msg200320 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2013-10-18 22:31
Ready for release!
msg200323 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2013-10-18 22:47
2013/10/19 Guido van Rossum <report@bugs.python.org>:
> Ready for release!
>
> resolution:  -> fixed
> status: open -> closed

The new module has no documentation at all. Do you plan to open a new
issue for the documentation? Or maybe you don't plan to add
documentation? :-)
msg200325 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2013-10-18 22:59
I'll track that separately: http://bugs.python.org/issue19291

For now, PEP 3156 has a lot of information.  (But the PEP also needs to be updated to track recent developments.)
msg200363 - (view) Author: David Edelsohn (David.Edelsohn) * Date: 2013-10-19 04:11
How is this ready for release? The patch does not work on numerous POSIX systems.
msg200369 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2013-10-19 05:00
Please be more specific.
History
Date User Action Args
2013-10-19 05:00:06gvanrossumsetmessages: + msg200369
2013-10-19 04:11:19David.Edelsohnsetmessages: + msg200363
2013-10-18 22:59:32gvanrossumsetmessages: + msg200325
2013-10-18 22:47:11vstinnersetmessages: + msg200323
2013-10-18 22:31:56gvanrossumsetstatus: open -> closed
resolution: fixed
messages: + msg200320
2013-10-18 22:31:32gvanrossumsetmessages: + msg200319
2013-10-18 22:23:40gvanrossumsetmessages: + msg200317
2013-10-18 21:34:24koobssetmessages: + msg200309
2013-10-18 21:29:38David.Edelsohnsetnosy: + David.Edelsohn
messages: + msg200307
2013-10-18 17:22:07gvanrossumsetmessages: + msg200288
2013-10-18 16:49:02skrahsetnosy: + skrah
messages: + msg200283
2013-10-18 15:02:02gvanrossumsetmessages: + msg200268
2013-10-18 14:17:54gvanrossumsetmessages: + msg200261
2013-10-18 09:49:04pitrousetmessages: + msg200235
2013-10-18 09:09:42koobssetfiles: + koobs-freebsd9-py3x-build182.log
2013-10-18 09:09:19koobssetfiles: + koobs-freebsd9-py3x-build180.log
2013-10-18 09:08:35koobssetfiles: + koobs-freebsd9-py3x-build178.log

messages: + msg200231
2013-10-18 07:38:55koobssetnosy: + koobs
messages: + msg200188
2013-10-18 01:09:10python-devsetnosy: + python-dev
messages: + msg200179
2013-10-17 20:46:18gvanrossumsetkeywords: + needs review, - patch
2013-10-17 20:45:47gvanrossumsetmessages: + msg200167
2013-10-17 20:08:49larrysetmessages: + msg200163
2013-10-17 20:05:31gvanrossumsetmessages: + msg200161
2013-10-17 19:59:33larrysetmessages: + msg200159
2013-10-17 19:42:06gvanrossumsetfiles: + asyncio8.patch

messages: + msg200156
2013-10-16 20:25:17benjamin.petersonsetnosy: + benjamin.peterson
messages: + msg200082
2013-10-16 20:23:26gvanrossumsetmessages: + msg200081
2013-10-16 19:58:57sbtsetmessages: + msg200080
2013-10-16 19:40:17gvanrossumsetmessages: + msg200079
2013-10-16 19:39:41sbtsetmessages: + msg200078
2013-10-16 19:36:00ned.deilysetmessages: + msg200077
2013-10-16 19:30:05ned.deilysetnosy: + ned.deily
messages: + msg200075
2013-10-16 19:14:27gvanrossumsetmessages: + msg200073
2013-10-16 19:12:44gvanrossumsetfiles: + asyncio7.patch

messages: + msg200072
2013-10-16 19:08:05giampaolo.rodolasetnosy: + giampaolo.rodola
2013-10-16 00:03:56gvanrossumsetmessages: + msg200030
2013-10-15 23:53:33gvanrossumsetfiles: - winasyncio.diff
2013-10-15 23:53:08gvanrossumsetfiles: + asyncio6.patch

messages: + msg200029
2013-10-15 23:36:48gvanrossumsetfiles: + winasyncio.diff

messages: + msg200028
2013-10-15 16:16:00gvanrossumsetfiles: + asyncio5.patch

messages: + msg200006
2013-10-15 08:24:17vstinnersetnosy: + vstinner
2013-10-15 08:04:38pitrousetmessages: + msg199981
2013-10-15 03:28:29gvanrossumsetfiles: + asyncio4.patch

messages: + msg199973
2013-10-14 22:50:56gvanrossumsetfiles: + asyncio3.patch

messages: + msg199965
2013-10-14 22:33:16pitrousetmessages: + msg199963
2013-10-14 22:23:57gvanrossumsetmessages: + msg199962
2013-10-14 22:16:13gvanrossumsetfiles: + asyncio2.patch
keywords: + patch
messages: + msg199959
2013-10-14 21:28:24georg.brandlsetnosy: + georg.brandl
messages: + msg199957
2013-10-14 21:26:27serhiy.storchakasettitle: Add asyncio (tulip, PEP 3156) to stdlin -> Add asyncio (tulip, PEP 3156) to stdlib
2013-10-14 21:21:26pitrousetmessages: + msg199955
2013-10-14 21:21:02pitrousetnosy: + sbt
messages: + msg199954
2013-10-14 21:19:24serhiy.storchakasetnosy: + serhiy.storchaka
2013-10-14 21:18:19gvanrossumcreate