We could use a How To or a tutorial for asyncio in the docs.

Yes, this is a very longstanding open issue. I've already discussed this with Guido many times, and even found someone to help us. Will try to kickstart the project this weekend.

See Issue 27579, where Victor wanted to focus on <https://github.com/asyncio-doc/asyncio-doc> outside of Python.

I think it will still be useful to have an authoritative tutorial within CPython docs.

Mariatta Wijaya added the comment:

I think it will still be useful to have an authoritative tutorial within
CPython docs.

The problem is that Python stdlib is quite limited: no HTTP, no SQL client,
etc. An external tutorial can use 3rd party modules like aiohttp.

> The problem is that Python stdlib is quite limited: no HTTP, no SQL client,
etc. An external tutorial can use 3rd party modules like aiohttp.

Even without aiohttp we can show how to use asyncio correctly. I agree with Mariatta that we should have a tutorial etc. Working on it.

Sounds great! Thanks, Yury!

Victor, do you think it's worth including a link to http://asyncio.readthedocs.io/en/latest/ somewhere in the docs, if we haven't already?

Le 1 mai 2017 19:06, "Mariatta Wijaya" <report@bugs.python.org> a écrit :

Mariatta Wijaya added the comment:

Sounds great! Thanks, Yury!

Victor, do you think it's worth including a link to
http://asyncio.readthedocs.io/en/latest/ somewhere in the docs, if we
haven't already?

Yes!

A while ago, Eric Appelt stepped forward to help with the asyncio 
documentation.  Here's my response to his email in which we discuss
the direction for the new docs.

On May 24, 2017 at 1:04:56 PM, Eric Appelt (eric.appelt@gmail.com) wrote:
> Hi Yury,
> 
> I (briefly) went through the curio and trio tutorials and wanted to
> summarize a few thoughts. I'm going to go through them again slowly and
> actually do the exercises, but a few things that I found notable:
> 
> Both Curio and Trio start with an emphasis of tasks, spawning tasks at
> various points, and then later joining them. Trio even has a "nursery"
> async context manager where you spawn tasks inside and then they are all
> joined on exit. Following various exchanges, I understand that the
> coroutine is to be the preferred abstraction layer for use in the
> application developer, i.e. one would generally be using
> asyncio.gather(...) rather than create_task as the routine way of invoking
> concurrency. If that's correct, I would expect an asyncio tutorial to be a
> bit more focused on using gather, but I expect that manipulating tasks is
> of enough interest that it deserves some discussion.

What I like about Trio documentation is that it tries to explain 
async/await first, gradually paving the way to more advanced concepts
and the library API.

I’m going to propose a concrete structure for the revamped documentation.
It’s not something set in stone, let’s discuss and potentially 
restructure it:

1. Tutorial
 a/ Why async IO?
 b/ async/await in Python
 c/ Simple example in asyncio (Tasks + sleep) + explanation
 d/ An example of using aiohttp
 e/ How asyncio works and what is the event loop
 f/ Tasks, asyncio.gather, wait_for, cancellation

2. Advanced Tutorial
 a/ A short primer on network IO
 b/ Let’s implement a memcache driver using streams!
 c/ Let’s implement a memcache driver with transports!

3. API reference — there are a few things we’ll need to restructure
there.


In more detail:

1a - we need to answer the question of why would you even bother
learning asyncio and async/await.  What are the benefits?  When do
you want to use asyncio and when you really shouldn’t.

1b - we need to explain async/await in a friendly way.  In particular,
you don’t need to understand the details of the async/await protocol 
etc.  What you need to know is that there are two types of functions
in Python: regular functions and coroutines.  Coroutines can be
paused and resumed in: `await`, `async with`, and `async for` 
statements.  Fundamentally, this is all you need to know about
async/await in Python. You simply use `await` for coroutines, that’s
it.

1c - A simple example similar to
https://curio.readthedocs.io/en/latest/tutorial.html#getting-started

1d - Here I propose to just use aiohttp to implement a simple
web client or hello-world-server.  I think it can be a valuable
early tutorial point to show people a real example, something they
will actually need and use asyncio for.  It’s not a problem to
use some non-standard library in the tutorial.

1e - Here we’ll need to discuss some fundamental asyncio building
blocks: event loop, callbacks, Future, and Task.  Briefly, the point
is to get the user familiar with the terminology and give them a
high-level understanding of how things work.

1f - All high-level asyncio primitives that everyone needs to know:
Tasks, gather, wait_for, etc.  No more than 5 API functions should
be covered here, culminating in a simple example.


2a - Now we are entering the “advanced” part of the tutorial.  Let’s
explain sockets and selectors and how we tie them to callbacks using
the loop.

2b - I propose to take a simple protocol like Memcache or 
Redis and simply implement it using the streams API.  We’ll only
need two methods: set and get; and in the end we’ll teach the user
how things really work and how to design async APIs.

2c - Same as 1h but with Transports and Protocols.  Explain how
to decouple protocol parser from the IO and the benefits of that.
Show that the end result is faster than 1h, and not that much
more complex.


3 - We’ll need to restructure the current API reference a bit.  Things
that we will cover in the tutorial won’t be needed to be explained 
in detail again.  The idea is to get a simple to digest, concise
API reference.

Some things are currently documented in several different places. Like
event loops / policies, I believe we have three different pages 
covering them.  Some things aren’t needed to be fully covered at all,
like `AbstractEventLoop`.

Anyways, I think that ‘3’ is an easy part. The hard part is the
tutorial because it’s completely new.


> 
> Curio has a nice discussion of tasks that consume lots of CPU time
> crunching numbers, and the cooperative nature of coroutine multitasking,
> which I think is helpful.

Yes, I agree.  Maybe this should go to 1a.

> 
> Trio starts with a nice discussion of “regular” and “async” functions, good
> explanation of how coroutine objects get created and run.

We need this too: 1b.

> 
> Both Curio and Trio have a good discussion of their respective monitoring
> tools allowing the beginner to watch what is happening without peppering
> the code with too many prints and sleeps.

We don’t have this yet.  I think I saw a package on github 
(in aio-libs, IIRC) that implements this.  If it works we can use it in
the tutorial.

> 
> They both start with coroutines that sleep and print. Curio has a more
> entertaining example, and they both have the traditional socket echo
> server. On this point it seems somewhat unfortunate - IMO - from the
> perspective of an application developer print/sleep and socket examples are
> not the most compelling examples. The print/sleep examples aren't doing
> anything useful, and sockets are more low level than the typical
> application developer has to worry about. In my own professional
> experience, I have rarely had to use the python socket library - really
> just once to talk to HAProxy. What would be more interesting would be web
> client/servers, but this would rely on a third party library.
> 
> Curio has examples of a publish/subscribe system, and ends in a chat
> server. I really like this tutorial example as it highlights one of the
> most obvious situations where coroutines are useful - when you have a
> zillion connected clients doing nothing almost all of the time.

Yeah, this sounds entertaining.  Let’s have it too: 1d?

> 
> Somewhat related I gave a talk at PyTennessee on async/await/asyncio for
> novices and I submitted a tutorial to PyOhio that I might be able to draw
> from in this project. The one difference is that I'm able to rely on third
> party libraries (requests, aiohttp) and have a little more fun with real IO
> without having to get into sockets which I think is less familiar for most
> novices than simple http. Much like the curio example, I was playing with
> simple publish/subscribe examples possibly to use with the tutorial:
> https://gist.github.com/appeltel/fd3ddeeed6c330c7208502462639d2c9

This might be a bit too advanced for 1a-1f, we can have something like
this in the advanced tutorial.

> 
> I feel like now would be a good time to get started - is there already an
> outline of a new asyncio tutorial that I can help with, or is it worth
> taking some of these ideas from Curio and Trio and drafting out a prototype
> outline? Please let me know what you think.
> 

I’ll ask Brett if we can have a cpython-aiodocs fork under CPython 
organization.  This way we’ll be able to grant push privileges easily
to those who’s interested to work on this, and we’ll also have Issues/PRs
for workflow.

> 2b - I propose to take a simple protocol like Memcache or 
> Redis and simply implement it using the streams API.  We’ll only
> need two methods: set and get; and in the end we’ll teach the user
> how things really work and how to design async APIs.

I decided to make myself a test subject for this idea using Redis since although I use Redis a lot at work I've never looked at the underlying Redis protocol. I also haven't used the streams API in a long time, and never for anything other than simple exploration.

I worked on this the other night and found the Redis protocol simple enough to understand and was able to write a simple and somewhat clumsy client the other night with a bit of work: https://gist.github.com/appeltel/09d77eb489494ae1e2703933108cb60a

One thing that might be good about the Redis protocol for this purpose is that the parsing isn't completely trivial but it isn't overly complex either. For example, a response string can be given in either the simple format, which begins with a "+" byte and then terminates with "\r\n", or it can be in a binary safe format where a token of the form "$123\r\n" is sent first, signaling that the next 123 bytes are the result to be followed by another "\r\n" termination. There is also the sentinel value "$-1\r\n" which may signal an unset key. Thus the string "foo" might be sent as "+foo\r\n" or "$3\r\nfoo\r\n".

So I think this can be reasonably described in a few brief paragraphs and it is much more interesting (to me) than an echo client/server example.

The tutorial should probably also include a link to a rudimentary server implementation of an in-memory key/value store implementing the Redis protocol and supporting just GET and SET, as the reader may not be able to easily stand up Redis for a variety of reasons or lack of knowledge.

I can experiment with memcached as well but I think this proposal is a good idea and would work well with Redis.

I've created a repo for discussing and making next version of asyncio docs: https://github.com/asyncio-docs/cpython-aiodocs. Feel free to join!

I propose to start using the issues [1] as there are so many topics we'll have to discuss.  Feel free to create new issue for whatever idea you have!

[1] https://github.com/asyncio-docs/cpython-aiodocs/issues

> I can experiment with memcached as well but I think this proposal is a good idea and would work well with Redis.

I like it! Let's use Redis as an example. Please open an issue at asyncio-docs to discuss the details.

Closing this issue. I'll open a new one for the planned asyncio docs overhaul.