Author vstinner
Recipients gvanrossum, neologix, vstinner
Date 2014-02-03.23:34:15
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1391470456.74.0.460953910028.issue20505@psf.upfronthosting.co.za>
In-reply-to
Content
To solve a performance issue in asyncio, I added a new resolution attribute to selectors.BaseSelector and a new _granularity attribute to asyncio.BaseEventLoop. If I understood correctly, Charles-François (author and so maintainer of the new selectors module) doesn't like the new resolution attribute because it is a lie (if I understood correctly Charles-François complain).

He's right: the kernel can use a clock with a worse resolution. For example, on Linux kernel older than 2.6.28, all selectors (select, poll, epoll) had a resolution of 1/HZ where HZ is usually 100, 250 or 1000. So the best resolution was 1 ms (10^-3), whereas the timeval structure used in select() has a resolution of 1 us (10^-6).

On Windows, there is a different issue: the clock used by time.monotonic() has a resolution of 15.6 ms, whereas the type to describe the timeouf of the proactor selector (GetQueuedCompletionStatus function) has a resolution of 1 ms. So the resolution of the clock, time.get_clock_info('monotonic').resolution, should also be used.

Attached patch removes the two new attributes to make the code simpler.

I created this issue to reopen the discussion, but I would prefer to keep the attributes. I wrote the patch to be able to compare the different options. It's just to ensure that everyone agree on the solution.

The patch is a comprise, it solves partially the asyncio performance issue. With the patch, BaseEventLoop._run_once() may not executed a task even if there was a scheduled task, just because of timing rounding issues. So I modified the existing patch to tolerate "useless calls" to _run_once().

It was hard for me to analyze and choose the best solution for these issues because on my Fedora 20 (Linux kernel 3.12), the timings look "perfect". I'm unable to reproduce the sporadic timing issues (selector sleeping 8 ms or 9 ms whereas we asked to sleep 10 ms), whereas the test failed on many different buildbots. I suppose that depending on the hardware and the Linux kernel version, the exact resolution selectors is very different.

I didn't test the patch on Windows yet.

See issues #20311 and #20452 for the background and raw numbers.

Note: the test_timeout_rounding() test of test_asyncio now pass on all buildbots: various platforms, selectors and hardwares.

Note 2: If we decide to keep the BaseSelector.resolution attribute, I hesitate to replace it with a method because we might improve the read resolution later, which may require to execute code. For example, on Linux older than 2.6.28, the resolution can be computed by retrieving the HZ constant. It requires to check the OS and kernel version.
History
Date User Action Args
2014-02-03 23:34:17vstinnersetrecipients: + vstinner, gvanrossum, neologix
2014-02-03 23:34:16vstinnersetmessageid: <1391470456.74.0.460953910028.issue20505@psf.upfronthosting.co.za>
2014-02-03 23:34:16vstinnerlinkissue20505 messages
2014-02-03 23:34:16vstinnercreate