classification
Title: asyncio: set TCP_NODELAY flag by default
Type: performance Stage: resolved
Components: asyncio Versions: Python 3.5, Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: haypo, j1m, python-dev, socketpair, yselivanov
Priority: normal Keywords:

Created on 2016-07-05 15:27 by j1m, last changed 2016-09-12 01:44 by yselivanov. This issue is now closed.

Messages (13)
msg269826 - (view) Author: Jim Fulton (j1m) * (Python committer) Date: 2016-07-05 15:27
tl;dr TCP_NODELAY should be set by default and/or there should be a
      proper way to set it.

I've ported ZEO, ZODB's client-server networking layer to asyncio.
Things were going pretty well.  I've been developing on a
Mac. Yesterday, I ran some performance measurements on Linux and found
some tests ran 30x slower.

After some digging, I came across this:

https://github.com/python/asyncio/issues/311

Then led me to try setting TCP_NODELAY, which provided Linux
performance comparable to Mac OS performance.

Issue 311 suggested that this was a kernal-version issue.  I think
this is a failure to set, or at least provide a way to set a "don't be
stupid" option.

I originally tried this on Ubuntu 14.04, with kernal
3.13.0-74-generic.  I then tried Ubuntu 16.04, in a docker image, with
kernal 4.4.12-boot2docker. For both of these, performance for the
write tests were 30x slower unless I set TCP_NODELAY.

Finally, I tried an Amazon standard AMI, which runs some version of
RedHat, with kernal 4.4.11-23.53.amzn1.x86_64.  On that system,
performance of the tests was similar to Mac OS X without setting
TCP_NODELAY.

Note that the non-slow kernal version was lower than the slow (Ubuntu)
one. I don't think this is mearly a kernal version issue, nor do I
think this should have been dismissed as one.

I couldn't find a way to set TCP_NODELAY cleanly. Did I miss something?
https://github.com/python/asyncio/issues/286 suggests there isn't one.

I ended up having to set the option on _sock transport attributes,
which is dirty and, I assume, won't work with uvloop. (BTW, uvloop was
only ~15x slower on linux systems with this problem.)

I think TCP_NODELAY should be set by default.  Perhaps it shouldn't be
set on mobile, by everywhere else, I think it's a "don't be stupid"
option.

I also think there should be a way to set it cleanly.

IMO, this is extremely important. Linux is a wildly important platform
for networking applications and Python, and, for better or worse,
Ubuntu is a very commonly used distribution.  Having asyncio, perform
so poorly on these platforms is a big deal.
msg269831 - (view) Author: Jim Fulton (j1m) * (Python committer) Date: 2016-07-05 16:00
Gaaa, forgot to set meta data.
msg269836 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2016-07-05 16:15
See also https://github.com/python/asyncio/issues/286

I also wanted to raise this question.  I think transports should set NODELAY by default (as Golang, for instance).  I've been hit by this thing a few times already -- performance of protocols over TCP is terrible because somebody forgot to set this flag.
msg269837 - (view) Author: Jim Fulton (j1m) * (Python committer) Date: 2016-07-05 16:24
I forgot to switch to the asyncio branch of ZEO when testing on the Amazon/RedHat linux. When I do, the tests run slow there too. Asyncio is dog slow on every linux I've tried.
msg269838 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2016-07-05 16:27
Fine with me -- I have no idea what this flag does (nor the desire to learn :-).
msg269839 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2016-07-05 16:28
Jim, I can make a PR, unless you want to do that.
msg269840 - (view) Author: Jim Fulton (j1m) * (Python committer) Date: 2016-07-05 16:30
Yury, I'd be fine with you making a PR. :)

Is there a similar update that can be made to uvloop?
msg269841 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2016-07-05 16:37
> Is there a similar update that can be made to uvloop?

Yes.  Once it's committed to asyncio, I'll add it to uvloop immediately.  That's one benefit of using uvloop -- it gets updates faster ;)
msg269857 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2016-07-05 23:30
PR: https://github.com/python/asyncio/pull/373
msg269933 - (view) Author: Jim Fulton (j1m) * (Python committer) Date: 2016-07-07 13:14
I missed the point that you can get a transport's socket using get_extra_info. IMO, this provides an adequate escape from the default and qualifies as a "proper way to set it".

I still think the default should be to enable TCP_NODELAY.
msg274251 - (view) Author: Марк Коренберг (socketpair) * Date: 2016-09-02 15:50
vote +10 for that
msg275907 - (view) Author: Roundup Robot (python-dev) Date: 2016-09-12 01:44
New changeset 3f7e4ae9eba3 by Yury Selivanov in branch '3.5':
Issue #27456: asyncio: Set TCP_NODELAY by default.
https://hg.python.org/cpython/rev/3f7e4ae9eba3

New changeset 10384c5c18f5 by Yury Selivanov in branch 'default':
Merge 3.5 (issue #27456)
https://hg.python.org/cpython/rev/10384c5c18f5
msg275908 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2016-09-12 01:44
Committed, should be in 3.6 b1.
History
Date User Action Args
2016-09-12 01:44:38yselivanovsetstatus: open -> closed
resolution: fixed
messages: + msg275908

stage: resolved
2016-09-12 01:44:09python-devsetnosy: + python-dev
messages: + msg275907
2016-09-02 15:50:00socketpairsetnosy: + socketpair
messages: + msg274251
2016-09-02 11:10:06hayposettitle: TCP_NODELAY -> asyncio: set TCP_NODELAY flag by default
2016-07-07 13:14:44j1msetmessages: + msg269933
2016-07-05 23:30:23yselivanovsetmessages: + msg269857
2016-07-05 16:37:10yselivanovsetmessages: + msg269841
2016-07-05 16:30:43j1msetmessages: + msg269840
2016-07-05 16:30:19gvanrossumsetnosy: - gvanrossum
2016-07-05 16:28:13yselivanovsetmessages: + msg269839
2016-07-05 16:27:12gvanrossumsetmessages: + msg269838
2016-07-05 16:24:59j1msetmessages: + msg269837
2016-07-05 16:15:00yselivanovsetmessages: + msg269836
2016-07-05 16:00:20j1msetversions: + Python 3.4, Python 3.5
nosy: + gvanrossum, haypo, yselivanov

messages: + msg269831

components: + asyncio
type: performance
2016-07-05 15:27:48j1mcreate