classification
Title: Asyncio reuseport
Type: behavior Stage:
Components: asyncio Versions: Python 3.6, Python 3.4, Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: gvanrossum Nosy List: Boris.FELD, chris laws, gvanrossum, j1o1h1n, python-dev, vstinner, yselivanov, ysionneau
Priority: normal Keywords: patch

Created on 2015-04-16 11:52 by Boris.FELD, last changed 2015-10-06 15:25 by python-dev. This issue is now closed.

Files
File name Uploaded Description Edit
23972_cjl.patch chris laws, 2015-07-15 15:55 Patch, tests and docs updates for 23972 review
23972_cjl_v002.patch chris laws, 2015-08-23 05:36 Updated patch, tests, docs for 23972 review
23972_cjl_v003.patch chris laws, 2015-09-29 09:53 rebase patch, tests, docs for 23972 review
23972_cjl_v004.patch chris laws, 2015-10-03 03:29 Patch, tests, docs updates for 23972
23972_cjl_v005.patch gvanrossum, 2015-10-03 16:23 review
23972_cjl_v006.patch chris laws, 2015-10-06 08:00 refine tests to also work on Windows and Debian platforms review
Messages (16)
msg241213 - (view) Author: Boris FELD (Boris.FELD) * Date: 2015-04-16 11:52
I'm trying to create some UDP sockets for doing multicast communication. I have the working code in synchronous way and try to port it to asyncio.

One last issue is blocking for me, I'm on Mac OS X and for multicast UDP to work, the SO_REUSEPORT must be set on socket before bind. The problem is that I don't have access on socket, it's created inside asyncio method _create_connection_transport.

I've seen that SO_REUSEADDR is already set (https://github.com/gvanrossum/tulip-try3/blob/7b2d8abfce1d7ef18ef516f9b1b7032172630375/asyncio/base_events.py#L720), so maybe we could also set SO_REUSEPORT only on platforms where it's available. `if hasattr(socket, 'SO_REUSEPORT')` should works.

Or we could add an optional arguments with could be used to set some socket options, it could be more flexible that set SO_REUSEPORT.

I could provide a patch for the best solution selected.
msg241220 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2015-04-16 14:47
I think we left this unfinished. I think it would be best if we added a sock parameter (like for create_server()) but also added reuse_address and reuse_port parameters, treated similarly like the one on create_server(), and added a reuse_port parameter to create_server().

Or maybe there should only be one parameter that guides both; read http://stackoverflow.com/questions/14388706/socket-options-so-reuseaddr-and-so-reuseport-how-do-they-differ-do-they-mean-t before you decide!
msg246691 - (view) Author: chris laws (chris laws) * Date: 2015-07-13 14:10
I encountered this issue too. I needed it resolved ASAP for my work so I created a loop patch that partially implements the suggestion solution by overriding the create_datagram_endpoint method. Perhaps this might be of some use to the eventual ticket resolver.

It can be found in the following gist: https://gist.github.com/claws/d4076b4b155695844d54

If this looks like it's progressing in the right direction I can try to implement it properly in `base_events.py` and submit it via a patch.
msg246768 - (view) Author: chris laws (chris laws) * Date: 2015-07-15 15:55
Attached is a patch that implements the suggested solution along with tests and associated doc updates. Hope this helps.
msg246779 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-07-15 22:01
A quick search told me that "Windows only knows the SO_REUSEADDR option, there is no SO_REUSEPORT". It should be at least documented that SO_REUSEADDR is not supported on Windows.

Maybe we should raise an exception on Windows if reuse_port=True? Ignoring the parameter is a different option, but it should be well documented that the option is ignored if socket.SO_REUSEPORT is not defined.
msg249000 - (view) Author: chris laws (chris laws) * Date: 2015-08-23 05:36
I have updated the patch to address comments raised by haypo. 

An exception is now raised if reuse_port is explicitly used and the platform does not support SOREUSEPORT. The docs have also been updated to make it more explicit that this feature is not supported on Windows.
msg251242 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-09-21 16:48
Hum, the latest patch was not reviewed yet :-(
msg251837 - (view) Author: chris laws (chris laws) * Date: 2015-09-29 09:53
Rebase patch onto current master.
msg251884 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2015-09-29 20:47
I added a whole bunch of review comments. Please send a new patch!
msg252189 - (view) Author: chris laws (chris laws) * Date: 2015-10-03 03:29
Updates addressing review comments.
msg252223 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2015-10-03 16:23
I'm adding a rebased version of Chris's v4 patch to see if I can get the code review to trigger.
msg252338 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-10-05 16:30
New changeset 5e7e9b131904 by Guido van Rossum in branch '3.4':
Issue #23972: updates to asyncio datagram API. By Chris Laws.
https://hg.python.org/cpython/rev/5e7e9b131904

New changeset ba956289fe66 by Guido van Rossum in branch '3.5':
Issue #23972: updates to asyncio datagram API. By Chris Laws. (Merge 3.4->3.5.)
https://hg.python.org/cpython/rev/ba956289fe66

New changeset c0f1f882737c by Guido van Rossum in branch 'default':
Issue #23972: updates to asyncio datagram API. By Chris Laws. (Merge 3.5->3.6.)
https://hg.python.org/cpython/rev/c0f1f882737c
msg252339 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2015-10-05 16:33
Thanks for the patch! It's live now. Closing. Watch the buildbots for me please!
msg252373 - (view) Author: chris laws (chris laws) * Date: 2015-10-06 02:30
I've checked the Buildbot results and observed a few errors related to this change. Specifically the issues affect Windows and Debian.

Builder AMD64 Debian root 3.x build #2772 - http://buildbot.python.org/all/builders/AMD64%20Debian%20root%203.x/builds/2772
Builder AMD64 Windows7 SP1 3.x build #6808 - http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%203.x/builds/6808

In test_create_datagram_endpoint_sock the Windows-based buildbot raises an error related to using sock.getsockname(). This issue has been observed before on Windows using Python.

Refs:
https://mail.python.org/pipermail/python-list/2015-January/683777.html
http://stackoverflow.com/questions/15638214/socket-error-invalid-argument-supplied

In this test the socket is not really used so I was trying to get away with minimal actions to set up the test. The suggested solution would be to bind the socket before using it in the test.

sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.bind(('0.0.0.0', 0))


In test_create_datagram_endpoint_sockopts we can't use the (else clause at line 1310) because if SO_REUSEPORT is not defined it can not be used in a check. This affects Windows and Debian where SO_REUSEPORT is not defined.

From:
if reuseport_supported:
    self.assertTrue(
        sock.getsockopt(
            socket.SOL_SOCKET, socket.SO_REUSEPORT))
else:
    self.assertFalse(
        sock.getsockopt(
            socket.SOL_SOCKET, socket.SO_REUSEPORT))

To:

if reuseport_supported:
    self.assertTrue(
        sock.getsockopt(
            socket.SOL_SOCKET, socket.SO_REUSEPORT))

I'll submit a patch to resolve these issues later today.
msg252379 - (view) Author: chris laws (chris laws) * Date: 2015-10-06 08:00
This patch contains minor updates to resolve the Buildbot issues observed on the Windows and Debian platforms after the initial #23972 change set was committed.
msg252396 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-10-06 15:25
New changeset aebbf205ef6f by Guido van Rossum in branch '3.4':
Issue #23972: Fix tests for Windows and Debian.
https://hg.python.org/cpython/rev/aebbf205ef6f

New changeset 4d643c5df2a5 by Guido van Rossum in branch '3.5':
Issue #23972: Fix tests for Windows and Debian. (Merge 3.4->3.5)
https://hg.python.org/cpython/rev/4d643c5df2a5

New changeset 3e2218a4e629 by Guido van Rossum in branch 'default':
Issue #23972: Fix tests for Windows and Debian. (Merge 3.5->3.6)
https://hg.python.org/cpython/rev/3e2218a4e629
History
Date User Action Args
2015-10-06 15:25:58python-devsetmessages: + msg252396
2015-10-06 08:00:06chris lawssetfiles: + 23972_cjl_v006.patch

messages: + msg252379
2015-10-06 02:31:00chris lawssetmessages: + msg252373
2015-10-05 16:33:28gvanrossumsetstatus: open -> closed
versions: + Python 3.6
messages: + msg252339

assignee: gvanrossum
resolution: fixed
2015-10-05 16:30:49python-devsetnosy: + python-dev
messages: + msg252338
2015-10-03 16:23:08gvanrossumsetfiles: + 23972_cjl_v005.patch

messages: + msg252223
2015-10-03 03:29:29chris lawssetfiles: + 23972_cjl_v004.patch

messages: + msg252189
2015-09-29 20:47:03gvanrossumsetmessages: + msg251884
2015-09-29 09:53:23chris lawssetfiles: + 23972_cjl_v003.patch

messages: + msg251837
2015-09-21 16:48:17vstinnersetnosy: + ysionneau
messages: + msg251242
2015-08-23 05:36:29chris lawssetfiles: + 23972_cjl_v002.patch

messages: + msg249000
2015-08-18 12:11:31j1o1h1nsetnosy: + j1o1h1n
2015-07-15 22:01:25vstinnersetmessages: + msg246779
2015-07-15 15:55:33chris lawssetfiles: + 23972_cjl.patch
keywords: + patch
messages: + msg246768
2015-07-13 14:10:32chris lawssetnosy: + chris laws
messages: + msg246691
2015-04-17 21:45:59terry.reedysetversions: - Python 3.6
2015-04-16 14:47:10gvanrossumsetmessages: + msg241220
2015-04-16 11:52:15Boris.FELDcreate