classification
Title: setting SO_REUSEPORT fails
Type: behavior Stage: needs patch
Components: asyncio Versions: Python 3.7, Python 3.6
process
Status: open Resolution:
Dependencies: 27027 Superseder:
Assigned To: Nosy List: Alex.Willmer, Chi Hsuan Yen, Roman.Evstifeev, giampaolo.rodola, gvanrossum, haypo, pjenvey, xdegaye, yselivanov
Priority: normal Keywords: patch

Created on 2016-04-26 13:37 by xdegaye, last changed 2017-09-17 00:15 by gvanrossum.

Files
File name Uploaded Description Edit
test.asyncio.patch xdegaye, 2016-04-26 13:37 review
test.asyncio_2.patch xdegaye, 2016-05-22 08:24 review
Messages (17)
msg264281 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2016-04-26 13:37
Android defines SO_REUSEPORT on android API 21 but setting this option in the asyncio tests raises OSError: [Errno 92] Protocol not available.

The attached patch assumes there is a platform.android_ver() function to detect that this is the android platform.  The patch does not take into account the fact that this may be fixed in future versions of android.
msg264602 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2016-05-01 17:36
Where is the patch that adds android_ver()? At the very least that should be listed as a dependency here.

I don't very much like to have changes to asyncio that can only work in Python 3.6; there's still an upstream version (https://github.com/python/asyncio) that is used to do PyPI releases of asyncio for Python 3.3. I try to keep that in sync with the stdlib on all platforms, otherwise figuring out whether any changes need to be packported is a nightmare.

You might also update the comment somewhere that currently states that reuse_port is not supported on Windows.

PS. It amuses me that each platform apparently has a different way to check (there are three different types of platform checks in the lines you change).
msg264605 - (view) Author: Chi Hsuan Yen (Chi Hsuan Yen) * Date: 2016-05-01 18:18
> Where is the patch that adds android_ver()?

http://bugs.python.org/issue26855

Xavier de Gaye's implementation is buggy while works for most cases.
msg264635 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2016-05-02 09:27
The patch was not intended to be proposed as a way to fix this problem, but as an indication of what the problem is. This issue is a dependency of issue #26865: Meta-issue: support of the android platform.

I should have posted instead the result of running 'python -m test -v test_asyncio' on an android emulator running an x86 system image at API level 21, which is:

======================================================================
ERROR: test_create_datagram_endpoint_sockopts (test.test_asyncio.test_base_events.BaseEventLoopWithS
electorTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/test_asyncio/test_base_events.py", line 1506,
 in test_create_datagram_endpoint_sockopts
    socket.SOL_SOCKET, socket.SO_REUSEPORT))
OSError: [Errno 92] Protocol not available

======================================================================
ERROR: test_create_server_reuse_port (test.test_asyncio.test_events.EPollEventLoopTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/test_asyncio/test_events.py", line 830, in te
st_create_server_reuse_port
    socket.SOL_SOCKET, socket.SO_REUSEPORT))
OSError: [Errno 92] Protocol not available

======================================================================
ERROR: test_create_server_reuse_port (test.test_asyncio.test_events.PollEventLoopTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/test_asyncio/test_events.py", line 830, in te
st_create_server_reuse_port
    socket.SOL_SOCKET, socket.SO_REUSEPORT))
OSError: [Errno 92] Protocol not available

======================================================================
ERROR: test_create_server_reuse_port (test.test_asyncio.test_events.SelectEventLoopTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/test_asyncio/test_events.py", line 830, in te
st_create_server_reuse_port
    socket.SOL_SOCKET, socket.SO_REUSEPORT))
OSError: [Errno 92] Protocol not available

----------------------------------------------------------------------
Ran 991 tests in 16.528s

FAILED (errors=4, skipped=2)
test test_asyncio failed
1 test failed:
    test_asyncio
Total duration: 0:00:18
msg264653 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2016-05-02 16:29
OK, I'll wait until Android support is closer. Do you have a core dev who's mentoring/reviewing here?
msg264662 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2016-05-02 19:17
Stefan has already commited and closed some of the issues listed at issue #26865. Martin has closed issue #22359, a blocker for the cross-compilation of python.

I will enter an issue for all the failures listed at https://bitbucket.org/xdegaye/pyona/wiki/testsuite and add them to the list in the android meta-issue.
msg264666 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2016-05-02 20:27
If there are a lot of Android issue maybe you could get a tracker manager
to add a new keyword or category or whatever?
msg264690 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2016-05-03 07:10
Running the "Search" button on the tracker with "Components" set as "Cross-Build" lists 42 issues (20 in the last year).

I guess using this existing category is fine if the titles' issues are prefixed with "android:".
msg265928 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2016-05-20 10:11
Should use boolean 'is_android' being added to module test.support by issue #27027, instead of platform.android_ver()[0].
msg266061 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2016-05-22 08:24
New patch test.asyncio_2.patch uses test.support.is_android and adds a dependency to issue #27027.
msg276615 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2016-09-15 20:46
This issue was fixed in #28174.
msg285392 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2017-01-13 12:39
The changes made in issue #28174 fix the problem for the Android x86 platform and for the armv7 platform at Android API level 24, but the problem is still there on the armv7 platform at Android API level 21.

======================================================================
ERROR: test_create_datagram_endpoint_sockopts (test.test_asyncio.test_base_events.BaseEventLoopWithS
electorTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/sdcard/org.bitbucket.pyona/lib/python3.7/test/test_asyncio/test_base_events.py", line 1593,
 in test_create_datagram_endpoint_sockopts
    socket.SOL_SOCKET, socket.SO_REUSEPORT))
OSError: [Errno 92] Protocol not available

======================================================================
ERROR: test_create_server_reuse_port (test.test_asyncio.test_events.EPollEventLoopTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/sdcard/org.bitbucket.pyona/lib/python3.7/test/test_asyncio/test_events.py", line 913, in te
st_create_server_reuse_port
    socket.SOL_SOCKET, socket.SO_REUSEPORT))
OSError: [Errno 92] Protocol not available

======================================================================
ERROR: test_create_server_reuse_port (test.test_asyncio.test_events.PollEventLoopTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/sdcard/org.bitbucket.pyona/lib/python3.7/test/test_asyncio/test_events.py", line 913, in te
st_create_server_reuse_port
    socket.SOL_SOCKET, socket.SO_REUSEPORT))
OSError: [Errno 92] Protocol not available

======================================================================
ERROR: test_create_server_reuse_port (test.test_asyncio.test_events.SelectEventLoopTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/sdcard/org.bitbucket.pyona/lib/python3.7/test/test_asyncio/test_events.py", line 913, in te
st_create_server_reuse_port
    socket.SOL_SOCKET, socket.SO_REUSEPORT))
OSError: [Errno 92] Protocol not available

----------------------------------------------------------------------
Ran 1476 tests in 382.973s

FAILED (errors=4, skipped=2)
test test_asyncio failed
test_asyncio failed in 7 min 25 sec

1 test failed:
    test_asyncio

Total duration: 7 min 27 sec
Tests result: FAILURE
msg285394 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-01-13 12:51
IMHO it's ok to skip a few asyncio tests on Android. It's safe if it doesn't touch the code of asyncio itself.

It don't really care if the Android version is not checked yet. Moreover, if we really want to check the version, we should check the *runtime* verison, not the *build* version of Android. We need the discussed platform.android_ver() function which doesn't exist yet. So it's fine to postpone the version check in asyncio tests.
msg285402 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2017-01-13 15:03
Skipping those tests on Android has already been proposed with the existing patches in this issue.
The reason why the changes made by issue 28174 fix only partly the problem should be understood.
msg285406 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-01-13 15:20
> The reason why the changes made by issue 28174 fix only partly the problem should be understood.

The change c1c247cf3488 catchs OSError on sock.setsockopt() in asyncio/base_events.py, whereas the test fails while calling sock.getsockopt() in test_asyncio/tets_events.py. Extract of the test:

    def test_create_server_reuse_port(self):
        proto = MyProto(self.loop)
        f = self.loop.create_server(
            lambda: proto, '0.0.0.0', 0)
        server = self.loop.run_until_complete(f)
        self.assertEqual(len(server.sockets), 1)
        sock = server.sockets[0]
        self.assertFalse(
            sock.getsockopt(
                socket.SOL_SOCKET, socket.SO_REUSEPORT))
        server.close()

The change c1c247cf3488 doesn't touch the code of this unit test.

test_base_events.py also calls directly sock.getsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT).
msg302351 - (view) Author: Philip Jenvey (pjenvey) * (Python committer) Date: 2017-09-16 19:15
Note that this isn't an android specific issue: See issue19901, and 9791c5d55f52 for an appropriate solution patch.

(Renaming title)
msg302357 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2017-09-17 00:15
Can't you just submit a PR? It's so easy now!
History
Date User Action Args
2017-09-17 00:15:36gvanrossumsetmessages: + msg302357
2017-09-16 19:15:09pjenveysetnosy: + pjenvey

messages: + msg302351
title: android: setting SO_REUSEPORT fails -> setting SO_REUSEPORT fails
2017-01-13 15:20:55hayposetmessages: + msg285406
2017-01-13 15:03:02xdegayesetmessages: + msg285402
2017-01-13 12:51:04hayposetmessages: + msg285394
2017-01-13 12:39:06xdegayesetstatus: closed -> open
versions: + Python 3.7
messages: + msg285392

components: + asyncio, - Cross-Build
resolution: fixed ->
stage: resolved -> needs patch
2016-09-15 20:46:29yselivanovsetstatus: open -> closed
resolution: fixed
messages: + msg276615

stage: resolved
2016-05-22 08:24:57xdegayesetfiles: + test.asyncio_2.patch

dependencies: + add the 'is_android' attribute to test.support
messages: + msg266061
2016-05-20 10:11:57xdegayesetmessages: + msg265928
2016-05-03 07:11:36xdegayesettitle: setting SO_REUSEPORT fails on android -> android: setting SO_REUSEPORT fails
2016-05-03 07:10:54xdegayesetmessages: + msg264690
2016-05-02 20:34:34pitrousetnosy: - pitrou
2016-05-02 20:27:12gvanrossumsetmessages: + msg264666
2016-05-02 19:17:17xdegayesetmessages: + msg264662
2016-05-02 16:29:32gvanrossumsetmessages: + msg264653
2016-05-02 09:27:06xdegayesetmessages: + msg264635
2016-05-01 18:18:35Chi Hsuan Yensetnosy: + Chi Hsuan Yen
messages: + msg264605
2016-05-01 17:36:54gvanrossumsetmessages: + msg264602
2016-05-01 07:15:08xdegayesetnosy: + gvanrossum, pitrou, haypo, giampaolo.rodola, yselivanov
2016-04-26 16:04:41zach.warelinkissue26865 dependencies
2016-04-26 16:00:35Roman.Evstifeevsetnosy: + Roman.Evstifeev
2016-04-26 13:37:22xdegayecreate