classification
Title: DatagramProtocol + IPv6 does not work with ProactorEventLoop
Type: behavior Stage: patch review
Components: asyncio, Windows Versions: Python 3.9, Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Marcin Wiśniewsk, Michael.Felt, alex.gronholm, asvetlov, kbr, miss-islington, njs, paul.moore, steve.dower, tim.golden, yselivanov, zach.ware
Priority: normal Keywords: patch

Created on 2019-12-28 10:55 by alex.gronholm, last changed 2020-05-18 10:52 by Michael.Felt.

Files
File name Uploaded Description Edit
udpreceive.py alex.gronholm, 2019-12-28 10:55 Script to reproduce the error
Pull Requests
URL Status Linked Edit
PR 19121 merged kbr, 2020-03-23 18:26
PR 20169 merged miss-islington, 2020-05-18 06:23
PR 20170 merged njs, 2020-05-18 07:15
Messages (20)
msg358940 - (view) Author: Alex Grönholm (alex.gronholm) * Date: 2019-12-28 10:55
Receiving a UDP datagram using DatagramProtocol on the Proactor event loop results in error_received() being called with WinError 87 (Invalid Parameter). The low-level sock_recv() works fine, but naturally loses the sender address information. The attached script works fine as-is on Linux, and on Windows if ::1 is replaced with 127.0.0.1.

There were extensive tests added for UDP support on IOCP, but unfortunately all of them use only IPv4 sockets so they could not catch this problem.
msg364803 - (view) Author: Kjell Braden (kbr) * Date: 2020-03-22 12:52
I tried some debugging with Python 3.9.0a4 and it looks like unparse_address in overlapped.c (https://github.com/python/cpython/blob/v3.9.0a4/Modules/overlapped.c#L689) raises the error. Almost seems like ENABLE_IPV6 is not set...
msg364807 - (view) Author: Kjell Braden (kbr) * Date: 2020-03-22 16:01
Just to confirm, for some reason ENABLE_IPV6 is not set when compiling _overlapped. I'm not familiar enough with the windows build infrastructure to submit a PR though.
msg364837 - (view) Author: Alex Grönholm (alex.gronholm) * Date: 2020-03-23 06:02
Well, I found this: https://github.com/python/cpython/blob/3c97e1e457033bbb8bbe0b7198bd13fc794a12b0/configure.ac#L3278-L3279
Could it be that the official binaries were compiled without --enable-ipv6?
msg364841 - (view) Author: Kjell Braden (kbr) * Date: 2020-03-23 09:23
./configure is not run on Windows, but I believe the define should go here:

https://github.com/python/cpython/blob/8510f430781118d9b603c3a2f06945d6ebc5fe42/PC/pyconfig.h#L678
msg364842 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2020-03-23 09:57
Would somebody be a champion for the issue?
msg364861 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-03-23 16:21
Main thing we need is a PR (straightforward) and confirming that the relevant tests all run. We can use the buildbot fleet to make sure that all supported platforms have the necessary APIs, but it wouldn't surprise me if there's some weird edge cases here.
msg365813 - (view) Author: Kjell Braden (kbr) * Date: 2020-04-05 07:19
PR is up. Any chance we get this reviewed for inclusion in 3.9.0a6 / 3.8.3rc1?
msg366934 - (view) Author: Kjell Braden (kbr) * Date: 2020-04-21 18:08
Please let me know if there is anything else I can do to get this going.
msg367096 - (view) Author: Alex Grönholm (alex.gronholm) * Date: 2020-04-23 11:41
Which PR is it?
msg367098 - (view) Author: Alex Grönholm (alex.gronholm) * Date: 2020-04-23 11:45
Oh, it's https://github.com/python/cpython/pull/19121. I think it would be prudent to add a test as well to make sure this doesn't happen again.
msg367174 - (view) Author: Kjell Braden (kbr) * Date: 2020-04-24 08:08
Fair enough, PR updated.
msg367177 - (view) Author: Alex Grönholm (alex.gronholm) * Date: 2020-04-24 08:16
Thanks, looks good to me now!
msg369107 - (view) Author: Alex Grönholm (alex.gronholm) * Date: 2020-05-17 09:24
The PR is still awaiting for a core developer to be reviewed. It's too bad we missed the 3.8.3 window, but perhaps this can get included in 3.9.0 at least.
msg369185 - (view) Author: miss-islington (miss-islington) Date: 2020-05-18 06:21
New changeset 442634c42fcaf31c636f693951a97734042c3e7b by Kjell Braden in branch 'master':
bpo-39148: enable ipv6 for datagrams in Proactor (GH-19121)
https://github.com/python/cpython/commit/442634c42fcaf31c636f693951a97734042c3e7b
msg369186 - (view) Author: miss-islington (miss-islington) Date: 2020-05-18 06:42
New changeset 94d9c5e5afdee4f46be9d9faaa39d6be40c34849 by Miss Islington (bot) in branch '3.8':
bpo-39148: enable ipv6 for datagrams in Proactor (GH-19121)
https://github.com/python/cpython/commit/94d9c5e5afdee4f46be9d9faaa39d6be40c34849
msg369187 - (view) Author: Michael Felt (Michael.Felt) * Date: 2020-05-18 06:42
Bot failed for AIX     https://buildbot.python.org/all/#builders/227/builds/978 with:

0:07:11 Re-running test_asyncio in verbose mode
Failed to import test module: test.test_asyncio.test_events
Traceback (most recent call last):
  File "/home/shager/cpython-buildarea/3.x.edelsohn-aix-ppc64/build/Lib/unittest/loader.py", line 436, in _find_test_path
    module = self._get_module_from_name(name)
  File "/home/shager/cpython-buildarea/3.x.edelsohn-aix-ppc64/build/Lib/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/home/shager/cpython-buildarea/3.x.edelsohn-aix-ppc64/build/Lib/test/test_asyncio/test_events.py", line 239, in <module>
    class EventLoopTestsMixin:
  File "/home/shager/cpython-buildarea/3.x.edelsohn-aix-ppc64/build/Lib/test/test_asyncio/test_events.py", line 1262, in EventLoopTestsMixin
    @unittest.skipUnless(support.IPV6_ENABLED, 'IPv6 not supported or enabled')
AttributeError: module 'test.support' has no attribute 'IPV6_ENABLED'
test test_asyncio crashed -- Traceback (most recent call last):
  File "/home/shager/cpython-buildarea/3.x.edelsohn-aix-ppc64/build/Lib/test/libregrtest/runtest.py", line 270, in _runtest_inner
    refleak = _runtest_inner2(ns, test_name)
  File "/home/shager/cpython-buildarea/3.x.edelsohn-aix-ppc64/build/Lib/test/libregrtest/runtest.py", line 234, in _runtest_inner2
    test_runner()
  File "/home/shager/cpython-buildarea/3.x.edelsohn-aix-ppc64/build/Lib/test/libregrtest/runtest.py", line 208, in _test_module
    raise Exception("errors while loading tests")
Exception: errors while loading tests
1 test failed again:
    test_asyncio
msg369192 - (view) Author: Nathaniel Smith (njs) * (Python committer) Date: 2020-05-18 07:56
New changeset 58205a0217e91232cc1e945dbfe4e387d636eb76 by Nathaniel J. Smith in branch 'master':
bpo-39148: fixup to account for IPV6_ENABLED being moved (GH-20170)
https://github.com/python/cpython/commit/58205a0217e91232cc1e945dbfe4e387d636eb76
msg369193 - (view) Author: Nathaniel Smith (njs) * (Python committer) Date: 2020-05-18 08:07
I think I fixed the buildbot issues in GH-20170, but I can't seem to reach the buildbot site right now, so it's hard to know for sure!
msg369205 - (view) Author: Michael Felt (Michael.Felt) * Date: 2020-05-18 10:52
I could not "fathom" the buildbot test results - however, a manual build of PR29170 on 3.9 works: I'll try 3.8, but then from master (assuming it is already part of master) -- and that works as well.

Thanks for the quick update!
History
Date User Action Args
2020-05-18 10:52:49Michael.Feltsetmessages: + msg369205
2020-05-18 08:07:53njssetmessages: + msg369193
versions: - Python 3.6
2020-05-18 07:56:53njssetmessages: + msg369192
2020-05-18 07:15:43njssetpull_requests: + pull_request19469
2020-05-18 06:42:34Michael.Feltsetnosy: + Michael.Felt
messages: + msg369187
2020-05-18 06:42:32miss-islingtonsetmessages: + msg369186
2020-05-18 06:23:01miss-islingtonsetpull_requests: + pull_request19468
2020-05-18 06:21:41miss-islingtonsetnosy: + miss-islington
messages: + msg369185
2020-05-17 09:24:36alex.gronholmsetmessages: + msg369107
2020-04-24 08:16:38alex.gronholmsetmessages: + msg367177
2020-04-24 08:08:10kbrsetmessages: + msg367174
2020-04-23 17:17:05vstinnersetfiles: - README.md20629842+superfon@users.noreply.github.com
2020-04-23 17:16:36vstinnersetmessages: - msg367131
2020-04-23 17:09:36Marcin Wiśniewsksetfiles: + README.md20629842+superfon@users.noreply.github.com
versions: + Python 3.6
nosy: + Marcin Wiśniewsk

messages: + msg367131
2020-04-23 11:45:21alex.gronholmsetmessages: + msg367098
2020-04-23 11:41:29alex.gronholmsetmessages: + msg367096
2020-04-21 18:08:00kbrsetmessages: + msg366934
2020-04-05 07:19:28kbrsetmessages: + msg365813
2020-03-23 18:26:20kbrsetkeywords: + patch
stage: patch review
pull_requests: + pull_request18482
2020-03-23 16:21:48steve.dowersetmessages: + msg364861
2020-03-23 09:57:37asvetlovsetmessages: + msg364842
2020-03-23 09:23:54kbrsetmessages: + msg364841
2020-03-23 06:02:28alex.gronholmsetmessages: + msg364837
2020-03-22 16:01:11kbrsetmessages: + msg364807
versions: + Python 3.9
2020-03-22 12:52:40kbrsetmessages: + msg364803
2020-03-22 10:21:39kbrsetfiles: - udp_ipv6_server.py
2020-03-22 10:21:29kbrsetfiles: + udp_ipv6_server.py
nosy: + kbr
2019-12-28 22:23:10njssetnosy: + njs
2019-12-28 10:57:07alex.gronholmsetnosy: + yselivanov
components: + asyncio
2019-12-28 10:55:11alex.gronholmcreate