classification
Title: socket module missing IPPROTO_IPV6, IPPROTO_IPV4 on Windows
Type: behavior Stage: patch review
Components: Windows Versions: Python 3.8
process
Status: pending Resolution:
Dependencies: 17561 Superseder:
Assigned To: Nosy List: abrezovsky, christian.heimes, eryksun, giampaolo.rodola, jgosmann, mhils, paul.moore, steve.dower, tim.golden, zach.ware
Priority: normal Keywords: easy (C), patch

Created on 2017-02-09 18:27 by mhils, last changed 2019-03-28 14:26 by giampaolo.rodola.

Pull Requests
URL Status Linked Edit
PR 12183 merged giampaolo.rodola, 2019-03-05 17:12
Messages (18)
msg287449 - (view) Author: Maximilian Hils (mhils) * Date: 2017-02-09 18:27
The latest Windows builds for Python 3.5.3/3.6.0 do not export socket.IPPROTO_IPV6, even though e.g. socket.IPV6_V6ONLY is exported. This seems to be wrong to me as IPV6_V6ONLY requires the corresponding socket option level IPPROTO_IPV6 to be actually useful. The same issue at least also applies to IPPROTO_IPV4.

christian.heimes mentioned that this is intended behaviour in https://bugs.python.org/issue6926 as Windows would not define the constants. Now I am all but an expert on this, but it seems to me that IPPROTO_IPV6 is defined in Windows:
https://msdn.microsoft.com/en-us/library/windows/hardware/ff543746(v=vs.85).aspx. Apologies if I'm missing something here.

Python 3.6.0 (v3.6.0:41df79263a11, Dec 23 2016, 07:18:10) [MSC v.1900 32 bit (Intel)] on win32
>>> import socket; socket.IPPROTO_IPV6
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'socket' has no attribute 'IPPROTO_IPV6'

As a workaround, IPPROTO_IPV6 can be substituted with the hardcoded constant 41.
msg287472 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2017-02-10 02:01
The macro is defined but not defined. If I insert the following line before the #ifdef check:

    #define IPPROTO_IPV6 IPPROTO_IPV6

then the constant gets added:

    >>> import _socket
    >>> _socket.IPPROTO_IPV6
    41

The same applies to IPPROTO_IPV4 and probably others.
msg287477 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2017-02-10 07:25
Unless someone has a better (more automated) way to handle the Winsock IPPROTO enum, I suggest we either special-case the individual tests for MS_WINDOWS when we know that Winsock defines the value; or just define macros for the values in the enum:

    #ifdef MS_WINDOWS /* Macros based on the IPPROTO enum. */
    #define IPPROTO_ICMP IPPROTO_ICMP
    #define IPPROTO_IGMP IPPROTO_IGMP
    #define IPPROTO_GGP IPPROTO_GGP
    #define IPPROTO_TCP IPPROTO_TCP
    #define IPPROTO_PUP IPPROTO_PUP
    #define IPPROTO_UDP IPPROTO_UDP
    #define IPPROTO_IDP IPPROTO_IDP
    #define IPPROTO_ND IPPROTO_ND 
    #define IPPROTO_RAW IPPROTO_RAW
    #define IPPROTO_MAX IPPROTO_MAX
    #if (_WIN32_WINNT >= 0x0501)
    #define IPPROTO_HOPOPTS IPPROTO_HOPOPTS
    #define IPPROTO_IPV4 IPPROTO_IPV4
    #define IPPROTO_IPV6 IPPROTO_IPV6
    #define IPPROTO_ROUTING IPPROTO_ROUTING
    #define IPPROTO_FRAGMENT IPPROTO_FRAGMENT
    #define IPPROTO_ESP IPPROTO_ESP
    #define IPPROTO_AH IPPROTO_AH
    #define IPPROTO_ICMPV6 IPPROTO_ICMPV6
    #define IPPROTO_NONE IPPROTO_NONE
    #define IPPROTO_DSTOPTS IPPROTO_DSTOPTS
    #define IPPROTO_ICLFXBM IPPROTO_ICLFXBM
    #endif /* (_WIN32_WINNT >= 0x0501) */
    #if (_WIN32_WINNT >= 0x0600)
    #define IPPROTO_ST IPPROTO_ST
    #define IPPROTO_CBT IPPROTO_CBT
    #define IPPROTO_EGP IPPROTO_EGP
    #define IPPROTO_IGP IPPROTO_IGP
    #define IPPROTO_RDP IPPROTO_RDP
    #define IPPROTO_PIM IPPROTO_PIM
    #define IPPROTO_PGM IPPROTO_PGM
    #define IPPROTO_L2TP IPPROTO_L2TP
    #define IPPROTO_SCTP IPPROTO_SCTP
    #endif /* (_WIN32_WINNT >= 0x0600) */
    #endif /* MS_WINDOWS */

or call PyModule_AddIntConstant(m, "IPPROTO_ICMP", IPPROTO_ICMP) and so on for each enum value.
msg287541 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2017-02-10 14:43
We may just want to copy the values from the enum if there are different versions when they were introduced. But if that's not a problem, adding the MS_WINDOWS check is better than defining new macros.
msg333133 - (view) Author: Andrew Brezovsky (abrezovsky) * Date: 2019-01-07 03:51
Can confirm this is still a problem in latest versions of 3.5, 3.6, and 3.7

Any progress made on this?
msg333727 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-01-15 18:37
No progress, but I like the extra defines idea best (directly in socketmodule.c, not in a public header file). That's the easiest way to close the gap between (apparently) real constants used on Windows and the preprocessor defines (apparently) used elsewhere.
msg335924 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2019-02-19 10:44
It turns out having this fix would be useful for proceeding with issue17561, which right now cannot support dual-stack IPv4/6 on Windows because IPPROTO_IPV6 is missing, see my comment:
https://github.com/python/cpython/pull/11784#issuecomment-465078646
msg335929 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2019-02-19 11:44
@eryksun I tried your patch (https://bugs.python.org/issue29515#msg287477) on Windows 10 and it looks good. It introduces the following new constants:

IPPROTO_IGMP
IPPROTO_GGP
IPPROTO_PUP
IPPROTO_IDP
IPPROTO_ND
IPPROTO_MAX
IPPROTO_HOPOPTS
IPPROTO_IPV4
IPPROTO_IPV6
IPPROTO_ROUTING
IPPROTO_FRAGMENT
IPPROTO_ESP
IPPROTO_AH
IPPROTO_ICMPV6
IPPROTO_NONE
IPPROTO_DSTOPTS
IPPROTO_EGP
IPPROTO_PIM
IPPROTO_SCTP

...plus these Windows-only ones for which I had to add an additional PyModule_AddIntMacro() call:

IPPROTO_ICLFXBM
IPPROTO_ST
IPPROTO_CBT
IPPROTO_IGP
IPPROTO_RDP
IPPROTO_PGM
IPPROTO_L2TP

Up 'till now only these ones were exposed:

IPPROTO_ICMP
IPPROTO_TCP
IPPROTO_UDP
IPPROTO_RAW

I think we can proceed with this. Care to provide a patch?
msg337194 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2019-03-05 14:38
@Eryk do you mind if I make a PR with your code(I will of course author you)? Regardless from issue17561 I think this is an important fix because without IPPROTO_IPV6 is not possible to implement a dual-stack IPv4/6 socket on Windows.
msg337229 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2019-03-05 16:51
> do you mind if I make a PR with your code(I will of course author you)?

Go for it. I prefer no credit, but you're free to do as you wish.
msg337240 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2019-03-05 21:56
Done:
https://github.com/python/cpython/pull/12183/
This adds quite a bunch of new constants so I would be keen on considering it an enhancement rather than a bugfix and land it in 3.8 only. As for 3.7 and 3.6 (and 2.7?) it may make sense to fix IPPROTO_IPV6 only. Thoughts?
msg337242 - (view) Author: Andrew Brezovsky (abrezovsky) * Date: 2019-03-05 22:32
I lean towards it being considered a bug fix. First, the lack of parity between Windows and Linux-based OSs causes confusion. Second, the current workaround to hard-code the value in is far from best practice.
msg337251 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-03-06 00:49
The problem is that if we add lots of new constants in 3.7.3, code written in that version very easily becomes incompatible with 3.7.2 and earlier. So we don't like making changes like that.

If we know which ones are "normally" defined on Linux machines, where specifying them on Windows will make it just work, then I'd just add those ones.

Anything that is Windows-specific (if any?) or that may cause applications to break if they're testing for the constant but it doesn't work smoothly should not go back to 3.7.
msg337269 - (view) Author: Andrew Brezovsky (abrezovsky) * Date: 2019-03-06 02:19
I see your point.

Then perhaps only a subset of the more vital ones get added to previous releases?

#define IPPROTO_IPV4 IPPROTO_IPV4
#define IPPROTO_IPV6 IPPROTO_IPV6
#define IPPROTO_TCP IPPROTO_TCP
#define IPPROTO_UDP IPPROTO_UDP
#define IPPROTO_ICMP IPPROTO_ICMP
#define IPPROTO_ICMPV6 IPPROTO_ICMPV6
#define IPPROTO_RAW IPPROTO_RAW
msg337273 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2019-03-06 03:17
IPPROTO_TCP/UDP/RAW/ICMP are already there. AFAICT the only useful one here is IPPROTO_IPV6, because it's needed for enabling dual-stack. I don't think IPPROTO_IPV4 is a must (e.g. it's not available on Linux). Same for IPPROTO_ICMPV6. The only reason we may decide to backport one of these is because they are not officially documented (see issue27409). That gives us some flexibility but potentially it may do more harm than good because an app running on 3.7.3 would break on 3.7.2. Note: the original ticket (issue6926) is from 2009, meaning that whoever wanted these missing constants on Windows already hard-coded them in their Python code long ago.
msg338230 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2019-03-18 14:34
I'd like to move forward with this in order to fix issue17561. If there are no complaints I'm gonna merge this in ~ a week.
msg339047 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2019-03-28 14:20
New changeset 3eca28c61363a03b81b9fb12775490d6e42d8ecf by Giampaolo Rodola in branch 'master':
bpo-29515: add missing socket.IPPROTO_* constants on Windows (GH-12183)
https://github.com/python/cpython/commit/3eca28c61363a03b81b9fb12775490d6e42d8ecf
msg339048 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2019-03-28 14:26
Merged. I'm agnostic about backporting IPPROTO_IPV6 (or others) on < 3.8 so I'll leave it up to somebody else to decide/implement.
History
Date User Action Args
2019-03-28 14:26:10giampaolo.rodolasetstatus: open -> pending

messages: + msg339048
versions: - Python 3.7
2019-03-28 14:20:47giampaolo.rodolasetmessages: + msg339047
2019-03-18 14:34:10giampaolo.rodolasetmessages: + msg338230
2019-03-06 03:17:59giampaolo.rodolasetmessages: + msg337273
2019-03-06 02:19:11abrezovskysetmessages: + msg337269
2019-03-06 00:49:49steve.dowersetmessages: + msg337251
2019-03-05 22:32:53abrezovskysetmessages: + msg337242
2019-03-05 21:56:45giampaolo.rodolasetmessages: + msg337240
2019-03-05 17:12:54giampaolo.rodolasetkeywords: + patch
stage: patch review
pull_requests: + pull_request12179
2019-03-05 16:51:56eryksunsetmessages: + msg337229
2019-03-05 14:38:21giampaolo.rodolasetmessages: + msg337194
2019-02-19 18:34:16gregory.p.smithsetversions: + Python 3.8, - Python 3.5, Python 3.6
2019-02-19 11:44:09giampaolo.rodolasetmessages: + msg335929
2019-02-19 10:44:58giampaolo.rodolasetnosy: + giampaolo.rodola
dependencies: + Add socket.bind_socket() convenience function
messages: + msg335924
2019-01-15 18:37:36steve.dowersetkeywords: + easy (C)

messages: + msg333727
2019-01-07 03:51:13abrezovskysetnosy: + abrezovsky
messages: + msg333133
2017-10-30 17:24:15jgosmannsetnosy: + jgosmann
2017-02-10 14:43:15steve.dowersetmessages: + msg287541
2017-02-10 07:25:57eryksunsetmessages: + msg287477
2017-02-10 02:01:49eryksunsetnosy: + eryksun

messages: + msg287472
versions: + Python 3.5, Python 3.7
2017-02-09 18:27:42mhilscreate