On Windows, python 3.6.3 code "hasattr(socket, 'TCP_KEEPCNT')" gives False, python 3.6.4 gives True. This behavior breaks many libraries that i use.

dtdev@dtdev-centos $ python3
Python 3.6.3 (default, Oct 11 2017, 18:17:01) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-18)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> hasattr(socket, 'TCP_KEEPCNT')
True
>>> 

I have linux system. Above is the output i got.

Which libraries break? And which version of Windows are you running on?

If TCP_KEEPCNT was only added recently (perhaps in an SDK update) then it may not really be available on all versions.

(+Ned for awareness of the regression - not sure we need to respin immediately, but let's figure it out asap)

It's a compile-time option in socketmodule.c.

https://github.com/python/cpython/blob/3.6/Modules/socketmodule.c#L7466

The MSDN page suggests that it was added for Win10:

https://msdn.microsoft.com/en-us/library/windows/desktop/ms738596(v=vs.85).aspx

Is it possible that the build machine has changed its OS/SDK between
building 3.6.3 and 3.6.4?

Yeah, I updated the build machine before doing 3.6.4rc1 (I never update between rc and final releases). I'm intrigued to see why it breaks libraries though - typically unsupported enum values like this are silently ignored on older Windows versions.

My OS version is Windows 7 x64.
I ran the script on the same computer, but with different versions of the python:

import socket
import platform

print('1) OS Info: ', platform.architecture(), platform.platform())
print('2) Python Info: ', platform.python_build(), platform.python_compiler())
print('3) TCP_KEEPCNT = ', hasattr(socket, 'TCP_KEEPCNT') )

Result for python 3.6.3:

1. OS Info: ('64bit', 'WindowsPE') Windows-7-6.1.7601-SP1
2. Python Info: ('v3.6.3:2c5fed8', 'Oct 3 2017 18:11:49') MSC v.1900 64 bit (AMD64)
3. TCP_KEEPCNT = False

Result for python 3.6.4:

1. OS Info: ('64bit', 'WindowsPE') Windows-7-6.1.7601-SP1
2. Python Info: ('v3.6.4:d48eceb', 'Dec 19 2017 06:54:40') MSC v.1900 64 bit (AMD64)
3. TCP_KEEPCNT = True

Yes, I recognise that the change happened. I don't understand what breaks as a result.

You said it breaks "many libraries" - can you name some of them and provide sample code?

websocket-client 0.44.0
https://pypi.python.org/pypi/websocket-client/0.44.0

My script gives the following Erroe:

File "C:\Program Files\Python36\lib\site-packages\websocket\_http.py", line 108, in _open_socket
sock.setsockopt(*opts)
OSError: [WinError 10042] Для вызова getsockopt или setsockopt был указан неизвестный, недопустимый или неподдерживаемый параметр
или уровень

Okay, so it looks like we don't have any prior art to conditionally omit constants from _socket based on OS level, and nothing in the setsockopt() doc suggests that it may fail like this.

So we either need to explicitly exclude this symbol on Windows (at least until we drop support for pre-Windows 10 versions) or silently ignore setsockopt errors for future good arguments. I'm inclined to do the former - other opinions?

Excluding for now seems like a simple option. (Basically a reversion to
previous behaviour). And allows us easily to include again later easily.

Messing with setsockopt seems a little more risky.

In short: I'm with you -- exclude for now.

I suggest inserting the following code into socket.py:

if hasattr(sys, 'getwindowsversion') and sys.getwindowsversion()[0] < 10:
    del socket.TCP_KEEPCNT
    del socket.TCP_FASTOPEN

Steve, is there something to be done here for 3.7.0?

I like Kamil's suggestion.

Base on Kamil's code, a little improvements.

if hasattr(sys, 'getwindowsversion'):
    WIN_MAJOR, _, WIN_BUILD, *_ = sys.getwindowsversion()
    if WIN_MAJOR == 10:
        if WIN_BUILD >= 15063:    # Windows 10 1703
            pass
        elif WIN_BUILD >= 14393:  # Windows 10 1607
            del socket.TCP_KEEPCNT
        else:
            del socket.TCP_KEEPCNT
            del socket.TCP_FASTOPEN
    elif WIN_MAJOR < 10:
        del socket.TCP_KEEPCNT
        del socket.TCP_FASTOPEN

Windows 10 versions:
https://technet.microsoft.com/en-us/windows/release-info.aspx

Nice. Think you can turn that into a pull request on GitHub?

So we either need to explicitly exclude this symbol on Windows (at least until we drop support for pre-Windows 10 versions) or silently ignore setsockopt errors for future good arguments. I'm inclined to do the former - other opinions?
--------------
I'm not a socket expert, I have a question about the former one.

Imagine someone changed those values in his/her code, for example:
socket.TCP_KEEPCNT = 20

His/her code works fine on new version Windows. Will this line break things again on old version Windows?

Yes, adding the member back will put you back in the broken state, but there's nothing we can do to stop it.

Thanks for the PR - I'll take a look, and if I'm able to log into GitHub on my phone maybe finish it.

PR looks good to me. Unfortunately without SMS (I'm travelling) I can't get into my account from my phone, so if someone else wants to add the backport tags (3.7 and 3.6) and finish it then feel free.

Glad to see PR 5468 not merged, I found it makes socket.py complicated.

Now I'm inclined to patch the code in PyInit__socket(void) of socketmodule.c
https://github.com/python/cpython/blob/3.6/Modules/socketmodule.c#L6504

There already has a MS-Windows version checking
https://github.com/python/cpython/blob/3.6/Modules/socketmodule.c#L6511

In there, we can use GetVersionEx instead of GetVersion to get BuildNumber, then delete unusable opinions.

(I don't have enough skill to modify .c code)

I suggest inserting the following code into socketmodule.c:

CHANGE:
#ifdef  TCP_KEEPCNT    
        PyModule_AddIntMacro(m, TCP_KEEPCNT);
#endif

TO:
#ifdef  TCP_KEEPCNT    
    #if defined(_MSC_VER) && _MSC_VER >= 1800
    // Windows 10 1703 (15063)
    if (IsWindows10CreatorsOrGreater()) {
        PyModule_AddIntMacro(m, TCP_KEEPCNT);        
    }
    #endif
#endif

AND CHANGE:
#ifdef  TCP_FASTOPEN
        PyModule_AddIntMacro(m, TCP_FASTOPEN);
#endif

TO:
#ifdef  TCP_FASTOPEN
    #if defined(_MSC_VER) && _MSC_VER >= 1800
    // Windows 10 1703 (Build: 14393)
    if (IsWindows10AnniversaryOrGreater()) {
        PyModule_AddIntMacro(m, TCP_FASTOPEN);
    }
    #endif 
#endif

I create a new one (PR 5523), I'm not a C & socket expert, so if you want to improve/polish the patch, feel free to create a new PR based on (or not) it.

We don't remove unsupported socket flags on Unix, why should we do it for Windows?

Socket constants a compile time values, obviously concrete operation system might not support a flag -- but we do nothing with it in runtime.

All flags available on compile time are exposed, it's true for modules like socket, os, select etc.

The other option would be to always hide the new constant on Windows in 3.6, and make it unconditionally available on 3.7.

We don't remove unsupported socket flags on Unix, why should we do it for Windows?

We have this problem because: compile with new Windows SDK, but run on old version Windows.
On Linux/Unix, the compile-time headers always consist with the system, so there should not has this problem.
Correct me if I'm wrong.

The other option would be to always hide the new constant on Windows in 3.6, and make it unconditionally available on 3.7.

Search on GitHub [1], most people only check whether socket has such flags, like this:
if hasattr(socket, "TCP_KEEPCNT"):
...

Most of they don't check platform or Python version, so I'm -1 on this option.

-----------------
TCP_KEEPIDLE and TCP_KEEPINTVL were added in Windows 10 1709. [2]
The master branch on AppVeyor is using 10.0.16229 (1709) SDK. [3]
While 3.6 branch is using 10.0.15062 (1703) SDK. [4]
If you agree the way of PR 5523, maybe we should remove these two flags as well.

[1] https://github.com/search?l=Python&p=1&q=TCP_KEEPCNT&type=Code&utf8=%E2%9C%93
[2] https://msdn.microsoft.com/en-us/library/windows/desktop/ms738596.aspx
[3] https://github.com/isuruf/cpython/blob/master/PCbuild/python.props#L78
[4] https://github.com/isuruf/cpython/blob/3.6/PCbuild/python.props#L77

On Linux/Unix, the compile-time headers always consist with the system, so there should not has this problem.
Correct me if I'm wrong.

No. Compile-time and run-time system is not always consist.
Kernel version may be upgraded / downgraded after Python is built.
And header version may not be consistent with kernel version.

There are some cases that system may return error for unsupported setsockopt() on Linux.

So I think websocket-client should catch OSError for setsockopt.

But if there are many broken libraries in the world, it's considerable
that hide it on Python side.

Kamil said "This behavior breaks many libraries that i use."
But I saw only websocket-client. How many other libraries?

It seems Linux has TCP_KEEPCNT from very old ages and
just checking it's existence was OK for many years.
So I'm +0.5 on this Python-side fix.

What's about other OS/flags?
Should we commit that every exposed socket flag is supported in runtime?
It looks like very heavy burden.

Or the issue is specific for TCP_KEEPCNT for Windows only?

What's about other OS/flags?
Should we commit that every exposed socket flag is supported in runtime?
It looks like very heavy burden.

I agree with you. It almost impossible.

Or the issue is specific for TCP_KEEPCNT for Windows only?

As far as my understanding, Yes. This issue is only for it.

I definitely don't think we should get into the game of trying to guess which flags are supported at runtime and only exposing those. It's not as simple as keeping a table of OS versions -- which would be hard enough to get right -- but on Linux you can have things like vendor backports of features to old versions, or a new kernel that happens to have had a particular feature configured out of it. (For example, AFAIK some major cloud providers still use kernels that have had IPv6 support removed entirely.)

Or the issue is specific for TCP_KEEPCNT for Windows only?

Four flags involved.
In this table, result column is search results from GitHub.

keyword available result
TCP_FASTOPEN win10 1607+ 778
TCP_KEEPCNT win10 1703+ 3356
TCP_KEEPIDLE win10 1709+ 4820
TCP_KEEPINTVL win10 1709+ 3410

3.6 branch is using 1703 SDK, 3.7/3.8 branches are using 1709 SDK.

This discussion is beyond my knowledge, I keep watch.

I suggest closing the issue as "won't fix": checking in runtime for only for TCP flags on Windows is a weird exception.
Checking all flags on all supported platforms is impossible.

Client libraries should process such situations themself.

I'm sympathetic to the idea that we don't want to carry around these checks, but also to the idea that this caused a regression in a micro release and that's not cool. Hence the idea that maybe we should keep everything the way it is in 3.7, but disable the new constants in 3.6, so that it arrives as part of a major release.

OTOH it's been in 3.6 for like 6 weeks already so I guess a lot of the affected parties are already working around it.

If the fix will land into 3.6 bugfix release only -- I can live with it, while the overall looks tricky.

Ned Deily, what do you think about?

Ned Deily, what do you think about?

I would like to have Steve's opinion.

In this case I like the flags disappearing on older versions, just as they would if you built CPython on a version of Linux that didn't have the flags. The problem is that the Windows SDK always defines enum values for all Windows versions even if you are targeting an older version, as most APIs silently ignore unknown flags. Since there's no way to reliably remove the flags at build time, it'll have to be done at import time.

In Python, existence normally implies availability, so we should maintain that here, especially since this API raises errors with these flags.

Oh, and checking Windows version is hard. Better for us to get it right than every single library to risk getting it wrong. (The code used in the second PR is not right - there are IsWindowsVersion* macros that are better.)

Ok

Here is PR 5585 for 3.6 branch.

For 3.7+, I would suggest patch in socketmodule.c like this:

PyMODINIT_FUNC
PyInit__socket(void)
{
    PyObject *m, *has_ipv6;
    ...
    ...
    ...
+#ifdef MS_WINDOWS
+   return remove_unusable_flags(m);
+#else
    return m;
+#endif
}

In this way, we handle the flags in a separated function remove_unusable_flags(m).
It keeps both socket.py and socketmodule.c neat.

Timelines FYI:

3.6.5 candidate: 2018-03-12 (tenative)
3.6.5 final: 2018-03-26 (tentative)

3.7.0 beta 2: 2018-02-26
3.7.0 beta 3: 2018-03-26
3.7.0 beta 4: 2018-04-30

What's about other OS/flags?
Should we commit that every exposed socket flag is supported in runtime?
It looks like very heavy burden.

I have an idea about this concern, I will post it after some experiments.

What's about other OS/flags?
Should we commit that every exposed socket flag is supported in runtime?
It looks like very heavy burden.

Let alone run-time check. Flags only depend on .C code and the building SDK, therefore, for a certain official release (e.g. CPython 3.6.5), the flags are fixed.
Then it is possible to get a flag-snapshot of a certain official release.

I wrote a script to dump/compare these flags in some Windows related modules (written in C language), see attached file winsdk_watchdog.py.

Let me demonstrate how to use it:

Comparing from A to B:
A: 3.6.3 (v3.6.3:2c5fed8, Oct 3 2017, 18:11:49) [MSC v.1900 64 bit (AMD64)]
B: 3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:54:40) [MSC v.1900 64 bit (AMD64)]

socket added 2 constants: {'TCP_KEEPCNT', 'TCP_FASTOPEN'}

Finished, 1 modules changed constants.

Comparing official 3.6.3 (1607 SDK) with official 3.6.4 (1703 SDK).
It caught the 2 flags lead to this issue.

Comparing from A to B:
A: 3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:54:40) [MSC v.1900 64 bit (AMD64)]
B: 3.7.0b1 (v3.7.0b1:9561d7f, Jan 31 2018, 07:26:34) [MSC v.1900 64 bit (AMD64)]

_winapi added 15 constants: {'NORMAL_PRIORITY_CLASS', 'FILE_TYPE_DISK', 'IDLE_PRIORITY_C
LASS', 'CREATE_DEFAULT_ERROR_MODE', 'CREATE_BREAKAWAY_FROM_JOB', 'FILE_TYPE_PIPE', 'FILE
_TYPE_REMOTE', 'BELOW_NORMAL_PRIORITY_CLASS', 'DETACHED_PROCESS', 'FILE_TYPE_CHAR', 'REA
LTIME_PRIORITY_CLASS', 'FILE_TYPE_UNKNOWN', 'ABOVE_NORMAL_PRIORITY_CLASS', 'CREATE_NO_WI
NDOW', 'HIGH_PRIORITY_CLASS'}

socket added 3 constants: {'TCP_KEEPIDLE', 'TCP_KEEPINTVL', 'MSG_ERRQUEUE'}

mmap added 1 constants: {'ACCESS_DEFAULT'}

Finished, 3 modules changed constants.

Comparing official 3.6.4 (1703 SDK) with official 3.7.0b1 (1709 SDK).

_winapi added 15 constants, after searching on GitHub repository, we know they were added in 2 commits:
b5d9e08#diff-c5f7cb301f3746a4c77e8bcd91d9f897
b2a6083#diff-c5f7cb301f3746a4c77e8bcd91d9f897
So they can be ignored.

socket added 3 constants.
After exploring, we know the 2 flags (TCP_KEEPIDLE, TCP_KEEPINTVL) were added by 1709 SDK, so we need to handle them during run-time as well.
Another new flag MSG_ERRQUEUE was also added by 1709 SDK, we need a socket expert decide what to do.

mmap added 1 constants.
It was added in:
5a8a84b#diff-a8a9c2d912381058181c8ffe496aa39b
Also ignore it.

This check is only needed after switching to a newer Windows SDK. As the file name, it's a watchdog of Windows SDK.

Some people build third-party-build by themselves, it's also possible to create a unittest, and teach it how to recognize flexible-flags (may be removed during run-time).

For example, a man builds CPython 3.7.0b1 with 1607 SDK (official 3.7.0b1 build with 1709 SDK), then he got a prompt like this:

These flags are missing in socket module:
'TCP_KEEPCNT', 'TCP_FASTOPEN', 'TCP_KEEPIDLE', 'TCP_KEEPINTVL', 'MSG_ERRQUEUE'
Maybe you are using a older SDK than official release's, or these flags are removed in this machine during run-time.

If he build CPython 3.7.0b1 with 1903 SDK in two years later, he may got prompt like this:

Unknown flags appear in socket module:
'TCP_XXXXXX', 'TCP_YYYYYY', 'TCP_ZZZZZZ'
They were added by newer Windows SDK, please make sure....

PR 5523 is prepared for master/3.7 branches.
PR 5585 is prepared for 3.6 branch.

What's the status of this issue? 3.7.0b2 is tagging in 48 hours or so and 3.6.5rc1 is in less than 2 weeks.

I think PRs could be merged

New changeset 19e7d48 by Steve Dower (animalize) in branch 'master':
bpo-32394: Remove some TCP options on old version Windows. (GH-5523)
19e7d48

New changeset 1278c21 by Steve Dower (animalize) in branch '3.6':
[3.6] bpo-32394: Remove some TCP options on older version Windows. (GH-5585)
1278c21

Agreed. I've merged these (and Miss Islington should get the 3.7 backport when CI completes)

New changeset 53d3f8a by Miss Islington (bot) in branch '3.7':
bpo-32394: Remove some TCP options on old version Windows. (GH-5523)
53d3f8a