classification
Title: socket.TCP_NOTSENT_LOWAT is missing in official macOS builds
Type: Stage: patch review
Components: Extension Modules, macOS Versions: Python 3.9, Python 3.8, Python 3.7
process
Status: open Resolution:
Dependencies: 41100 Superseder:
Assigned To: Nosy List: Dima.Tisnek, Mariatta, malin, ned.deily, njs, ronaldoussoren
Priority: normal Keywords: patch

Created on 2020-03-25 07:20 by Dima.Tisnek, last changed 2020-10-26 00:07 by Dima.Tisnek.

Pull Requests
URL Status Linked Edit
PR 19402 open Dima.Tisnek, 2020-04-07 01:38
Messages (11)
msg364984 - (view) Author: Dima Tisnek (Dima.Tisnek) * Date: 2020-03-25 07:20
Somehow, it turns out that `TCP_NOTSENT_LOWAT` that's available since 3.7.x is not available in the official macOS builds 🙀:

> python3.7
Python 3.7.4 (v3.7.4:e09359112e, Jul  8 2019, 14:54:52)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> socket.TCP_NOTSENT_LOWAT
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'socket' has no attribute 'TCP_NOTSENT_LOWAT'

> python3.8
Python 3.8.2 (v3.8.2:7b3ab5921f, Feb 24 2020, 17:52:18)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> socket.TCP_NOTSENT_LOWAT
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'socket' has no attribute 'TCP_NOTSENT_LOWAT'

> python3.9
Python 3.9.0a4 (v3.9.0a4:6e02691f30, Feb 25 2020, 18:14:13)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> socket.TCP_NOTSENT_LOWAT
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'socket' has no attribute 'TCP_NOTSENT_LOWAT'

And my local build has it đŸ˜ē:

> ~/cpython/python.exe
Python 3.9.0a4+ (heads/master:be501ca241, Mar  4 2020, 15:16:49)
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> socket.TCP_NOTSENT_LOWAT
513

So... my guess is official builds are using old SDK or header files. 🤔

My system has it e.g. here:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/Kernel.framework/Versions/A/Headers/netinet/tcp.h
230:#define TCP_NOTSENT_LOWAT       0x201   /* Low water mark for TCP unsent data */

And in fact it's present in every `netinet/tcp.h` on my system: CommandLineTools 10.14 and 10.5 sdks; MacOSX dev sdk, {AppleTV,Watch,iPhone}{OS,Simulator} sdks.
msg365847 - (view) Author: Dima Tisnek (Dima.Tisnek) * Date: 2020-04-06 11:47
+macos team, because I can't for the life of me figure out how official builds are made ☚ī¸

In short: my local build has socket.TCP_NOTSENT_LOWAT but the official build does not.
msg365849 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2020-04-06 12:28
AFAIK the macOS builds are still build on the oldest macOS release supported by the installer (that is, a macOS 10.9 system). This means the build won't use macOS APIs introduced in macOS 10.10 or later.

----

It would be nice to build the installer using the latest compiler and SDK (more APIs available, better compiler, ...), but that requires work to explicitly avoid using APIs that aren't available on the system the binary is running on.
msg365883 - (view) Author: Dima Tisnek (Dima.Tisnek) * Date: 2020-04-07 00:05
Thank you for the explanation, Ronald.

`socket.TCP_NOTSENT_LOWAT` is just a constant though, to be passed to `setsockopt`. 

What do you think of `ifndef ... define ...` work-around, akin to a few other constants in socket module?
https://github.com/python/cpython/blob/799d7d61a91eb0ad3256ef9a45a90029cef93b7c/Modules/socketmodule.h#L162-L168
msg365889 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2020-04-07 06:00
I don't like that workaround. I'm also not sure if the value of these constants is the same on macOS and iOS (I do know that some constants higher up in the stack are not the same).

This issue is a duplicate of an earlier bug about missing functions that I can't find at the moment. That issue can only be solved by building on a recent enough version of macOS.
msg365891 - (view) Author: Dima Tisnek (Dima.Tisnek) * Date: 2020-04-07 06:26
The constant value is the same for macOS and iOS: iphone, watch, tv:

~ > locate netinet/tcp.h | xargs grep LOWAT
/Applications/Xcode.app/Contents/Developer/Platforms/AppleTVOS.platform/Developer/SDKs/AppleTVOS.sdk/usr/include/netinet/tcp.h:#define TCP_NOTSENT_LOWAT       0x201   /* Low water mark for TCP unsent data */
/Applications/Xcode.app/Contents/Developer/Platforms/AppleTVSimulator.platform/Developer/SDKs/AppleTVSimulator.sdk/usr/include/netinet/tcp.h:#define TCP_NOTSENT_LOWAT       0x201   /* Low water mark for TCP unsent data */
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/Kernel.framework/Versions/A/Headers/netinet/tcp.h:#define TCP_NOTSENT_LOWAT       0x201   /* Low water mark for TCP unsent data */
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/netinet/tcp.h:#define TCP_NOTSENT_LOWAT       0x201   /* Low water mark for TCP unsent data */
/Applications/Xcode.app/Contents/Developer/Platforms/WatchOS.platform/Developer/SDKs/WatchOS.sdk/usr/include/netinet/tcp.h:#define TCP_NOTSENT_LOWAT       0x201   /* Low water mark for TCP unsent data */
/Applications/Xcode.app/Contents/Developer/Platforms/WatchSimulator.platform/Developer/SDKs/WatchSimulator.sdk/usr/include/netinet/tcp.h:#define TCP_NOTSENT_LOWAT       0x201   /* Low water mark for TCP unsent data */
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk/usr/include/netinet/tcp.h:#define TCP_NOTSENT_LOWAT       0x201   /* Low water mark for TCP unsent data */
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator.sdk/usr/include/netinet/tcp.h:#define TCP_NOTSENT_LOWAT       0x201   /* Low water mark for TCP unsent data */
/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Kernel.framework/Versions/A/Headers/netinet/tcp.h:#define TCP_NOTSENT_LOWAT       0x201   /* Low water mark for TCP unsent data */
/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/usr/include/netinet/tcp.h:#define TCP_NOTSENT_LOWAT       0x201   /* Low water mark for TCP unsent data */
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/System/Library/Frameworks/Kernel.framework/Versions/A/Headers/netinet/tcp.h:#define TCP_NOTSENT_LOWAT       0x201   /* Low water mark for TCP unsent data */
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/netinet/tcp.h:#define TCP_NOTSENT_LOWAT       0x201   /* Low water mark for TCP unsent data */
msg365893 - (view) Author: Ma Lin (malin) * Date: 2020-04-07 06:54
Windows build encountered a similar problem, see issue32394.
The solution is to check the runtime system version when importing socket module, if it is an older system, delete the constants. [1]

issue32394 has a small script (winsdk_watchdog.py) to help find such constants, usage:
1, build a CPython build with old SDK.
2, use winsdk_watchdog.py, dump possible affected constants to a file `winsdk_dump.json`.
3, build a CPython build with new SDK.
4, use winsdk_watchdog.py, compare constants between two builds .

If a new constant is introduced by new SDK/API, we remove it on older system during runtime.
Otherwise we can ignore this new constant, this means it has nothing to do with the new SDK.
(msg311858 is a demo.)

We don't need to use winsdk_watchdog.py routinely, just use it when updating the building SDK, this process only takes about 10~20 minutes.

I think macOS build can also uses this process.

[1]
The commit:
https://github.com/python/cpython/commit/19e7d48ce89422091f9af93038b9fee075d46e9e

Note that there was a minor fix later:
https://github.com/python/cpython/commit/8905fcc85a6fc3ac394bc89b0bbf40897e9497a#diff-a47fd74731aeb547ad780900bb8e6953
msg365896 - (view) Author: Dima Tisnek (Dima.Tisnek) * Date: 2020-04-07 08:41
Wow, very curious.

Yes, it's very much like `socket.TCP_KEEPCNT` in that respect, though, admittedly I don't have a very old Mac to test this right now. I think there were VMs for that maybe? 🤔



I wonder, what failure would be best for a naive code below, a NameError or OSError(errno=42)?

sock.setsockopt(socket.SOL_TCP, socket.TCP_NOTSENT_LOWAT, 42)

I guess the question is, at what level should the users catch exceptions...
After all, we don't delete this constant on Linux, and surely someone somewhere runs a very old kernel...



Oddly according to https://github.com/search?l=Python&q=%22socket.TCP_NOTSENT_LOWAT%22&type=Code none (in the OSS community) appears to be using this feature yet?

The search without `socket.` prefix yields a bunch of vendored mypy pyi's, but no actual code either.

And some even work around the constant being optional: https://github.com/python-trio/trio/blob/5b91edb2a860d024ab057e2be55fb34f311bf8ed/trio/socket.py#L170-L178



So, would this be a "not a regression" if none appears to use this constant yet?

Or do we take "don't break existing code" so seriously, that in this case too, we ought to assume that there's someone out there who has private code like below which we must not break?:

if code := getattr(socket, "TCP_NOTSENT_LOWAT", None):
    sock.setsockopt(socket.SOL_TCP, code, 42)



P.S.
If someone wants to take https://github.com/python/cpython/pull/19402 forward, by all means :)
Or I can try to hack up delete-at-runtime...
msg365908 - (view) Author: Ma Lin (malin) * Date: 2020-04-07 14:45
It seems that people usually use the socket module like this, I think it's safe to respect this habit:

    if hasattr(socket, "FLAG_NAME"):
        do_something

If use PR19402, your program will have problem on the older version system, not only "don't break existing code".

So I think delete-at-runtime is a suitable way.
msg379440 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2020-10-23 15:06
The active PR on bpo-41100 will make it possible to build the installer on the latest version of macOS (which will be needed to be able to provide support for "Apple Silicon").

IMHO it is not necessary to remove unavailable constants, on some platforms (in particular Linux) some of the constants might not work because the relevant kernel module is not loaded. That's something you can only find by actively probing, which is too expensive and error prone. 

I propose closing this issue: the installers are what they are for a reason, and a better solution than proposed in this issue is on its way.
msg379622 - (view) Author: Dima Tisnek (Dima.Tisnek) * Date: 2020-10-26 00:07
Indeed, this issue can be closed, when the mac build update is confirmed to be included in a specific upcoming Python version.
History
Date User Action Args
2020-10-26 00:07:27Dima.Tisneksetmessages: + msg379622
2020-10-23 15:06:35ronaldoussorensetdependencies: + Support macOS 11 and Apple Silicon Macs
messages: + msg379440
2020-05-16 03:01:20terry.reedysetcomponents: + macOS
2020-04-07 14:45:54malinsetmessages: + msg365908
2020-04-07 08:41:21Dima.Tisneksetmessages: + msg365896
2020-04-07 06:54:10malinsetnosy: + malin
messages: + msg365893
2020-04-07 06:26:31Dima.Tisneksetmessages: + msg365891
2020-04-07 06:00:06ronaldoussorensetmessages: + msg365889
2020-04-07 01:38:44Dima.Tisneksetkeywords: + patch
stage: patch review
pull_requests: + pull_request18764
2020-04-07 00:05:01Dima.Tisneksetmessages: + msg365883
2020-04-06 12:28:18ronaldoussorensetmessages: + msg365849
2020-04-06 11:47:47Dima.Tisneksetnosy: + ronaldoussoren, ned.deily
messages: + msg365847
2020-03-25 07:20:26Dima.Tisnekcreate