Author njs
Recipients asvetlov, christian.heimes, inada.naoki, njs, pitrou, vstinner, yselivanov
Date 2017-12-16.04:17:08
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1513397831.78.0.213398074469.issue32331@psf.upfronthosting.co.za>
In-reply-to
Content
I did a little digging. It's more complicated than just "on Linux, these show up in the socket type".

Background: SOCK_NONBLOCK and SOCK_CLOEXEC are Linux-isms. The standard way to control blocking-ness and cloexec-ness is via ioctl/fcntl, and Linux supports that too. But as an extension, it also has these flags that can be passed as part of the socket type, which is nice because it lets you set the flag atomically at the time the socket is created, which is very important for CLOEXEC, and ... kinda pointless for NONBLOCK, but whatever, Linux supports it.

Note that as far as Linux is concerned, this is just a sneaky way to smuggle an extra argument through the socket() call. The flags *do not become part of the socket type*. If you create a socket with type=SOCK_STREAM | SOCK_NONBLOCK, and then do getsockopt(SOL_SOCKET, SOCK_TYPE) on it, Linux reports that the socket's type is just SOCK_STREAM, *without* mentioning SOCK_NONBLOCK.

One more important piece of background: Python's socket.type attribute doesn't really have any connection to the OS-level socket type. (This is part of what bpo-28134 is about.) Python just takes whatever value the user passed as the type= argument when calling socket(), and stashes it in an instance variable so it can be retrieved later.

Okay, so what's going on with socket.type? Python gained support for SOCK_NONBLOCK and SOCK_CLOEXEC in commit b1c54967381062bccef7db574b8e84f48a0eca76 (bpo-7523), which mainly just exposed the constants, so that users could manually pass these in as part of the type= argument when creating the socket. However, it also made another change: it made it so that that whenever Python toggles the blocking state of the socket, it also toggles the SOCK_NONBLOCK bit in the Python-level variable that represents socket.type.

Logically, this doesn't make any sense. As noted above, SOCK_NONBLOCK is never considered part of the socket type, on any operating system. And the function that is doing this doesn't even have anything to do with SOCK_NONBLOCK; it's actually using ioctl/fcntl, which is a totally unrelated mechanism for toggling blocking-ness. And it only does this on Linux (because only Linux *has* a SOCK_NONBLOCK bit to toggle), so this is kind of just introducing gratuitous brokenness.

The one advantage of this is that it makes these two pieces of code consistent:

   # Version A
   s = socket.socket(type=socket.SOCK_STREAM | socket.SOCK_NONBLOCK)
   # Blindly returns what we passed to type=, even though it's
   # not the actual type:
   s.type

   # Version B
   s = socket.socket()
   s.setblocking(False)
   s.type

This is a weird choice. Basically version A here is revealing a bug in how Python's type tracking works (s.type disagrees with s.getsockopt(SOL_SOCKET, SOL_TYPE)), that is only triggered if the user takes advantage of a rarely-used, non-portable feature. Then version B was written to intentionally introduce the same bug in a much more common case, I guess for consistency. But the bug is still non-portable.

The only time SOCK_CLOEXEC shows up in socket.type is if the user explicitly passed it in their call to socket.socket(), which has been a no-op (except for its effect on socket.type) since PEP 446 made SOCK_CLOEXEC the default.

I think there's a pretty strong argument here that what we really ought to do is *never* show SOCK_NONBLOCK or SOCK_CLOEXEC in in socket.type. That's what Linux itself does, it's the only possibility on every other OS, and it fixes this very annoying issue. (To add to Yury's comments about bugs in asycnio, it took me ages to figure out wtf was going on here and work around it in trio. The current behavior really does cause bugs.) It also sets the stage for fixing bpo-28134 -- right now socket.type's semantics are "whatever you passed in, whether that's actually the socket's type or not", and that's something that's impossible to determine reliably for a bare fd; I'm suggesting they should become "the socket's type", which *is* possible to determine reliably for a bare fd, at least in some cases.
History
Date User Action Args
2017-12-16 04:17:11njssetrecipients: + njs, pitrou, vstinner, christian.heimes, asvetlov, inada.naoki, yselivanov
2017-12-16 04:17:11njssetmessageid: <1513397831.78.0.213398074469.issue32331@psf.upfronthosting.co.za>
2017-12-16 04:17:11njslinkissue32331 messages
2017-12-16 04:17:08njscreate