To be able to use GPIO Sysfs Interface on our embedded platforms we require exceptional event support.

> To be able to use GPIO Sysfs Interface on our embedded platforms we require exceptional event support.

Antoine Pitrou noticed that "exception" term can be confusion in Python, since exceptions are like "raise ValueError(...)".

The manual page mentions "out-of-band (OOB) data".

http://man7.org/linux/man-pages/man2/select_tut.2.html

       exceptfds
              This set is watched for "exceptional conditions".  In
              practice, only one such exceptional condition is common: the
              availability of out-of-band (OOB) data for reading from a TCP
              socket.  See recv(2), send(2), and tcp(7) for more details
              about OOB data.  (One other less common case where select(2)
              indicates an exceptional condition occurs with pseudoterminals
              in packet mode; see ioctl_tty(2).)  After select() has
              returned, exceptfds will be cleared of all file descriptors
              except for those for which an exceptional condition has
              occurred.

Is it what you need for a GPIO? GPIO is unrelated to TCP, right?

http://man7.org/linux/man-pages/man7/tcp.7.html



   Sockets API
       TCP provides limited support for out-of-band data, in the form of (a
       single byte of) urgent data.  In Linux this means if the other end
       sends newer out-of-band data the older urgent data is inserted as
       normal data into the stream (even when SO_OOBINLINE is not set).
       This differs from BSD-based stacks.

       Linux uses the BSD compatible interpretation of the urgent pointer
       field by default.  This violates RFC 1122, but is required for
       interoperability with other stacks.  It can be changed via
       /proc/sys/net/ipv4/tcp_stdurg.

       It is possible to peek at out-of-band data using the recv(2) MSG_PEEK
       flag.

       Since version 2.4, Linux supports the use of MSG_TRUNC in the flags
       argument of recv(2) (and recvmsg(2)).  This flag causes the received
       bytes of data to be discarded, rather than passed back in a caller-
       supplied buffer.  Since Linux 2.4.4, MSG_TRUNC also has this effect
       when used in conjunction with MSG_OOB to receive out-of-band data.

I don't know anything about OOB data, so I continue to add more pointers :-)

https://en.wikipedia.org/wiki/Out-of-band_data

"Out of band data is a logically independent transmission channel between a pair of connected stream sockets."

About naming: it is true that the manpage for select() uses the wording "exceptional condition".  But the Linux man page for poll() (and by extension epoll_wait()) does not use that wording, it says "There is urgent data to read".  The POSIX man page for poll() says "High-priority data may be read without blocking".

I would not argue about this if this weren't for the fact that "exception" already has a different meaning in Python land.  Re-using the term for something else is confusing.  Since "urgent data" and "high-priority data" are already accepted elsewhere for the same concept, reusing one of them should be ok.

> "urgent data", "High-priority data"

How is an application supposed to handle these data? Read them before any other data?

The selectors API returns a list of (key, events) tuples. So an application has to iterate on this list twice? A first time to look for urgent data, and then iterate again to handle other events?

Pseudo-code:

ready = selector.select(timeout)
for key, events in ready:
   if events & selectors.EVENT_URGENT:
       process_urgent_event(key, events)
for key, events in ready:
   process_other_events(key, events)

I just want to make sure that I understand correctly how these events should be used.

Would it be worth it to provide an helper to group urgent event and other events? Maybe a new select_urgent() method which would return two lists?

A selector which creates the ready list already knows if an event is urgent or not, and so could directly group them in two separated lists.

"Urgent event" doens't mean that key.events would only contain EVENT_URGENT, it can contain other events. So maybe the grouping function should be something like:

---
ready_urgent = []
ready = []
for ...:
   key = ...
   events = ...
   if not key:
      continue
   if events == EVENT_URGENT:
      ready_urgent.append((key, key.events & events))
   elif events & EVENT_URGENT:
      ready_urgent.append((key, key.events & events))
      ready.append((key, key.events & events))
   else:
      ready.append((key, key.events & events))
---

I don't know if it makes sense :-) Maybe it's better to let applications handle that themself ;-)

> "The selectors API returns a list of (key, events) tuples. So an application has to iterate on this list twice?"

No. "urgent data" means 'urgent' towards other events for thís key (key being the file object), not towards events for other file objects. 

AFAIK the returned ready list contains a single tuple for each file object, containing all events for that file object. Most likely urgent data events should be handled before handling other events for a given file object, but IMO there is no need to handle urgent data events of all file objects, before handling other events.

Pim Klanke: "(...) IMO there is no need to handle urgent data events of all file objects, before handling other events."

Hum, ok. So no need to extend the selectors API for that. I also understand that it's better to let applications decide how to prioritize these events :-) I'm totally fine if we only provide events and let the application decide how to handle them.

The question will be more important in the asyncio API, bpo-30847.

I'm confused about the wrapper method around winsock select and curious to why this is necessary. I have send an email to neologix to share some light on the subject.

In the selectors module, the winsock select method is wrapped. The second parameter (writefds) is passed to both writefds and exceptfds and the returned number of fds for write and except are summed and returned in the second element of the tuple.

Can anyone explain to me why the winsock select method is wrapped like this?

I'm not confortable with the change because of following questions:

* It seems like your patch changes the SelectSelector behaviour on Windows. How is a selectors user supposed to upgrade his/her code to get the same behaviour on Python 3.6 and 3.7? Would it make sense to add a flag to SelectSelectors get the old behaviour?

* KqueueSelector doesn't support urgent event. How is a selectors user suppose to be aware of them? Use a blacklist of selectors which doesn't support urgent events? This list might evolve in the future, so maybe the selector should announce which events are supported?

* This change alone is going to change asyncio behaviour on Windows, no? Because of the SelectSelector behaviour change on Windows.

Using Git history, I found the following commit which added "r, w, x = select(r, w, w, timeout)" for Windows in tulip (old name of the asyncio project, when it was developed outside CPython):

commit 84124f3d725c9931249d083e78f43fcda91c383a
Author: Richard Oudkerk <shibturn@gmail.com>
Date:   Fri Jan 18 13:03:09 2013 +0000

    Minimal chages to make tests pass on Windows

https://github.com/python/asyncio/commit/84124f3d725c9931249d083e78f43fcda91c383a

On 26-09-17 11:51, STINNER Victor wrote:
> 
> STINNER Victor added the comment:
> 
> I'm not confortable with the change because of following questions:
> 
> * It seems like your patch changes the SelectSelector behaviour on Windows. How is a selectors user supposed to upgrade his/her code to get the same behaviour on Python 3.6 and 3.7? Would it make sense to add a flag to SelectSelectors get the old behaviour?
The wrapper method around the winsock select method causes 
SelectSelector to behave differently on Windows, which is wrong IMHO. 
The behaviour should be the same. Why this wrapper exists is unclear to 
me and since it conflicts with the change I want to make, I asked 
questions about it 8 weeks ago. Now time has passed and no answers were 
given, I decided to remove it. Without further documentation I do not 
know how to modify it otherwise.

The other solution would be to add the new feature to all OS's but 
Windows, but again, since the wrapper method defies logic and is 
undocumented, I chose to remove it.
> 
> * KqueueSelector doesn't support urgent event. How is a selectors user suppose to be aware of them? Use a blacklist of selectors which doesn't support urgent events? This list might evolve in the future, so maybe the selector should announce which events are supported?
KqueueSelector raises a ValueError on registering an invalid event. 
Accepted events defined in the base class are READ, WRITE and URGENT. 
Accepted events is overruled in KQueueSelector, allowing only READ and 
WRITE.

> 
> * This change alone is going to change asyncio behaviour on Windows, no? Because of the SelectSelector behaviour change on Windows.
Reviewing the documentation of winsock select, I can think of no reason 
why this should change the runtime behaviour of SelectSelector, as well 
as asyncio.
> 
> ----------
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue30844>
> _______________________________________
>

On 26-09-17 12:29, STINNER Victor wrote:
> 
> STINNER Victor added the comment:
> 
> Using Git history, I found the following commit which added "r, w, x = select(r, w, w, timeout)" for Windows in tulip (old name of the asyncio project, when it was developed outside CPython):
> 
> commit 84124f3d725c9931249d083e78f43fcda91c383a
> Author: Richard Oudkerk <shibturn@gmail.com>
> Date:   Fri Jan 18 13:03:09 2013 +0000
> 
>      Minimal chages to make tests pass on Windows
> 
> https://github.com/python/asyncio/commit/84124f3d725c9931249d083e78f43fcda91c383a
Thanks!

Although this still does not teach us why the test failed on Windows and 
what is solved by adding the wrapper method, it does suggest that indeed 
the runtime behaviour will not be affected by removing the wrapper method.

The unit test might. To be honest, I did not run unit tests on Windows. 
Maybe by doing this, it might just tell us exactly what is accomplished 
with the wrapper method.
> 
> ----------
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue30844>
> _______________________________________
>

Anothe piece of history, the creation of the selectors module, bpo-16853, directly with the win32 "select.select(r, w, w, timeout)":

commit 243d8d85debaa319a2be0143003a9e881a0f5646
Author: Charles-François Natali <cf.natali@gmail.com>
Date:   Wed Sep 4 19:02:49 2013 +0200

    Issue #16853: Add new selectors module.

On Windows, exceptfds of select() is not only related to urgent ("out of band") data, it also notifies connect() failure:

exceptfds:

    If processing a connect call (nonblocking), connection attempt failed.
    OOB data is available for reading (only if SO_OOBINLINE is disabled).

https://msdn.microsoft.com/en-us/library/windows/desktop/ms740141(v=vs.85).aspx

I'm not sure that we can easily simplify the third parameter of select() as "urgent data". The exact semantics seems to not be portable at all.

poll() describes better the event types. Extract of my local Linux poll() manual:

POLLPRI: "There is urgent data to read (e.g., out-of-band data on TCP socket; pseudoterminal master in packet mode has seen state change in slave)."

POLLERR: "Error condition (only returned in revents; ignored in events)."

POLLHUP: "Hang up (only returned in revents; ignored in events).  Note that when reading from a channel such as a pipe or a stream socket, this event merely indicates that  the  peer closed its end of the channel.  Subsequent reads from the channel will return 0 (end of file) only after all outstanding data in the channel has been consumed."

etc.

It would help to look how Twisted, eventlet, gevent and others handle "urgent data" and "exceptions". Check if they succeeded to formalize these events.

asyncore uses select() or poll().

asyncore.poll() uses select.select(). It adds the fd to exceptfds if the fd is in the readfds or writefds, then asyncore calls _exception():

        r = []; w = []; e = []
        for fd, obj in map.items():
            is_r = obj.readable()
            is_w = obj.writable()
            (...)
            if is_r or is_w:
                e.append(fd)
        (...)

        try:
            r, w, e = select.select(r, w, e, timeout)
        except select.error, err:
            (...)

        (...)

        for fd in e:
            obj = map.get(fd)
            if obj is None:
                continue
            _exception(obj)

asyncore.poll2() uses select.poll(). It only uses POLLPRI if the fd is readable but always checks for error condition (POLLERR) (if asyncio waits for read and/or write events):

        for fd, obj in map.items():
            flags = 0
            if obj.readable():
                flags |= select.POLLIN | select.POLLPRI
            # accepting sockets should not be writable
            if obj.writable() and not obj.accepting:
                flags |= select.POLLOUT
            if flags:
                # Only check for exceptions if object was either readable
                # or writable.
                flags |= select.POLLERR | select.POLLHUP | select.POLLNVAL
                pollster.register(fd, flags)

On 26-09-17 14:10, STINNER Victor wrote:
> 
> STINNER Victor added the comment:
> 
> On Windows, exceptfds of select() is not only related to urgent ("out of band") data, it also notifies connect() failure:
> 
> exceptfds:
> 
>      If processing a connect call (nonblocking), connection attempt failed.
>      OOB data is available for reading (only if SO_OOBINLINE is disabled).
> 
> https://msdn.microsoft.com/en-us/library/windows/desktop/ms740141(v=vs.85).aspx
Ok. So what is achieved with the wrapper method is to catch a connection 
attempt failure error on the writefds, by signaling a WRITE_EVENT on the 
writefd.

Application will then most likely try to write to the fd and an error 
will occur, telling us the connection attempt failed.

I will try to change the wrapper method so that it still contains this 
hack (because that is what it is IMO), but also allows other fds to be 
monitored for exceptional events.

> 
> I'm not sure that we can easily simplify the third parameter of select() as "urgent data". The exact semantics seems to not be portable at all.
The goal of the patch is not to simplify the third parameter of select, 
but simply to extend the selectors module with the ability to subscribe 
for the POLLPRI ('urgent data to read') event, so selectors module can 
be used with sysfs gpio kernel driver.

To support this in SelectSelector the third parameter of select needs to 
be used, causing other exceptional events like POLLERR and POLLHUP to 
trigger select as well. For PollLikeSelector, this has always been so, 
because one does not need to register for POLLERR and POLLHUP (and 
POLLNVAL).

The goal of the selectors module is to offer a platform-independent 
abstraction layer on top of I/O monitoring functions in select. Our goal 
is to extend this with the ability to handle exceptional conditions.

Because some people get confused using the word 'exceptional' when 
programming in Python, I was asked to use one of the accepted synonyms. 
Since POLLPRI is the only exceptional condition event we explicitly need 
to register for, this is the description we choose. IMHO, this 
description makes what we try to accomplish harder to understand, than 
it was confusing.

> 
> poll() describes better the event types. Extract of my local Linux poll() manual:
> 
> POLLPRI: "There is urgent data to read (e.g., out-of-band data on TCP socket; pseudoterminal master in packet mode has seen state change in slave)."

> 
> POLLERR: "Error condition (only returned in revents; ignored in events)."
> 
> POLLHUP: "Hang up (only returned in revents; ignored in events).  Note that when reading from a channel such as a pipe or a stream socket, this event merely indicates that  the  peer closed its end of the channel.  Subsequent reads from the channel will return 0 (end of file) only after all outstanding data in the channel has been consumed."
> 
> etc.

> 
> ----------
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue30844>
> _______________________________________
>

On 26-09-17 14:01, STINNER Victor wrote:
> 
> STINNER Victor added the comment:
> 
> Anothe piece of history, the creation of the selectors module, bpo-16853, directly with the win32 "select.select(r, w, w, timeout)":
> 
> commit 243d8d85debaa319a2be0143003a9e881a0f5646
> Author: Charles-François Natali <cf.natali@gmail.com>
> Date:   Wed Sep 4 19:02:49 2013 +0200
> 
>      Issue #16853: Add new selectors module.
> 
> ----------
8 or so weeks ago I've send an email to Charles-François Natali (a.k.a. 
neologix). I asked him if he could explain the wrapper method. Neologix 
is in the nosy list as well, but has not answered any questions up till now
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue30844>
> _______________________________________
>

On 26-09-17 14:13, STINNER Victor wrote:
> 
> STINNER Victor added the comment:
> 
> It would help to look how Twisted, eventlet, gevent and others handle "urgent data" and "exceptions". 
First of all, there are no exceptions, only "exceptional conditions". 
Second of all, There is no "urgent data" AND "exceptional conditions"; 
"urgent data" is one of possible exceptional conditions, but not all 
exceptional conditions mean there is urgent data to read.

Check if they succeeded to formalize these events.
IMO, they have made it more confusing by using the word 'exception'. In 
this case I would have stuck with 'exceptional'. But at least it is 
better than having to replace it with 'urgent data'
> 
> asyncore uses select() or poll().
> 
> asyncore.poll() uses select.select(). It adds the fd to exceptfds if the fd is in the readfds or writefds, then asyncore calls _exception():
> 
>          r = []; w = []; e = []
>          for fd, obj in map.items():
>              is_r = obj.readable()
>              is_w = obj.writable()
>              (...)
>              if is_r or is_w:
>                  e.append(fd)
>          (...)
> 
>          try:
>              r, w, e = select.select(r, w, e, timeout)
>          except select.error, err:
>              (...)
> 
>          (...)
> 
>          for fd in e:
>              obj = map.get(fd)
>              if obj is None:
>                  continue
>              _exception(obj)
> 
> asyncore.poll2() uses select.poll(). It only uses POLLPRI if the fd is readable but always checks for error condition (POLLERR) (if asyncio waits for read and/or write events):
It registers for POLLERR, POLLHUP and POLLNVAL, events one does need not 
to subscribe for. These events are output events only and should not be 
added to mask.
> 
>          for fd, obj in map.items():
>              flags = 0
>              if obj.readable():
>                  flags |= select.POLLIN | select.POLLPRI
>              # accepting sockets should not be writable
>              if obj.writable() and not obj.accepting:
>                  flags |= select.POLLOUT
>              if flags:
>                  # Only check for exceptions if object was either readable
>                  # or writable.
>                  flags |= select.POLLERR | select.POLLHUP | select.POLLNVAL
>                  pollster.register(fd, flags)
> 
Adding POLLPRI to the flags if fd is readable doesn't cut it when using 
it with sysfs gpio kernel driver. The fd for a sysfs input is ALWAYS 
readable. We want to ONLY wake on POLLPRI event.
> ----------
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue30844>
> _______________________________________
>

It seems like kqueue supports urgent data:

"EV_OOBAND: Read filter on socket may set this flag to indicate the presence of out of band data on the descriptor."

Example: https://github.com/daurnimator/cqueues/commit/52baaf1c25bc7e6f7cb4685cb05f4ed47a3f404a

Be careful, I also found:

// Older versions of Mac OS X may not define EV_OOBAND.
#if !defined(EV_OOBAND)
# define EV_OOBAND EV_FLAG1
#endif // !defined(EV_OOBAND)

On 02-11-17 16:54, STINNER Victor wrote:
> STINNER Victor <victor.stinner@gmail.com> added the comment:
>
> It seems like kqueue supports urgent data:
>
> "EV_OOBAND: Read filter on socket may set this flag to indicate the presence of out of band data on the descriptor."
>
> Example: https://github.com/daurnimator/cqueues/commit/52baaf1c25bc7e6f7cb4685cb05f4ed47a3f404a
Looks promising. I will have to look into this.
> Be careful, I also found:
>
> // Older versions of Mac OS X may not define EV_OOBAND.
> #if !defined(EV_OOBAND)
> # define EV_OOBAND EV_FLAG1
> #endif // !defined(EV_OOBAND)
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue30844>
> _______________________________________
>