classification
Title: Provide an abstraction for a select-able Event
Type: enhancement Stage: needs patch
Components: Library (Lib) Versions: Python 3.10
process
Status: open Resolution:
Dependencies: 41001 Superseder:
Assigned To: Nosy List: FFY00, christian.heimes, paravoid
Priority: normal Keywords:

Created on 2020-05-03 15:33 by paravoid, last changed 2021-04-09 18:07 by paravoid.

Messages (6)
msg367975 - (view) Author: Faidon Liambotis (paravoid) Date: 2020-05-03 15:33
In certain codebases, it's useful to be able to wait for input from one or more file descriptors (e.g. a socket), while at the same time waiting for an event triggered by another thread, or perhaps multiprocessing process.

To wait for one or more file descriptors to get ready, the select module can be used. However, neither threading.Event() nor multiprocessing.Event() are select-able, i.e. they provide no fileno() method.

The standard way one can implement this on Unix is with os.pipe(), but it can be tricky (forgetting to use non-blocking I/O etc.). It is also limited to a pair of processes at a time. On Linux systems from the past decade, one can also implement this much more efficiently using the eventfd() system calls. I think similar functionality exists in other Unixes with kqueue etc.

It'd be great if stdlib provided an abstraction over this mechanism. In fact, multiprocessing.Event() itself could probably be a thin abstraction over eventfd() on Linux? Perhaps even multiprocessing.Semaphore with EFD_SEMAPHORE, although I admit I'm less familiar with how all that works.

(Select-able Queues would be even neater, but that's a story for a different issue :)
msg380923 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-11-13 18:49
os.eventfd() has landed in Python 3.10.
msg390635 - (view) Author: Faidon Liambotis (paravoid) Date: 2021-04-09 14:57
Thanks so much for all the work on os.eventfd(), it's exciting to see it come to fruition.

An eventfd variant of Event (compatible with the threading & multiprocessing APIs) is now as simple as:

class Event:
    _ONE = (1).to_bytes(8, byteorder=sys.byteorder)

    def __init__(self):
        self._event_fd = os.eventfd(0, os.EFD_NONBLOCK)
        self._selector = selectors.DefaultSelector()
        self._selector.register(self._event_fd, selectors.EVENT_READ)

    def is_set(self):
        return self.wait(timeout=0)

    def set(self):
        try:
            os.write(self._event_fd, self._ONE)
        except BlockingIOError:
            pass

    def clear(self):
        try:
            os.read(self._event_fd, 8)
        except BlockingIOError:
            pass

    def wait(self, timeout=None):
        return bool(self._selector.select(timeout=timeout))

    def fileno(self):
        return self._event_fd

Given this now has a fileno() method, it is now possible to wait for the event as part of a broader selector, among other events (e.g. a file or socket becoming available to read or write).

I don't know where (or how) such a variant would fit into stdlib. It's simpler and more lightweight (less fds) than both threading's and multiprocessing's, and could be used from both threads and processes. I'd love some guidance here. (If a maintainer or anyone else reading this wants to use the above code in a PR, feel free -- no copyright claimed or expected for this trivial piece of code above)
msg390644 - (view) Author: Faidon Liambotis (paravoid) Date: 2021-04-09 16:33
I missed that there is now also an os.eventfd_{write,read}(), so in the above
    os.write(self._event_fd, self._ONE)
can become
    os.eventfd_write(self._event_fd, 1)

and:
    os.read(self._event_fd, 8)
can become:
    os.eventfd_read(self._event_fd)

(the _ONE declaration will then be unnecessary and can be removed)
msg390651 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-04-09 17:21
Do you want to work on a feature for 3.10? Feature freeze is in less than 4 weeks.
msg390655 - (view) Author: Faidon Liambotis (paravoid) Date: 2021-04-09 18:07
Not sure if I understand the question! I'd like to see that happen, I don't particularly care if it makes to 3.10 or a later version, although of course the earlier the better :)

As an idea of a path forward, would it make sense to take the code above and push it to multiprocessing as an alternative to Event with an if hasattr(os, "eventfd") guard?
History
Date User Action Args
2021-04-09 18:07:03paravoidsetmessages: + msg390655
2021-04-09 17:21:03christian.heimessetmessages: + msg390651
components: + Library (Lib), - Extension Modules
stage: needs patch
2021-04-09 16:33:53paravoidsetmessages: + msg390644
2021-04-09 14:57:01paravoidsetmessages: + msg390635
2020-11-13 18:49:55christian.heimessetnosy: + christian.heimes

messages: + msg380923
versions: + Python 3.10, - Python 3.9
2020-06-17 15:34:48christian.heimessetdependencies: + Provide wrapper for eventfd
2020-05-03 16:18:20FFY00setnosy: + FFY00
2020-05-03 15:33:27paravoidcreate