This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author n8henrie
Recipients n8henrie, ned.deily
Date 2018-03-20.16:06:24
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1521561984.32.0.467229070634.issue33036@psf.upfronthosting.co.za>
In-reply-to
Content
I've continued looking into this.

If you have your limits configured higher than default, as I did (and which seems to be working fine):

```
sudo launchctl limit maxfiles 64000 524288
ulimit -Sn 64000
```

then you'll find that having a soft limit (`NUM_FDS`, prior to the `-=32`) greater than 10273 causes the error I was seeing.

First, I found this number by passing in NUM_FDS as an envvar to the unittest and looping in a bash script, but I have since extracted some of the relevant portions to a separate python script:

```python
import selectors
import socket
import sys

def main(NUM_FDS):
    s = selectors.PollSelector()

    NUM_FDS -= 32

    sockets = []

    try:
        for i in range(NUM_FDS // 2):
            try:
                rd, wr = socket.socketpair()
                sockets.extend([rd, wr])
            except OSError as e:
                print(f"err 2: {type(e)} {e}")
                # too many FDs, skip - note that we should only catch EMFILE
                # here, but apparently *BSD and Solaris can fail upon connect()
                # or bind() with EADDRNOTAVAIL, so let's be safe
                print("FD limit reached")

            try:
                s.register(rd, selectors.EVENT_READ)
                s.register(wr, selectors.EVENT_WRITE)
            except OSError as e:
                print(f"err 3: {type(e)} {e}")
                if e.errno == errno.ENOSPC:
                    # this can be raised by epoll if we go over
                    # fs.epoll.max_user_watches sysctl
                    print("FD limit reached")
                raise

        try:
            slen = len(s.select())
        except OSError as e:
            print(f"err 4: {type(e)} {e}")
            raise
        print(f"asserting {NUM_FDS // 2} == {slen}")
        assert NUM_FDS // 2 == slen

    finally:
        for sock in sockets:
            sock.close()
        s.close()
        print("Closed")

if __name__ == "__main__":
    main(int(sys.argv[1]))
```


```shellsession
$ ./python.exe tester.py 10273
asserting 5120 == 5120
Closed
$ ./python.exe tester.py 10274
err 4: <class 'OSError'> [Errno 22] Invalid argument
Closed
Traceback (most recent call last):
  File "tester.py", line 50, in <module>
    main(int(sys.argv[1]))
  File "tester.py", line 36, in main
    slen = len(s.select())
  File "cpython/Lib/selectors.py", line 376, in select
    fd_event_list = self._poll.poll(timeout)
OSError: [Errno 22] Invalid argument
```

Tested and confirmed that I can provoke the error with the above launchctl / ulimit settings on my wife's Macbook Air.

I wonder if the 10273 limit I'm running up against has something to do with the 10240 number I keep running into as a default kern.maxfilesperproc (https://unix.stackexchange.com/a/350824/77904), given the `NUM_FDS -= 32` in the test.
History
Date User Action Args
2018-03-20 16:06:24n8henriesetrecipients: + n8henrie, ned.deily
2018-03-20 16:06:24n8henriesetmessageid: <1521561984.32.0.467229070634.issue33036@psf.upfronthosting.co.za>
2018-03-20 16:06:24n8henrielinkissue33036 messages
2018-03-20 16:06:24n8henriecreate