Author Kirill Kolyshkin
Recipients Kirill Kolyshkin
Date 2019-01-17.04:29:00
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1547699341.03.0.843639889929.issue35757@roundup.psfhosted.org>
In-reply-to
Content
In case close_fds=True is passed to subprocess.Popen()
or its users (subprocess.call() etc), it might spend
some considerable time closing non-opened file descriptors,
as demonstrated by the following snippet from strace:

    close(3)                    = -1 EBADF (Bad file descriptor)
    close(5)                    = -1 EBADF (Bad file descriptor)
    close(6)                    = -1 EBADF (Bad file descriptor)
    close(7)                    = -1 EBADF (Bad file descriptor)
    ...
    close(1021)                 = -1 EBADF (Bad file descriptor)
    close(1022)                 = -1 EBADF (Bad file descriptor)
    close(1023)                 = -1 EBADF (Bad file descriptor)

This happens because the code in _close_fds() iterates from 3 up to
MAX_FDS = os.sysconf("SC_OPEN_MAX").

Now, syscalls are cheap, but SC_OPEN_MAX (also known as RLIMIT_NOFILE
or ulimit -n) can be quite high, for example:

    $ docker run --rm python python3 -c \
            $'import os\nprint(os.sysconf("SC_OPEN_MAX"))'
    1048576

This means a million syscalls before spawning a child process, which
can result in a major delay, like 0.1s as measured on my fast and mostly
idling laptop. Here is the comparison with python3 (which does not have this problem):

    $ docker run --rm python python3 -c $'import subprocess\nimport time\ns = time.time()\nsubprocess.check_call([\'/bin/true\'], close_fds=True)\nprint(time.time() - s)\n'
    0.0009245872497558594

    $ docker run --rm python python2 -c $'import subprocess\nimport time\ns = time.time()\nsubprocess.check_call([\'/bin/true\'], close_fds=True)\nprint(time.time() - s)\n'
    0.0964419841766
History
Date User Action Args
2019-01-17 04:29:04Kirill Kolyshkinsetrecipients: + Kirill Kolyshkin
2019-01-17 04:29:01Kirill Kolyshkinsetmessageid: <1547699341.03.0.843639889929.issue35757@roundup.psfhosted.org>
2019-01-17 04:29:00Kirill Kolyshkinlinkissue35757 messages
2019-01-17 04:29:00Kirill Kolyshkincreate