Author vstinner
Recipients koobs, vstinner
Date 2019-09-09.08:16:12
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1568016973.1.0.704428007587.issue38061@roundup.psfhosted.org>
In-reply-to
Content
Oh, _posixsubprocess uses /dev/fd/ on FreeBSD, but only if it's mounted file system (expected to be fdescfs filesystem):

#if defined(__FreeBSD__) || (defined(__APPLE__) && defined(__MACH__))
# define FD_DIR "/dev/fd"
#else
# define FD_DIR "/proc/self/fd"
#endif

#if defined(__FreeBSD__)
/* When /dev/fd isn't mounted it is often a static directory populated
 * with 0 1 2 or entries for 0 .. 63 on FreeBSD, NetBSD and OpenBSD.
 * NetBSD and OpenBSD have a /proc fs available (though not necessarily
 * mounted) and do not have fdescfs for /dev/fd.  MacOS X has a devfs
 * that properly supports /dev/fd.
 */
static int
_is_fdescfs_mounted_on_dev_fd(void)
{
    struct stat dev_stat;
    struct stat dev_fd_stat;
    if (stat("/dev", &dev_stat) != 0)
        return 0;
    if (stat(FD_DIR, &dev_fd_stat) != 0)
        return 0;
    if (dev_stat.st_dev == dev_fd_stat.st_dev)
        return 0;  /* / == /dev == /dev/fd means it is static. #fail */
    return 1;
}
#endif


On my FreeBSD 12 VM, MAXFD is around 230k which is quite large.

vstinner@freebsd$ uname -a
FreeBSD freebsd 12.0-RELEASE-p10 FreeBSD 12.0-RELEASE-p10 GENERIC  amd64

vstinner@freebsd$ ./python 
Python 3.9.0a0 (heads/master:4db25d5c39, Sep  9 2019, 07:52:01) 
>>> import os; os.sysconf("SC_OPEN_MAX")
229284

It's easy to measure the overhead of 230k close() syscalls:

vstinner@freebsd$ python3 -m pyperf timeit -s 'import subprocess; args=["/usr/bin/true"]' 'subprocess.Popen(args, close_fds=False).wait()' 
.....................
Mean +- std dev: 1.22 ms +- 0.12 ms
vstinner@freebsd$ python3 -m pyperf timeit -s 'import subprocess; args=["/usr/bin/true"]' 'subprocess.Popen(args, close_fds=True).wait()' 
.....................
Mean +- std dev: 53.3 ms +- 1.4 ms

The overhead is 52.08 ms per Popen().wait() call (with close_fds=True).



If I mount manually the fdescfs filesystem, suddenly, subprocess is efficient again:

vstinner@freebsd$ sudo mount -t fdescfs  /dev/fd 
vstinner@freebsd$ python3 -m pyperf timeit -s 'import subprocess; args=["/usr/bin/true"]' 'subprocess.Popen(args, close_fds=True).wait()' 
.....................
Mean +- std dev: 1.20 ms +- 0.06 ms

close_fds=True becomes as efficient as close_fds=False.
History
Date User Action Args
2019-09-09 08:16:13vstinnersetrecipients: + vstinner, koobs
2019-09-09 08:16:13vstinnersetmessageid: <1568016973.1.0.704428007587.issue38061@roundup.psfhosted.org>
2019-09-09 08:16:13vstinnerlinkissue38061 messages
2019-09-09 08:16:12vstinnercreate