Message351357
Oh, _posixsubprocess uses /dev/fd/ on FreeBSD, but only if it's mounted file system (expected to be fdescfs filesystem):
#if defined(__FreeBSD__) || (defined(__APPLE__) && defined(__MACH__))
# define FD_DIR "/dev/fd"
#else
# define FD_DIR "/proc/self/fd"
#endif
#if defined(__FreeBSD__)
/* When /dev/fd isn't mounted it is often a static directory populated
* with 0 1 2 or entries for 0 .. 63 on FreeBSD, NetBSD and OpenBSD.
* NetBSD and OpenBSD have a /proc fs available (though not necessarily
* mounted) and do not have fdescfs for /dev/fd. MacOS X has a devfs
* that properly supports /dev/fd.
*/
static int
_is_fdescfs_mounted_on_dev_fd(void)
{
struct stat dev_stat;
struct stat dev_fd_stat;
if (stat("/dev", &dev_stat) != 0)
return 0;
if (stat(FD_DIR, &dev_fd_stat) != 0)
return 0;
if (dev_stat.st_dev == dev_fd_stat.st_dev)
return 0; /* / == /dev == /dev/fd means it is static. #fail */
return 1;
}
#endif
On my FreeBSD 12 VM, MAXFD is around 230k which is quite large.
vstinner@freebsd$ uname -a
FreeBSD freebsd 12.0-RELEASE-p10 FreeBSD 12.0-RELEASE-p10 GENERIC amd64
vstinner@freebsd$ ./python
Python 3.9.0a0 (heads/master:4db25d5c39, Sep 9 2019, 07:52:01)
>>> import os; os.sysconf("SC_OPEN_MAX")
229284
It's easy to measure the overhead of 230k close() syscalls:
vstinner@freebsd$ python3 -m pyperf timeit -s 'import subprocess; args=["/usr/bin/true"]' 'subprocess.Popen(args, close_fds=False).wait()'
.....................
Mean +- std dev: 1.22 ms +- 0.12 ms
vstinner@freebsd$ python3 -m pyperf timeit -s 'import subprocess; args=["/usr/bin/true"]' 'subprocess.Popen(args, close_fds=True).wait()'
.....................
Mean +- std dev: 53.3 ms +- 1.4 ms
The overhead is 52.08 ms per Popen().wait() call (with close_fds=True).
If I mount manually the fdescfs filesystem, suddenly, subprocess is efficient again:
vstinner@freebsd$ sudo mount -t fdescfs /dev/fd
vstinner@freebsd$ python3 -m pyperf timeit -s 'import subprocess; args=["/usr/bin/true"]' 'subprocess.Popen(args, close_fds=True).wait()'
.....................
Mean +- std dev: 1.20 ms +- 0.06 ms
close_fds=True becomes as efficient as close_fds=False. |
|
Date |
User |
Action |
Args |
2019-09-09 08:16:13 | vstinner | set | recipients:
+ vstinner, koobs |
2019-09-09 08:16:13 | vstinner | set | messageid: <1568016973.1.0.704428007587.issue38061@roundup.psfhosted.org> |
2019-09-09 08:16:13 | vstinner | link | issue38061 messages |
2019-09-09 08:16:12 | vstinner | create | |
|