Author akira
Recipients abacabadabacaba, akira, benhoyt, giampaolo.rodola, josh.r, pitrou, socketpair, tebeka, tim.golden, vstinner
Date 2014-11-27.04:31:11
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1417062673.14.0.908320154024.issue22524@psf.upfronthosting.co.za>
In-reply-to
Content
To see what happens at syscall level, I've run various implementations
of get_tree_size() functions (see get_tree_size_listdir.diff) with
strace:

get_tree_size_listdir_fd   -- os.listdir(fd) + os.lstat
get_tree_size              -- os.scandir(path) + entry.lstat
get_tree_size_listdir_stat -- os.listdir(path) + os.lstat
get_tree_size_listdir      -- os.listdir(path) + os.path.isdir + os.lstat

Summary: 

- os.listdir() and os.scandir()-based variants use the same number of getdents() syscalls
- and the number of openat, lstat (newfstatat) syscalls is also comparable
  (except for the variant that uses os.path.isdir)

Log:

scandir$ /usr/bin/time strace -fco py.strace ../cpython/python -c 'from benchmark import get_tree_size_listdir_fd as f; import os; print(f(os.open("/usr/", os.O_RDONLY)))' && head -7 py.strace
5535240217
11.29user 8.14system 0:17.78elapsed 109%CPU (0avgtext+0avgdata 13460maxresident)k
0inputs+8outputs (0major+6781minor)pagefaults 0swaps
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 46.51    0.075252           0    264839           newfstatat
 16.88    0.027315           1     50460           getdents
 11.53    0.018660           0     75621           fcntl
  9.74    0.015758           0     50531           close
  6.87    0.011116           0     25214           openat
scandir$ /usr/bin/time strace -fco py.strace ../cpython/python -c 'from benchmark import get_tree_size as f; print(f("/usr/"))' && head -7 py.strace
5535240217
22.56user 8.47system 0:29.77elapsed 104%CPU (0avgtext+0avgdata 13280maxresident)k
0inputs+8outputs (0major+6306minor)pagefaults 0swaps
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 62.00    0.032822           0    239644           lstat
 19.97    0.010570           0     50460           getdents
  8.52    0.004510           0     25215           openat
  6.09    0.003224           0     25326           close
  0.55    0.000292           3        95           mmap
scandir$ /usr/bin/time strace -fco py.strace ../cpython/python -c 'from benchmark import get_tree_size_listdir_stat as f; print(f("/usr/"))' && head -7 py.strace
5535240217
13.70user 6.30system 0:18.84elapsed 106%CPU (0avgtext+0avgdata 13456maxresident)k
0inputs+8outputs (0major+6769minor)pagefaults 0swaps
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 64.79    0.050217           0    264849           lstat
 19.37    0.015011           0     50460           getdents
  8.22    0.006367           0     25215           openat
  5.76    0.004465           0     25326           close
  0.32    0.000247           2       114           mmap
scandir$ /usr/bin/time strace -fco py.strace ../cpython/python -c 'from benchmark import get_tree_size_listdir as f; print(f("/usr/"))' && head -7 py.strace
5535240217
19.53user 10.61system 0:28.16elapsed 107%CPU (0avgtext+0avgdata 13452maxresident)k
0inputs+8outputs (0major+6733minor)pagefaults 0swaps
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 42.04    0.063050           0    265195     37259 stat
 37.40    0.056086           0    265381           lstat
 11.46    0.017187           0     50460           getdents
  4.82    0.007232           0     25215           openat
  3.43    0.005139           0     25326           close
History
Date User Action Args
2014-11-27 04:31:13akirasetrecipients: + akira, tebeka, pitrou, vstinner, giampaolo.rodola, tim.golden, benhoyt, abacabadabacaba, socketpair, josh.r
2014-11-27 04:31:13akirasetmessageid: <1417062673.14.0.908320154024.issue22524@psf.upfronthosting.co.za>
2014-11-27 04:31:13akiralinkissue22524 messages
2014-11-27 04:31:12akiracreate