Message318175
Patch in attachment makes shutil.copytree() use os.scandir() and (differently from #33414) DirEntry instances are passed around so that cached stat()s are used also from within copy2() and copystat() functions. The number of times the filesystem gets accessed via os.stat() is therefore reduced quite consistently. A similar improvement can be done for rmtree() (but that's for another ticket). Patch and benchmark script are in attachment.
Linux (+13.5% speedup)
======================
--- without patch:
./python bench.py
Priming the system's cache...
7956 files and dirs, repeat 1/3... min = 0.551s
7956 files and dirs, repeat 2/3... min = 0.548s
7956 files and dirs, repeat 3/3... min = 0.548s
best result = 0.548s
--- with patch:
$ ./python bench.py
Priming the system's cache...
7956 files and dirs, repeat 1/3... min = 0.481s
7956 files and dirs, repeat 2/3... min = 0.479s
7956 files and dirs, repeat 3/3... min = 0.474s
best result = 0.474s
Windows (+17% speedup)
======================
--- without patch:
./python bench.py
Priming the system's cache...
7956 files and dirs, repeat 1/3... min = 9.015s
7956 files and dirs, repeat 2/3... min = 8.747s
7956 files and dirs, repeat 3/3... min = 8.614s
best result = 8.614s
--- with patch:
$ ./python bench.py
Priming the system's cache...
7956 files and dirs, repeat 1/3... min = 7.827s
7956 files and dirs, repeat 2/3... min = 7.369s
7956 files and dirs, repeat 3/3... min = 7.153s
best result = 7.153s
Windows SMB share (+30%)
========================
--- without patch:
C:\Users\user\Desktop\cpython>PCbuild\win32\python.exe bench.py
Priming the system's cache...
7956 files and dirs, repeat 1/3... min = 46.853s
7956 files and dirs, repeat 2/3... min = 46.330s
7956 files and dirs, repeat 3/3... min = 44.720s
best result = 44.720s
--- with patch:
C:\Users\user\Desktop\cpython>PCbuild\win32\python.exe bench.py
Priming the system's cache...
7956 files and dirs, repeat 1/3... min = 31.729s
7956 files and dirs, repeat 2/3... min = 30.936s
7956 files and dirs, repeat 3/3... min = 30.936s
best result = 30.936s
Number of stat() syscalls (-38%)
================================
--- without patch:
$ strace ./python bench.py 2>&1 | grep "stat(" | wc -l
324808
--- with patch:
$ strace ./python bench.py 2>&1 | grep "stat(" | wc -l
198768 |
|
Date |
User |
Action |
Args |
2018-05-30 12:22:35 | giampaolo.rodola | set | recipients:
+ giampaolo.rodola |
2018-05-30 12:22:35 | giampaolo.rodola | set | messageid: <1527682955.69.0.682650639539.issue33695@psf.upfronthosting.co.za> |
2018-05-30 12:22:35 | giampaolo.rodola | link | issue33695 messages |
2018-05-30 12:22:35 | giampaolo.rodola | create | |
|