Message231703
> STINNER Victor added the comment:
>
>> scandir is slower on my machine:
>
> Please share more information about your config: OS, disk type (hard
> drive, SSD, something else), filesystem, etc.
>
Ubuntu 14.04, SSD, ext4 filesystem. Results for different python
versions are the same (scandir on pypy is even slower due to ctype
usage).
What other information could be useful?
> Ben Hoyt added the comment:
>
> Akira, note the "Using slower ctypes version of scandir" -- this is
> the older, ctypes implementation of scandir. On Linux, depending on
> file system etc, that can often be slower. You need to at least be
> using the "fast C version" in _scandir.c, which is then half C, half
> Python. But ideally you'd use the all-C version that I've provided as
> a CPython 3.5 patch.
Yes. I've noticed it, that is why I asked :) *"What commands should I
run to benchmark the attached patch (scandir-2.patch)?"*
I've read benchmark.py's source; It happens that it already supports
benchmarking of os.scandir. In my python checkout:
cpython$ hg import --no-commit https://bugs.python.org/file36963/scandir-2.patch
cpython$ make && cd ../scandir/
scandir$ ../cpython/python benchmark.py -s /usr/
Using Python 3.5's builtin os.scandir()
Comparing against builtin version of os.walk()
Priming the system's cache...
Benchmarking walks on /usr/, repeat 1/3...
Benchmarking walks on /usr/, repeat 2/3...
Benchmarking walks on /usr/, repeat 3/3...
os.walk size 7925376343, scandir.walk size 5534939617 -- NOT EQUAL!
os.walk took 13.552s, scandir.walk took 6.032s -- 2.2x as fast
It seems os.walk and scandir.walk do a different work here.
I've written get_tree_size_listdir_fd() that mimics get_tree_size()
(see get_tree_size_listdir.diff patch) to get the same size:
scandir$ ../cpython/python benchmark.py -s /usr
Using Python 3.5's builtin os.scandir()
Comparing against builtin version of os.walk()
Comparing against get_tree_size_listdir_fd
Priming the system's cache...
Benchmarking walks on /usr, repeat 1/3...
Benchmarking walks on /usr, repeat 2/3...
Benchmarking walks on /usr, repeat 3/3...
os.walk size 5534939617, scandir.walk size 5534939617 -- equal
os.walk took 5.697s, scandir.walk took 5.621s -- 1.0x as fast
scandir is not noticeably faster but scandir-based code is much nicer here. |
|
Date |
User |
Action |
Args |
2014-11-26 10:59:02 | akira | set | recipients:
+ akira, tebeka, pitrou, vstinner, giampaolo.rodola, tim.golden, benhoyt, abacabadabacaba, socketpair, josh.r |
2014-11-26 10:59:02 | akira | set | messageid: <1416999542.78.0.615275701284.issue22524@psf.upfronthosting.co.za> |
2014-11-26 10:59:02 | akira | link | issue22524 messages |
2014-11-26 10:59:02 | akira | create | |
|