Author akira
Recipients abacabadabacaba, akira, benhoyt, giampaolo.rodola, josh.r, pitrou, socketpair, tebeka, tim.golden, vstinner
Date 2014-11-26.10:59:02
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1416999542.78.0.615275701284.issue22524@psf.upfronthosting.co.za>
In-reply-to
Content
> STINNER Victor added the comment:
>
>> scandir is slower on my machine:
>
> Please share more information about your config: OS, disk type (hard
> drive, SSD, something else), filesystem, etc.
>
Ubuntu 14.04, SSD, ext4 filesystem. Results for different python
versions are the same (scandir on pypy is even slower due to ctype
usage).

What other information could be useful?

> Ben Hoyt added the comment:
>
> Akira, note the "Using slower ctypes version of scandir" -- this is
> the older, ctypes implementation of scandir. On Linux, depending on
> file system etc, that can often be slower. You need to at least be
> using the "fast C version" in _scandir.c, which is then half C, half
> Python. But ideally you'd use the all-C version that I've provided as
> a CPython 3.5 patch.

Yes. I've noticed it, that is why I asked :) *"What commands should I
run to benchmark the attached patch (scandir-2.patch)?"*

I've read benchmark.py's source; It happens that it already supports
benchmarking of os.scandir. In my python checkout:

  cpython$ hg import --no-commit https://bugs.python.org/file36963/scandir-2.patch
  cpython$ make && cd ../scandir/
  scandir$ ../cpython/python benchmark.py -s /usr/
  Using Python 3.5's builtin os.scandir()
  Comparing against builtin version of os.walk()
  Priming the system's cache...
  Benchmarking walks on /usr/, repeat 1/3...
  Benchmarking walks on /usr/, repeat 2/3...
  Benchmarking walks on /usr/, repeat 3/3...
  os.walk size 7925376343, scandir.walk size 5534939617 -- NOT EQUAL!
  os.walk took 13.552s, scandir.walk took 6.032s -- 2.2x as fast

It seems os.walk and scandir.walk do a different work here.

I've written get_tree_size_listdir_fd() that mimics get_tree_size() 
(see get_tree_size_listdir.diff patch) to get the same size:

  scandir$ ../cpython/python benchmark.py -s /usr
  Using Python 3.5's builtin os.scandir()
  Comparing against builtin version of os.walk()
  Comparing against get_tree_size_listdir_fd
  Priming the system's cache...
  Benchmarking walks on /usr, repeat 1/3...
  Benchmarking walks on /usr, repeat 2/3...
  Benchmarking walks on /usr, repeat 3/3...
  os.walk size 5534939617, scandir.walk size 5534939617 -- equal
  os.walk took 5.697s, scandir.walk took 5.621s -- 1.0x as fast

scandir is not noticeably faster but scandir-based code is much nicer here.
History
Date User Action Args
2014-11-26 10:59:02akirasetrecipients: + akira, tebeka, pitrou, vstinner, giampaolo.rodola, tim.golden, benhoyt, abacabadabacaba, socketpair, josh.r
2014-11-26 10:59:02akirasetmessageid: <1416999542.78.0.615275701284.issue22524@psf.upfronthosting.co.za>
2014-11-26 10:59:02akiralinkissue22524 messages
2014-11-26 10:59:02akiracreate