This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author neologix
Recipients Trundle, benhoyt, brian.curtin, christian.heimes, eric.araujo, giampaolo.rodola, gregory.p.smith, loewis, ncoghlan, neologix, nvetoshkin, pitrou, rhettinger, serhiy.storchaka, socketpair, terry.reedy, tim.golden, torsten, twouters, vstinner
Date 2013-05-03.07:03:26
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <CAH_1eM1f70xAakhRMcdMVVZGRBKGai_Q5NcqF0vZuHdRZc=WhA@mail.gmail.com>
In-reply-to <1367483314.14.0.571258985959.issue11406@psf.upfronthosting.co.za>
Content
> However, the reason I'm keen on iterdir_stat() is that I'm seeing it speed up os.walk() by a factor of 10 in my recent tests (note that I've made local mods, so these results aren't reproducible for others yet). This is doing a walk on a dir tree with 7800 files and 155 dirs:
>
> Using fast _betterwalk
> Priming the system's cache...
> Benchmarking walks on C:\Work\betterwalk\benchtree, repeat 1/3...
> Benchmarking walks on C:\Work\betterwalk\benchtree, repeat 2/3...
> Benchmarking walks on C:\Work\betterwalk\benchtree, repeat 3/3...
> os.walk took 0.178s, BetterWalk took 0.017s -- 10.5x as fast
>
> Sometimes Windows will go into this "I'm really caching stat results good" mode -- I don't know what heuristic determines this -- and then I'm seeing a 40x speed increase. And no, you didn't read that wrong. :-)

I/O benchmarks shouldn't use timeit or repeated calls: after the first
run, most of your data is in cache, so subsequent runs are
meaningless.

I don't know about Windows, but on Linux you should do something like:
# echo 3 > /proc/sys/vm/drop_caches

to start out clean.
History
Date User Action Args
2013-05-03 07:03:27neologixsetrecipients: + neologix, loewis, twouters, rhettinger, terry.reedy, gregory.p.smith, ncoghlan, pitrou, vstinner, giampaolo.rodola, christian.heimes, tim.golden, eric.araujo, Trundle, brian.curtin, benhoyt, torsten, nvetoshkin, socketpair, serhiy.storchaka
2013-05-03 07:03:27neologixlinkissue11406 messages
2013-05-03 07:03:26neologixcreate