Author neologix
Recipients Trundle, benhoyt, brian.curtin, christian.heimes, eric.araujo, giampaolo.rodola, gregory.p.smith, loewis, ncoghlan, neologix, nvetoshkin, pitrou, rhettinger, serhiy.storchaka, socketpair, terry.reedy, tim.golden, torsten, twouters, vstinner
Date 2013-05-03.07:03:26
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <CAH_1eM1f70xAakhRMcdMVVZGRBKGai_Q5NcqF0vZuHdRZc=WhA@mail.gmail.com>
In-reply-to <1367483314.14.0.571258985959.issue11406@psf.upfronthosting.co.za>
Content
> However, the reason I'm keen on iterdir_stat() is that I'm seeing it speed up os.walk() by a factor of 10 in my recent tests (note that I've made local mods, so these results aren't reproducible for others yet). This is doing a walk on a dir tree with 7800 files and 155 dirs:
>
> Using fast _betterwalk
> Priming the system's cache...
> Benchmarking walks on C:\Work\betterwalk\benchtree, repeat 1/3...
> Benchmarking walks on C:\Work\betterwalk\benchtree, repeat 2/3...
> Benchmarking walks on C:\Work\betterwalk\benchtree, repeat 3/3...
> os.walk took 0.178s, BetterWalk took 0.017s -- 10.5x as fast
>
> Sometimes Windows will go into this "I'm really caching stat results good" mode -- I don't know what heuristic determines this -- and then I'm seeing a 40x speed increase. And no, you didn't read that wrong. :-)

I/O benchmarks shouldn't use timeit or repeated calls: after the first
run, most of your data is in cache, so subsequent runs are
meaningless.

I don't know about Windows, but on Linux you should do something like:
# echo 3 > /proc/sys/vm/drop_caches

to start out clean.
History
Date User Action Args
2013-05-03 07:03:27neologixsetrecipients: + neologix, loewis, twouters, rhettinger, terry.reedy, gregory.p.smith, ncoghlan, pitrou, vstinner, giampaolo.rodola, christian.heimes, tim.golden, eric.araujo, Trundle, brian.curtin, benhoyt, torsten, nvetoshkin, socketpair, serhiy.storchaka
2013-05-03 07:03:27neologixlinkissue11406 messages
2013-05-03 07:03:26neologixcreate