Author benhoyt
Recipients Trundle, abacabadabacaba, benhoyt, brian.curtin, christian.heimes, eric.araujo, giampaolo.rodola, gregory.p.smith, loewis, ncoghlan, neologix, nvetoshkin, pitrou, rhettinger, serhiy.storchaka, socketpair, terry.reedy, tim.golden, torsten, twouters, vstinner
Date 2013-05-05.21:43:22
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1367790202.96.0.70313907442.issue11406@psf.upfronthosting.co.za>
In-reply-to
Content
Yeah, I very much agree with what Nick says -- we really want a way to expose what the platform provides. It's  less important (though still the ideal), to expose that in a platform-independent way. Today the only way to get access to opendir/readdir on Linux and FindFirst/Next on Windows is by using a bunch of messy (and slowish) ctypes code. And yes, os.walk() would be the main cross-platform abstraction built on top of this.

Charles gave this example of code that would fall over:

size = 0
for name, st in scandir(path):
    if stat.S_ISREG(st.st_mode):
        size += st.st_size

I don't see it, though. In this case you need both .st_mode and .st_size, so a caller would check that those are not None, like so:

size = 0
for name, st in scandir(path):
    if st.st_mode is None or st.st_size is None:
        st = os.stat(os.path.join(path, name))
    if stat.S_ISREG(st.st_mode):
        size += st.st_size

One line of extra code for the caller, but a big performance gain in most cases.

Stinner said, "But as usual, a benchmark on a real platform would be more convicing". Here's a start: https://github.com/benhoyt/betterwalk#benchmarks -- but it's not nearly as good as it gets yet, because those figures are still using the ctypes version. I've got a C version that's half-finished, and on Windows it makes os.walk() literally 10x the speed of the default version. Not sure about Linux/opendir/readdir yet, but I intend to do that too.
History
Date User Action Args
2013-05-05 21:43:23benhoytsetrecipients: + benhoyt, loewis, twouters, rhettinger, terry.reedy, gregory.p.smith, ncoghlan, pitrou, vstinner, giampaolo.rodola, christian.heimes, tim.golden, eric.araujo, Trundle, brian.curtin, torsten, nvetoshkin, neologix, abacabadabacaba, socketpair, serhiy.storchaka
2013-05-05 21:43:22benhoytsetmessageid: <1367790202.96.0.70313907442.issue11406@psf.upfronthosting.co.za>
2013-05-05 21:43:22benhoytlinkissue11406 messages
2013-05-05 21:43:22benhoytcreate