Message188468
Yeah, I very much agree with what Nick says -- we really want a way to expose what the platform provides. It's less important (though still the ideal), to expose that in a platform-independent way. Today the only way to get access to opendir/readdir on Linux and FindFirst/Next on Windows is by using a bunch of messy (and slowish) ctypes code. And yes, os.walk() would be the main cross-platform abstraction built on top of this.
Charles gave this example of code that would fall over:
size = 0
for name, st in scandir(path):
if stat.S_ISREG(st.st_mode):
size += st.st_size
I don't see it, though. In this case you need both .st_mode and .st_size, so a caller would check that those are not None, like so:
size = 0
for name, st in scandir(path):
if st.st_mode is None or st.st_size is None:
st = os.stat(os.path.join(path, name))
if stat.S_ISREG(st.st_mode):
size += st.st_size
One line of extra code for the caller, but a big performance gain in most cases.
Stinner said, "But as usual, a benchmark on a real platform would be more convicing". Here's a start: https://github.com/benhoyt/betterwalk#benchmarks -- but it's not nearly as good as it gets yet, because those figures are still using the ctypes version. I've got a C version that's half-finished, and on Windows it makes os.walk() literally 10x the speed of the default version. Not sure about Linux/opendir/readdir yet, but I intend to do that too. |
|
Date |
User |
Action |
Args |
2013-05-05 21:43:23 | benhoyt | set | recipients:
+ benhoyt, loewis, twouters, rhettinger, terry.reedy, gregory.p.smith, ncoghlan, pitrou, vstinner, giampaolo.rodola, christian.heimes, tim.golden, eric.araujo, Trundle, brian.curtin, torsten, nvetoshkin, neologix, abacabadabacaba, socketpair, serhiy.storchaka |
2013-05-05 21:43:22 | benhoyt | set | messageid: <1367790202.96.0.70313907442.issue11406@psf.upfronthosting.co.za> |
2013-05-05 21:43:22 | benhoyt | link | issue11406 messages |
2013-05-05 21:43:22 | benhoyt | create | |
|