classification
Title: There is no os.listdir() equivalent returning generator instead of list
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 3.5
process
Status: closed Resolution: rejected
Dependencies: Superseder: PEP 471 implementation: os.scandir() directory scanning function
View: 22524
Assigned To: gregory.p.smith Nosy List: Trundle, abacabadabacaba, benhoyt, christian.heimes, eric.araujo, ethan.furman, giampaolo.rodola, gregory.p.smith, haypo, josh.r, loewis, mmarkk, nailor, ncoghlan, neologix, nvetoshkin, pitrou, rhettinger, terry.reedy, tim.golden, torsten, twouters
Priority: normal Keywords: patch

Created on 2011-03-05 10:17 by mmarkk, last changed 2014-09-30 14:47 by ncoghlan. This issue is now closed.

Files
File name Uploaded Description Edit
issue11406-gps01.diff gregory.p.smith, 2013-03-24 22:56 Implements os.scandir(path='.') that iterates review
issue11406-gps02.diff gregory.p.smith, 2013-03-31 21:08 implements os.scandir(path='.') that iterates review
Messages (77)
msg130111 - (view) Author: Марк Коренберг (mmarkk) Date: 2011-03-05 10:17
Big dirs are really slow to read at once. If user wants to read items one by one like here:
--------------
for i in os.listdir()
    use(i)
--------------
having generator will gain performance, as big directories often very fragmented on disk. Also, dir_cache in kernel used more effectively.
msg130112 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2011-03-05 10:45
> Big dirs are really slow to read at once. 

Do you a proof for that claim? How big, and how really slow?

> for i in os.listdir()
>     use(i)

Also, how long does use(i) take, and what reduction (in percent)
can you gain from listdir iterating?

In short, I'm skeptical that there is an actual problem to be solved here.
msg130114 - (view) Author: Марк Коренберг (mmarkk) Date: 2011-03-05 11:02
also, forgot... memory usage on big directories using list is a pain.

This is the same things as range() and xrange(). Why not to add os.xlistdir() ?

P.S.
Numerical answers will be available later.
msg130125 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-03-05 17:11
A generator listdir() geared towards performance should probably be able to work in batches, e.g. read 100 entries at once and buffer them in some internal storage (that might mean use readdir_r()). Bonus points if it doesn't release the GIL around each individual entry, but also batches that.
msg130129 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2011-03-05 18:48
> Big dirs are really slow to read at once. If user wants to read items one by one like here

The problem is that readdir doesn't read a directory entry one at a time.
When you call readdir on an open DIR * for the first time, the libc calls the getdents syscall, requesting a whole bunch of dentry at a time (32768 on my box).
Then, the subsequent readdir calls are virtually free, and don't involve any syscall/IO at all (that is, until you hit the last cached dent, and then another getdents is performed until end of directory).

> Also, dir_cache in kernel used more effectively.

You mean the dcache ? Could you elaborate ?

> also, forgot... memory usage on big directories using list is a pain.

This would indeed be a good reason. Do you have numbers ?

> A generator listdir() geared towards performance should probably be able to work in batches, e.g. read 100 entries at once and buffer them in some internal storage (that might mean use readdir_r()).

That's exactly what readdir is doing :-)

> Bonus points if it doesn't release the GIL around each individual entry, but also batches that.

Yes, since only one in 2**15 readdir call actually blocks, that could be a nice optimization (I've no idea of the potential gain though).

> Big dirs are really slow to read at once.

Are you using EXT3 ?
There are records of performance issues with getdents on EXT2/3 filesystems, see:
http://lwn.net/Articles/216948/
and this nice post by Linus:
https://lkml.org/lkml/2007/1/7/149

Could you provide the output of an "strace -ttT python <test script>"  (and also the time spent in os.listdir) ?
msg130167 - (view) Author: Vetoshkin Nikita (nvetoshkin) Date: 2011-03-06 12:01
Generator listdir() could be useful if I have a directory with several millions of files and I what to process just a hundred.
msg130256 - (view) Author: Vetoshkin Nikita (nvetoshkin) Date: 2011-03-07 13:57
Glibc's readdir() and readdir_r() already do caching, so getdents() syscall is called only once on my '/etc' directory. Should we include another caching level in xlistdir() function?
On the other hand, we don't know anything about caches at glibc's level, i.e. we can't tell if our next call to readdir() will result in syscall or even I/O (we could possible release GIL for that).
msg130286 - (view) Author: Марк Коренберг (mmarkk) Date: 2011-03-07 19:43
> Glibc's readdir() and readdir_r() already do caching
Yes, but glibc's readdir is the C analogue of python's generator. We do not need to create cache for cached values.
I think it's OK to make python's generator on top of readdir (instead of getdents).

Why not to create generator like this?
(pseudocode)
------------------
DIR *d;
struct dirent* entry, *e;
entry = malloc(offsetof(struct dirent, d_name) + pathconf(dirpath, _PC_NAME_MAX) + 1);
if (!e)
    raise Exception();
if (!(d= opendir(dirname)))
{
    free(e)
    raise IOException();
}

for (;;)
{
    if (readdir_r(d, entry, &e))
    {
        closedir(d);
        free(entry);
        raise IOException();
    }
    if (!e)
        break;
    yield e;
}
closedir(d);
free(entry);

------------------
msg130391 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-03-09 00:31
There has been discussion of this before, but it must have been on one of the lists, (possibly py3k list) as searching tracker for 'listdir generator' only returns this.

I believe I pointed out then that Miscrosoft C (also) has (did once) a 'nextdir' function. It's been so long that I forget details.

I thought then and still do that listdir should (have) change (d) like range, map, and filter did, for same reasons.
msg130392 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2011-03-09 00:39
> I thought then and still do that listdir should (have) change (d) like range, map, and filter did, for same reasons.

The reasons that applied to map and range don't apply to listdir(). The 
cost of materializing the list in the former cases may be significant, 
but will be marginal in the latter case.
msg130414 - (view) Author: Vetoshkin Nikita (nvetoshkin) Date: 2011-03-09 03:52
Can't we simply add os.xlistdir() leaving listdir() as is?
msg130415 - (view) Author: Brian Curtin (brian.curtin) * (Python committer) Date: 2011-03-09 03:55
-1 on going back through blah/xblah all over again.
msg130426 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2011-03-09 05:39
> Can't we simply add os.xlistdir() leaving listdir() as is?

Only if an advantage can be demonstrated. in a realistic application.
msg130428 - (view) Author: Марк Коренберг (mmarkk) Date: 2011-03-09 05:43
> -1 on going back through blah/xblah all over again.

Originally, I want return value of listdir to be changed from list to generator. But next, I thought about compatibility. It will break some code. For example, SimpleHTTPServer:

list = os.listdir(path)
list.sort(key=lambda a: a.lower())

will not work.
msg130440 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-03-09 10:38
> Can't we simply add os.xlistdir() leaving listdir() as is?

We could, but someone must:
1) provide a patch
2) demonstrate a significant improvement in some real-world situation
msg130467 - (view) Author: Vetoshkin Nikita (nvetoshkin) Date: 2011-03-09 21:21
>We could, but someone must:
>1) provide a patch
While working on a straightforward patch for linux, I had to make a lot of copy-paste job. posixmodule.c is quite a mess already :(
>2) demonstrate a significant improvement in some real-world situation
suppose we have a directory with several millions of files and a cron script which must process just a bunch of them at a time. There's no need to gather them all.
As mmarkk mentioned - readdir already provides generator style access to the directory contents, it could be nice to provide such API in Python.

http://pastebin.com/NCGmfF49 - here's a kind of test (cached and uncached)
http://pastebin.com/tTKRTiNc - here's a testcase for batch processing of directory contenst (first is xlistdir(), second - listdir()) both uncached.
msg130485 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2011-03-10 03:14
> a cron script which must process just a bunch of them at a time.
 > There's no need to gather them all.

Can you please be more explicit? What's the application in which you
have several millions of files in a directory? What's the task that
the processing script needs to perform?

> http://pastebin.com/NCGmfF49 - here's a kind of test (cached and uncached)

This isn't really convincing - the test looks at all files, so it isn't 
clear why xlistdir should do any better than listdir. And indeed, with
a cold cache, xlistdir is slower (IIUC).

> http://pastebin.com/tTKRTiNc - here's a testcase for batch processing of directory contenst (first is xlistdir(), second - listdir()) both uncached.

This is not a real-world application - there is no actual processing done.

BTW, can you publish your xlistdir implementation somewhere?
msg130530 - (view) Author: Vetoshkin Nikita (nvetoshkin) Date: 2011-03-10 21:55
>BTW, can you publish your xlistdir implementation somewhere?
http://pastebin.com/Qnni5HBa

Tests show 10 times smaller memory footprint during directory listing - 25Mb against 286Mb on directory with 800K entries.
msg130545 - (view) Author: Марк Коренберг (mmarkk) Date: 2011-03-11 05:25
http://lwn.net/Articles/216948/
Why kernel.org is slow

To proove that readdir is bad thing on large number of items in a directory.

Well, EXT4 has fixed some issues (http://ext2.sourceforge.net/2005-ols/paper-html/node3.html) But what about locking in linux kernel (vfs and ext4) code?

Also, some conservative linuxes still use ext3.
msg130552 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2011-03-11 09:43
> http://lwn.net/Articles/216948/
> Why kernel.org is slow

That's all independent of the issue at hand. Whether or not getdents is
slow doesn't really matter here because we need to call getdents,
anyway: it's the only API to get at the directory contents, iterative
or not.

The issue at hand is whether xlistdir actually provides any advantages
to a real application, and that cannot be answered by looking at
benchmarking results that benchmarked the kernel. The *only* way to
determine whether xlistdir will help is to measure it in a real
application.

I stand by my claim that
a) in cases where you use listdir, you typically have to consider
    all file names in the directory eventually. The total number
    of getdents calls to be made doesn't change when you switch from
    listdir to xlistdir (except in non-realistic micro-benchmarks).

    The cases that you don't need to look at all file names are
    typically dealt with by knowing the file name in advance
    (or perhaps a few alternative spellings it may have), and
    hence you don't use listdir at all (but stat/open).

b) If there is some real-world processing of the files (e.g.
    at least to look at the file modification time), this processing
    (and the IO that goes along with it) by far outweigh the cost
    of reading the directory. So even if you could speed up listdir
    by making it iterative, the overall gain would be very small.

There are also good reasons *not* to add xlistdir, primarily to
avoid user confusion. If xlistdir was added, all peope would run
off and change all applications of listdir "because it is faster",
and have then to deal with backwards compatibility, even though
in most applications, a single getdents call will fetch the entire
directory contents just fine (and hence there is *no* change
in xlistdir, except that the list is not created which uses
a few dozen bytes).
msg130554 - (view) Author: Vetoshkin Nikita (nvetoshkin) Date: 2011-03-11 09:53
My benchmarks show that xlistdir() gives the only memory usage advantage on large directories. No speed gain so far - maybe my patch is wrong.
msg130713 - (view) Author: Torsten Landschoff (torsten) Date: 2011-03-12 23:44
I would regard this as Type: resource usage, instead of performance. Given enough RAM, loading the whole directory at once will likely be faster.

The downsides of os.listdir:
a) You can't get a peek at the files so far, it's all or nothing. I only wanted to know if a directory is empty and I have to read the whole thing just to throw it away (maybe I missed another library function?)

b) Using it in a GUI basically requires you to use threads if you may run into a dir with many files. Especially on a slow filesystem (network). Because you won't regain control until the whole thing is read.

I would like to have an iterator version as well, but I also dislike another function (especially the "x" prefix). How about adding a keyword argument to select iterator behaviour?
msg130714 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-03-12 23:47
> I would like to have an iterator version as well, but I also dislike
> another function (especially the "x" prefix). How about adding a
> keyword argument to select iterator behaviour?

Changing the return type based on an argument is generally frown upon
so, if anything, I think it would be better to have a separate function
(whatever its name).
msg130715 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2011-03-13 00:43
> The downsides of os.listdir: a) You can't get a peek at the files so
> far, it's all or nothing. I only wanted to know if a directory is
> empty and I have to read the whole thing just to throw it away (maybe
> I missed another library function?)

This depends somewhat on the operating system. On Unix, doing os.stat
on the directory, then looking on st_nlink, will tell you whether
the directory is empty (st_nlink is 2 on an empty directory).

> b) Using it in a GUI basically requires you to use threads if you may
> run into a dir with many files. Especially on a slow filesystem
> (network). Because you won't regain control until the whole thing is
> read.

Hmm. In a GUI, you would typically want to sort the file names by
some criterion, which typically requires you to read all files
(even if you then only display some of them).

> I would like to have an iterator version as well, but I also dislike
> another function (especially the "x" prefix). How about adding a
> keyword argument to select iterator behaviour?

I still would like to see a demonstrable improvement in a real-world
application.
msg155407 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2012-03-11 19:13
> On Unix, doing os.stat
> on the directory, then looking on st_nlink, will tell you whether
> the directory is empty (st_nlink is 2 on an empty directory).

Directory with st_nlink==2 can contains any number of non-directory files. And one subdirectory if this directory is root.
msg183610 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2013-03-06 20:07
I'd like to take care of this at Python.  At least for posix (someone else can deal with the windows side if they want).

I just stumbled upon an extension module at work that someone wrote specifically because os.listdir consumed way too much ram by building up a huge list on large directories with tons of long filenames that it needed to process.  (when your process is in charge of keeping that directory in check... the inability to process it once it grows too large simply isn't acceptable)
msg183611 - (view) Author: Tim Golden (tim.golden) * (Python committer) Date: 2013-03-06 20:11
IIRC Nick Coghlan had put a bit of work into this a few months ago as an 
external module with a view to seeing if it got traction before putting 
anything into the stdlib. Might be worth pinging him, or looking to see 
what he'd done. Can't remember the keywords to search for, I'm afraid. 
Something like "directory walker"
msg183612 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2013-03-06 20:16
Nick’s lib is called walkdir.  See bitbucket, readthedocs, possibly hg.python.org.

(FTR Antoine’s OO abstraction is pathlib)
msg183617 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2013-03-06 21:57
> IIRC Nick Coghlan had put a bit of work into this a few months ago as an
> external module with a view to seeing if it got traction before putting
> anything into the stdlib. Might be worth pinging him, or looking to see
> what he'd done. Can't remember the keywords to search for, I'm afraid.
> Something like "directory walker"

Nick's walkdir is "just" an improved walk - with a lot of added
functionality, like filtering etc.
But it's still based on os.walk(), which in turn uses listdir(), since
it's currently the only way to list the content of a directory. So in
short, it won't help for this issue.

To limit the memory usage, on would have to have to use an iterator
based on readdir().
msg183618 - (view) Author: Tim Golden (tim.golden) * (Python committer) Date: 2013-03-06 21:58
OK, sorry for the noise then; I had the idea that it was doing something 
with iterators/generators.
msg183620 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2013-03-06 22:27
right he has a separate issue open tracking the walkdir stuff in
issue13229.  I saw it first before finding this issue which is exactly what
I wanted. :)
msg185167 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2013-03-24 22:56
Here's is an os.scandir(path='.') implementation that iterates reading the directory on the fly instead of pre-building a list.

os.listdir's implementation should ultimately be replaced by this as:

def listdir(path=None):
    if path is None:
        return list(os.scandir())
    return list(os.scandir(path))

Though I have not yet done that in this patch so that I could compare behavior of old vs new.

Why the scandir name?  Read the libc scandir man page.  It fits.

I have tested this on POSIX (Linux).  I don't have any ability to build Windows code so I expect that still has bugs and possibly compilation issues.  Please leave comments on the 'review' link.
msg185168 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2013-03-24 23:04
Since this is going to be a new API, I would like to return the file type per directory entry where supported. I suggest to start with the Linux set of file types (DT_BLK, ..., DT_UNKNOWN), perhaps under different names, giving 'unknown' on systems which don't support this.

People traversing a directory tree can then skip the stat call if it's neither 'directory' nor 'unknown'.
msg185169 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2013-03-24 23:08
you'll see my code already has TODOs in there for that.  windows API documentation suggests that windows returns even more (stat-like) info when traversing a directory.  returning a namedtuple with the relevant info from the platform at hand would be good.

I'd prefer to iterate on the code and get this working as is first, then update it to support returning the full details, namedtuple or otherwise.  perhaps via an os.scandir vs os.scandirfull or something.  I'd like to keep the ability to return just the names: no need to spend time building up the tuples that'll be discarded by the os.listdir compatibility code.
msg185174 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2013-03-25 02:14
I don't this would be much of a win and we're better off not adding yet another function to the already overloaded os module.
msg185180 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2013-03-25 02:57
Your objection is noted but it is wrong.

A Python program today cannot process arbitrarily large directories within a fixed amount of ram today due to os.listdir. This makes it unsuitable for file system cleanup tasks that we have run into on production servers.  This fixes that without requiring an extension module or fragile ctypes code to work around the deficiency.

It _would've been nice_ for os.listdir to be updated to be an iterator with 3.0 but it wasn't... so we're left supporting its legacy interface rather than just replacing it with this.

If think this functionality belongs in a module other than os, please suggest where.

long term: os.walk and os.fwalk are also unusable on arbitrary filesystems with large directories for the same reason because they use os.listdir.  providing a non-directory-preloading version of those is outside the scope of this issue.
msg185673 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2013-03-31 21:08
Here's an updated patch that fixes the windows build based on twouters' comments. It goes ahead and removes the old C implementation of listdir in favor of the trivial Python wrapping of scandir.
msg185674 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-03-31 21:12
Bikeshedding: I would find "iterdir" much easier to remember than "scandir" (especially in relationship with "listdir").
msg185676 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-03-31 21:16
I prefer iterdir also.
msg185679 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2013-03-31 21:33
While i don't personally like things with iter in the name I won't object.  That way the scandir name would be left available for a future version of this that yields namedtuples of directory entry details as Martin wants to see.
msg185723 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2013-04-01 08:23
> That way the scandir name would be left available for a future version of this that yields namedtuples of directory entry details as Martin wants to see.

Which might very w'ell be Nick's walkdir, see issue #13229.

BTW, I'm strongly +1 on this addition.
msg188254 - (view) Author: Ben Hoyt (benhoyt) * Date: 2013-05-02 07:48
Ah, this is great. I definitely like the idea of a generator version of os.listdir(). And I like the name iterdir() -- it fits with iteritems() etc. They've gone in 3.x, of course, but listdir didn't change to an iterator, so...

See also Betterwalk, my work-in-progress with very similar goals: https://github.com/benhoyt/betterwalk#readme

It implements iterdir(), as well as iterdir_stat() which yields (name, stat) tuples. iterdir_stat() is especially important on Windows, where the directory iteration functions (FindFirstFile/FindNextFile) already give you full stat information.

The intent of Betterwalk is to use these functions to speed up os.walk() by 2-3 times (and that's with the ctypes version, so it'll only get better in pure C).

So I'm +1 for adding iterdir(), and I'd love to see iterdir_stat() so users can write fast os.walk() type functions without resorting to C.

I'll look over the attached patches at some stage, especially the Windows code.
msg188256 - (view) Author: Ben Hoyt (benhoyt) * Date: 2013-05-02 08:28
Some folks have asked about benchmarks. I don't know about iterdir() vs listdir() -- I kind of suspect the speed gains there wouldn't be big. 

However, the reason I'm keen on iterdir_stat() is that I'm seeing it speed up os.walk() by a factor of 10 in my recent tests (note that I've made local mods, so these results aren't reproducible for others yet). This is doing a walk on a dir tree with 7800 files and 155 dirs:

Using fast _betterwalk
Priming the system's cache...
Benchmarking walks on C:\Work\betterwalk\benchtree, repeat 1/3...
Benchmarking walks on C:\Work\betterwalk\benchtree, repeat 2/3...
Benchmarking walks on C:\Work\betterwalk\benchtree, repeat 3/3...
os.walk took 0.178s, BetterWalk took 0.017s -- 10.5x as fast

Sometimes Windows will go into this "I'm really caching stat results good" mode -- I don't know what heuristic determines this -- and then I'm seeing a 40x speed increase. And no, you didn't read that wrong. :-)

Sorry, I'm getting carried away. This bug is really more about iterdir. But seeing Martin suggested the stat/d_type info...
msg188260 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-05-02 09:48
I've had the local Red Hat release engineering team express their displeasure at having to stat every file in a network mounted directory tree for info that is present in the dirent structure, so a definite +1 to os.scandir from me, so long as it makes that info available.
msg188293 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2013-05-03 07:03
> However, the reason I'm keen on iterdir_stat() is that I'm seeing it speed up os.walk() by a factor of 10 in my recent tests (note that I've made local mods, so these results aren't reproducible for others yet). This is doing a walk on a dir tree with 7800 files and 155 dirs:
>
> Using fast _betterwalk
> Priming the system's cache...
> Benchmarking walks on C:\Work\betterwalk\benchtree, repeat 1/3...
> Benchmarking walks on C:\Work\betterwalk\benchtree, repeat 2/3...
> Benchmarking walks on C:\Work\betterwalk\benchtree, repeat 3/3...
> os.walk took 0.178s, BetterWalk took 0.017s -- 10.5x as fast
>
> Sometimes Windows will go into this "I'm really caching stat results good" mode -- I don't know what heuristic determines this -- and then I'm seeing a 40x speed increase. And no, you didn't read that wrong. :-)

I/O benchmarks shouldn't use timeit or repeated calls: after the first
run, most of your data is in cache, so subsequent runs are
meaningless.

I don't know about Windows, but on Linux you should do something like:
# echo 3 > /proc/sys/vm/drop_caches

to start out clean.
msg188298 - (view) Author: Ben Hoyt (benhoyt) * Date: 2013-05-03 10:06
Thanks. I thought about that -- but I think I *want* to benchmark it when
they're cached, so that we're comparing apples with apples, cached system
calls with cached systems calls. The benchmark would almost certainly be a
lot "better" (BetterWalk would be even faster) if I was comparing the
non-cached results. I'll think about it some more though.

Thoughts?

-Ben

On Fri, May 3, 2013 at 7:03 PM, Charles-François Natali <
report@bugs.python.org> wrote:

>
> Charles-François Natali added the comment:
>
> > However, the reason I'm keen on iterdir_stat() is that I'm seeing it
> speed up os.walk() by a factor of 10 in my recent tests (note that I've
> made local mods, so these results aren't reproducible for others yet). This
> is doing a walk on a dir tree with 7800 files and 155 dirs:
> >
> > Using fast _betterwalk
> > Priming the system's cache...
> > Benchmarking walks on C:\Work\betterwalk\benchtree, repeat 1/3...
> > Benchmarking walks on C:\Work\betterwalk\benchtree, repeat 2/3...
> > Benchmarking walks on C:\Work\betterwalk\benchtree, repeat 3/3...
> > os.walk took 0.178s, BetterWalk took 0.017s -- 10.5x as fast
> >
> > Sometimes Windows will go into this "I'm really caching stat results
> good" mode -- I don't know what heuristic determines this -- and then I'm
> seeing a 40x speed increase. And no, you didn't read that wrong. :-)
>
> I/O benchmarks shouldn't use timeit or repeated calls: after the first
> run, most of your data is in cache, so subsequent runs are
> meaningless.
>
> I don't know about Windows, but on Linux you should do something like:
> # echo 3 > /proc/sys/vm/drop_caches
>
> to start out clean.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue11406>
> _______________________________________
>
msg188403 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-05-04 20:05
+1 for iterdir. However, if we get a separate scandir() returning entries with attributes, is iterdir() still useful?
msg188425 - (view) Author: Ben Hoyt (benhoyt) * Date: 2013-05-05 03:50
That's right: if we have a separate scandir() that returns (name, stat) tuples, then a plain iterdir() is pretty much unnecessary -- callers just ignore the second stat value if they don't care about it.

I'd slightly prefer the name iterdir_stat(), as that almost makes the (name, stat) return values explicit in the name. But that's kind of bikeshedding -- scandir() works too.
msg188431 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2013-05-05 08:31
> I'd slightly prefer the name iterdir_stat(), as that almost makes the (name, stat) return values explicit in the name. But that's kind of bikeshedding -- scandir() works too.

I find iterdir_stat() ugly :-)
I like the scandir name, which has some precedent with POSIX.

> That's right: if we have a separate scandir() that returns (name, stat) tuples, then a plain iterdir() is pretty much unnecessary -- callers just ignore the second stat value if they don't care about it.

Hum, wait.
scandir() cannot return (name, stat), because on POSIX, readdir() only
returns d_name and d_type (the type of the entry): to return a stat,
we would have to call stat() on each entry, which would defeat the
performance gain.
And that's the problem with scandir: it's not portable. Depending on
the OS/file system, you could very well get DT_UNKNOWN (and on Linux,
since it uses an adaptive heuristic for NFS filesystem, you could have
some entries with a resolved d_type and some others with DT_UNKNOWN,
on the same directory stream).

That's why scandir would be a rather low-level call, whose main user
would be walkdir, which only needs to know the entry time and not the
whole stat result.

Also, I don't know which information is returned by the readdir
equivalent on Windows, but if we want a consistent API, we have to
somehow map d_type and Windows's returned type to a common type, like
DT_FILE, DT_DIRECTORY, etc (which could be an enum).

The other approach would be to return a dummy stat object with only
st_mode set, but that would be kind of a hack to return a dummy stat
result with only part of the attributes set (some people will get
bitten by this).
msg188432 - (view) Author: Ben Hoyt (benhoyt) * Date: 2013-05-05 08:53
> I find iterdir_stat() ugly :-) I like the scandir name, which has some precedent with POSIX.

Fair enough. I'm cool with scandir().

> scandir() cannot return (name, stat), because on POSIX, readdir() only returns d_name and d_type (the type of the entry): to return a stat, we would have to call stat() on each entry, which would defeat the performance gain.

Yes, you're right. I "solved" this in BetterWalk with the solution you propose of returning a stat_result object with the fields it could get "for free" set, and the others set to None.

So on Linux, you'd get a stat_result with only st_mode set (or None for DT_UNKNOWN), and all the other fields None. However -- st_mode is the one you're most likely to use, usually looking just for whether it's a file or directory. So calling code would look something like this:

files = []
dirs = []
for name, st in scandir(path):
    if st.st_mode is None:
        st = os.stat(os.path.join(path, name))
    if stat.S_ISDIR(st.st_mode):
        dirs.append(name)
    else:
        files.append(name)

Meaning you'd get the speed improvements 99% of the time (when st_mode) was set, but if st_mode is None, you can call stat and handle errors and whatnot yourself.

> That's why scandir would be a rather low-level call, whose main user would be walkdir, which only needs to know the entry time and not the whole stat result.

Agreed. This is in the OS module after all, and there's tons of stuff that's OS-dependent in there. However, I think that doing something like the above, we can make it usable and performant on both Linux and Windows for use cases like walking directory trees.

> Also, I don't know which information is returned by the readdir equivalent on Windows, but if we want a consistent API, we have to somehow map d_type and Windows's returned type to a common type, like DT_FILE, DT_DIRECTORY, etc (which could be an enum).

The Windows scan directory functions (FindFirstFile/FindNextFile) return a *full* stat (or at least, as much info as you get from a stat in Windows). We *could* map them to a common type -- but I'm suggesting that common type might as well be "stat_result with None meaning not present". That way users don't have to learn a completely new type.

> The other approach would be to return a dummy stat object with only st_mode set, but that would be kind of a hack to return a dummy stat result with only part of the attributes set (some people will get bitten by this).

We could document any platform-specific stuff, and places you'd users could get bitten. But can you give me an example of where the stat_result-with-st_mode-or-None approach falls over completely?
msg188433 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-05-05 08:59
I think os.scandir is a case where we *want* a low level call that exposes everything we can retrieve efficiently about the directory entries given the underlying platform - not everything written in Python is written to be portable, especially when it comes to scripts rather than applications (e.g. given where I work, I write a fair bit of code that is Fedora/RHEL specific, and if that code happens to work anywhere else it's just a bonus rather than being of any specific value to me).

This may mean that we just return an "info" object for each item, where the available info is explicitly platform specific. Agreed it can be an actual stat object, though.

os.walk then become the cross-platform abstraction built on top of the low level scandir call (splitting files from directories is probably about all we can do consistently cross-platform without per-entry stat calls).
msg188434 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2013-05-05 09:23
> We could document any platform-specific stuff, and places you'd users could get bitten. But can you give me an example of where the stat_result-with-st_mode-or-None approach falls over completely?

Well, that's easy:

size = 0
for name, st in scandir(path):
    if stat.S_ISREG(st.st_mode):
        size += st.st_size

> Agreed it can be an actual stat object, though.

Well, the nice thing is that we don't have to create yet another info
object, the downside is that it can be tricky, see above.

We can probably use the DTTOIF macro to convert d_type to st_mode.
msg188443 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-05-05 13:37
I really like scandir() -> (name: str, stat: stat structure using None for
unknown fields).

I expect that this API to optimize use cases like:

- glob.glob("*.jpg") in a big directory with few JPEG picture
- os.walk(".") in a directory with many files: should reduce the number of
stat() to zero on most platforms

But as usual, a benchmark on a real platform would be more convicing.

Filtering entries in os.listdir() or os.scandir() would be faster (than
filtering their output), but it hard to design an API to filter arbitrary
fields (name, file type, size, ...) especially because the underlying C
functions does not provide required information. A generator is closer to
Python design and more natural.
msg188468 - (view) Author: Ben Hoyt (benhoyt) * Date: 2013-05-05 21:43
Yeah, I very much agree with what Nick says -- we really want a way to expose what the platform provides. It's  less important (though still the ideal), to expose that in a platform-independent way. Today the only way to get access to opendir/readdir on Linux and FindFirst/Next on Windows is by using a bunch of messy (and slowish) ctypes code. And yes, os.walk() would be the main cross-platform abstraction built on top of this.

Charles gave this example of code that would fall over:

size = 0
for name, st in scandir(path):
    if stat.S_ISREG(st.st_mode):
        size += st.st_size

I don't see it, though. In this case you need both .st_mode and .st_size, so a caller would check that those are not None, like so:

size = 0
for name, st in scandir(path):
    if st.st_mode is None or st.st_size is None:
        st = os.stat(os.path.join(path, name))
    if stat.S_ISREG(st.st_mode):
        size += st.st_size

One line of extra code for the caller, but a big performance gain in most cases.

Stinner said, "But as usual, a benchmark on a real platform would be more convicing". Here's a start: https://github.com/benhoyt/betterwalk#benchmarks -- but it's not nearly as good as it gets yet, because those figures are still using the ctypes version. I've got a C version that's half-finished, and on Windows it makes os.walk() literally 10x the speed of the default version. Not sure about Linux/opendir/readdir yet, but I intend to do that too.
msg188478 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-05-05 23:16
size = 0
for name, st in scandir(path):
    if st.st_mode is None or st.st_size is None:
        st = os.stat(os.path.join(path, name))
    if stat.S_ISREG(st.st_mode):
        size += st.st_size

It would be safer to use dir_fd parameter when supported, but I don't
think that os.scandir() should care of this problem. An higher level
API like pathlib, walkdir & cie which should be able to reuse *at() C
functions using dir_fd parameter.
msg188492 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2013-05-06 06:51
> Charles gave this example of code that would fall over:
>
> size = 0
> for name, st in scandir(path):
>     if stat.S_ISREG(st.st_mode):
>         size += st.st_size
>
> I don't see it, though. In this case you need both .st_mode and .st_size, so a caller would check that those are not None, like so:

Well, that's precisely the point.
A normal "caller" would never expect a stat object to be partially
populated: if a function has a prototype returning a stat object, then
I definitely expect it to be a regular stat object, with all the
fields guaranteed by POSIX set (st_size, st_ino, st_dev...). By
returning a dummy stat object, you break the stat interface, and I'm
positive this *will* puzzle users and introduce errors.

Now, if I'm the only one who finds this trick dangerous and ugly, you
can go ahead, but I stand by my claim that it's definitely a bad idea
(between this and the explicit Enum value assignent, I feel somewhat
lost lately :-)
msg188493 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-05-06 06:54
> (between this and the explicit Enum value assignent, I feel somewhat
> lost lately :-)

Don't worry, it sometimes happens :-)
msg188495 - (view) Author: Ben Hoyt (benhoyt) * Date: 2013-05-06 09:08
> A normal "caller" would never expect a stat object to be partially populated: if a function has a prototype returning a stat object, then I definitely expect it to be a regular stat object, with all the fields guaranteed by POSIX set (st_size, st_ino, st_dev...).

I don't think that's true in general, or true of how other Python APIs work. For instance, many APIs return a "file-like object", and you can only do certain things on that object, depending on what the documentation says, or what EAFP gets you. Some file-like object don't support seek/tell, some don't support close, etc. I've seen plenty of walk-like-a-duck checks like this:

if hasattr(f, 'close'):
    f.close()

Anyway, my point boils down to:

* scandir() is a new function, so there aren't old trends or things that will break
* we clearly document it as returning a tuple of (name, st), where st is a "stat-like object" whose invididual fields are None if they couldn't be determined for free with the directory scanning
* in fact, that's kind of the point of the "st" object in this function, so the example could be the one I gave above where you call os.stat() if either of the fields you want is None
* if that's clear in the documentation (of this new function) and the first example shows you exactly how it's meant to be used, I think that's pretty sane and sensible...
msg188500 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2013-05-06 09:36
> I don't think that's true in general, or true of how other Python APIs work. For instance, many APIs return a "file-like object", and you can only do certain things on that object, depending on what the documentation says, or what EAFP gets you. Some file-like object don't support seek/tell, some don't support close, etc. I've seen plenty of walk-like-a-duck checks like this:

Yes, I'm fully aware duck-typing ;-)
But here, you're saying that "a duck has a beak, but it *may* have
legs, a tail, etc".
It's just looks wrong to me on so many levels.

Please bring this up on python-dev.
msg188566 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2013-05-06 18:07
Actually I'm thinking this duck may only have a beak. Instead of a bunch of
fields of None I'd prefer just not having that attribute defined on the
object. I consider the os specific "stat-like" info from reading a
directory to be so os specific that i'd rather not let someone be confused
by it if it were to be returned up to a higher level caller. It's not a
stat.
On May 6, 2013 2:36 AM, "Charles-François Natali" <report@bugs.python.org>
wrote:

>
> Charles-François Natali added the comment:
>
> > I don't think that's true in general, or true of how other Python APIs
> work. For instance, many APIs return a "file-like object", and you can only
> do certain things on that object, depending on what the documentation says,
> or what EAFP gets you. Some file-like object don't support seek/tell, some
> don't support close, etc. I've seen plenty of walk-like-a-duck checks like
> this:
>
> Yes, I'm fully aware duck-typing ;-)
> But here, you're saying that "a duck has a beak, but it *may* have
> legs, a tail, etc".
> It's just looks wrong to me on so many levels.
>
> Please bring this up on python-dev.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue11406>
> _______________________________________
>
msg188896 - (view) Author: Ben Hoyt (benhoyt) * Date: 2013-05-11 04:25
> Please bring this up on python-dev.

Good idea. Thread started: http://mail.python.org/pipermail/python-dev/2013-May/126119.html
msg200757 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2013-10-21 12:04
Gregory, did you make any progress on this?
I think it would be a nice addition.
msg200758 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-10-21 12:06
Indeed! I'd like to see the feature in 3.4 so I can remove my own hack from our code base.
msg200800 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2013-10-21 15:25
I haven't had a chance to look at this since May. It'd still be a great addition.
msg204083 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2013-11-23 19:50
For reference the current state of things for this is the proposal in:
 https://mail.python.org/pipermail/python-dev/2013-May/126196.html

With a prototype using a ctypes based implementation as proof of concept in https://github.com/benhoyt/scandir.

A combination of that interface plus my existing scandir patch (-gps02) could be created for the final implementation.

As 3.4beta1 happens tonight, this isn't going to make 3.4 so i'm bumping this to 3.5.  I really like the proposed design outlined above.
msg221618 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2014-06-26 16:34
I'm with Martin and the other respondents who think this shouldn't be done.

Without compelling timings, the smacks of feature creep.  The platform specific issues may create an on-going maintenance problem.  The feature itself is prone to misuse, leaving hard-to-find race condition bugs in its wake.
msg221634 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2014-06-26 19:17
Maybethe development should start outside Python stdlib, on a project on
PyPI.
msg221645 - (view) Author: Ben Hoyt (benhoyt) * Date: 2014-06-26 20:45
Raymond, there are very compelling timings/benchmarks for this -- not so much the original issue here (generator vs list, that's not really an issue) but having a scandir() function that returns the stat-like info from the OS so you don't need extra stat calls. This speeds up os.walk() by 7-20 times on Windows and 4-5 times on Linux. See more at: https://github.com/benhoyt/scandir#benchmarks

I've written a draft PEP that I've sent to the PEP editors (if you're interested, it's at https://github.com/benhoyt/scandir/blob/master/PEP.txt). If any of the PEP editors are listening here ... would love some feedback on that at some stage. :-)

Victor -- development has started outside the stdlib here: https://github.com/benhoyt/scandir and PyPI module here: https://pypi.python.org/pypi/scandir Both are being used by various people.
msg221647 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2014-06-26 21:16
"I've written a draft PEP that I've sent to the PEP editors (if you're interested, it's at https://github.com/benhoyt/scandir/blob/master/PEP.txt). If any of the PEP editors are listening here ... would love some feedback on that at some stage. :-)"

Oh you wrote a PEP? Great! I pushed it to the PEP repository. It should be online in a few hours:
http://legacy.python.org/dev/peps/pep-0471/

PEP editors are still useful if you want to get access directly the Mercurial repository to modify directly your PEP.

If you have a PEP, it's time to send it to the python-dev mailing list. Don't attach it to your mail, but copy PEP in the body of your email for easier inline comments in replies.
msg221648 - (view) Author: Ben Hoyt (benhoyt) * Date: 2014-06-26 21:16
Thanks! Will post the PEP to python-dev in the next day or two.
msg221659 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-06-26 23:05
I suggest a pass through python-ideas first. python-ideas feedback tends to
be more oriented towards "is this proposal as persuasive as it could be?",
while python-dev is more aimed at the "is this a good idea or not?" yes/no
question. (python-ideas feedback naturally includes some of the latter as
well, but there's a lot more "I'm not sure I agree with the idea itself,
but I agree it's worth discussing further" feedback than is common on
python-dev)
msg221661 - (view) Author: Ben Hoyt (benhoyt) * Date: 2014-06-26 23:11
Nick -- sorry, already posted to python-dev before seeing your latest. However, I think it's the right place, as there's already been a fair bit of hashing this idea and API out on python-ideas first and then also python-dev. See links in the PEP here: http://legacy.python.org/dev/peps/pep-0471/#previous-discussion
msg227879 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-09-30 11:15
I haven't really followed, but now that the PEP is accepted, what is the progress on this one?
msg227882 - (view) Author: Ben Hoyt (benhoyt) * Date: 2014-09-30 11:59
Yes, PEP 471 has been accepted, and I've got a mostly-finished C implementation of os.scandir() for CPython 3.5, as well as tests and docs. If you want a sneak preview, see posixmodule_scandir*.c, test/test_scandir.py, and os.rst here: https://github.com/benhoyt/scandir

It's working well on Windows, but the Linux version has a couple of tiny issues yet (core dumps ;-).

Given that os.scandir() will solve this issue (as well as the bigger performance problem due to listdir throwing away file type info), can we close this issue and open another one to track the implementation of os.scandir() / PEP 471?
msg227929 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-09-30 14:19
> Given that os.scandir() will solve this issue (as well as the bigger performance problem due to listdir throwing away file type info), can we close this issue and open another one to track the implementation of os.scandir() / PEP 471?

This makes sense. Can you do it?
msg227934 - (view) Author: Ben Hoyt (benhoyt) * Date: 2014-09-30 14:43
Okay, I've opened http://bugs.python.org/issue22524, but I don't have the permissions to close this one, so could someone with bugs.python.org superpowers please do that?
msg227936 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-09-30 14:47
This approach has been rejected in favour of the accepted PEP 471 proposal to add os.scandir() (issue #22524)
History
Date User Action Args
2014-09-30 14:47:15ncoghlansetstatus: open -> closed
superseder: PEP 471 implementation: os.scandir() directory scanning function
messages: + msg227936

resolution: rejected
stage: needs patch -> resolved
2014-09-30 14:43:33benhoytsetmessages: + msg227934
2014-09-30 14:19:08pitrousetmessages: + msg227929
2014-09-30 11:59:29benhoytsetmessages: + msg227882
2014-09-30 11:15:45pitrousetmessages: + msg227879
2014-06-26 23:11:12benhoytsetmessages: + msg221661
2014-06-26 23:05:33ncoghlansetmessages: + msg221659
2014-06-26 21:16:57benhoytsetmessages: + msg221648
2014-06-26 21:16:03hayposetmessages: + msg221647
2014-06-26 20:45:20benhoytsetmessages: + msg221645
2014-06-26 19:17:28hayposetmessages: + msg221634
2014-06-26 16:34:29rhettingersetmessages: + msg221618
2014-06-26 15:22:17nailorsetnosy: + nailor
2014-03-06 22:42:50josh.rsetnosy: + josh.r
2014-02-14 01:33:33ethan.furmansetnosy: + ethan.furman
2013-11-23 19:50:38gregory.p.smithsetstage: patch review -> needs patch
messages: + msg204083
versions: + Python 3.5, - Python 3.4
2013-10-21 15:25:26gregory.p.smithsetmessages: + msg200800
2013-10-21 12:06:49christian.heimessetmessages: + msg200758
2013-10-21 12:04:54neologixsetmessages: + msg200757
2013-10-13 06:47:06neologixlinkissue19240 dependencies
2013-05-11 15:06:42brian.curtinsetnosy: - brian.curtin
2013-05-11 12:17:49serhiy.storchakasetnosy: - serhiy.storchaka
2013-05-11 04:25:39benhoytsetmessages: + msg188896
2013-05-06 18:07:11gregory.p.smithsetmessages: + msg188566
2013-05-06 09:36:30neologixsetmessages: + msg188500
2013-05-06 09:08:56benhoytsetmessages: + msg188495
2013-05-06 06:54:43pitrousetmessages: + msg188493
2013-05-06 06:51:57neologixsetmessages: + msg188492
2013-05-05 23:16:39hayposetmessages: + msg188478
2013-05-05 21:43:22benhoytsetmessages: + msg188468
2013-05-05 13:37:23hayposetmessages: + msg188443
2013-05-05 09:23:46neologixsetmessages: + msg188434
2013-05-05 08:59:57ncoghlansetmessages: + msg188433
2013-05-05 08:53:16benhoytsetmessages: + msg188432
2013-05-05 08:31:03neologixsetmessages: + msg188431
2013-05-05 03:50:42benhoytsetmessages: + msg188425
2013-05-04 20:05:04pitrousetmessages: + msg188403
2013-05-03 19:02:20abacabadabacabasetnosy: + abacabadabacaba
2013-05-03 10:06:04benhoytsetmessages: + msg188298
2013-05-03 07:03:27neologixsetmessages: + msg188293
2013-05-02 10:06:40hayposetnosy: + haypo
2013-05-02 09:48:54ncoghlansetnosy: + ncoghlan
messages: + msg188260
2013-05-02 08:28:34benhoytsetmessages: + msg188256
2013-05-02 07:48:41benhoytsetnosy: + benhoyt
messages: + msg188254
2013-04-02 12:19:52christian.heimessetnosy: + christian.heimes
2013-04-01 08:23:41neologixsetmessages: + msg185723
2013-03-31 21:33:46gregory.p.smithsetmessages: + msg185679
2013-03-31 21:16:32terry.reedysetmessages: + msg185676
2013-03-31 21:12:41pitrousetmessages: + msg185674
2013-03-31 21:09:02gregory.p.smithsetfiles: + issue11406-gps02.diff

messages: + msg185673
2013-03-29 01:19:00gregory.p.smithsetnosy: + twouters
2013-03-25 02:57:25gregory.p.smithsetmessages: + msg185180
2013-03-25 02:14:44rhettingersetnosy: + rhettinger
messages: + msg185174
2013-03-24 23:08:22gregory.p.smithsetmessages: + msg185169
2013-03-24 23:04:13loewissetmessages: + msg185168
2013-03-24 22:56:27gregory.p.smithsetfiles: + issue11406-gps01.diff
messages: + msg185167

keywords: + patch
type: performance -> enhancement
stage: patch review
2013-03-06 22:27:46gregory.p.smithsetmessages: + msg183620
2013-03-06 21:58:28tim.goldensetmessages: + msg183618
2013-03-06 21:57:45neologixsetmessages: + msg183617
2013-03-06 20:16:56eric.araujosetmessages: + msg183612
2013-03-06 20:11:32tim.goldensetnosy: + tim.golden
messages: + msg183611
2013-03-06 20:07:47gregory.p.smithsetversions: + Python 3.4, - Python 3.3
nosy: + gregory.p.smith

messages: + msg183610

assignee: gregory.p.smith
2012-03-11 19:13:51serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg155407
2011-03-13 00:43:13loewissetnosy: loewis, terry.reedy, pitrou, giampaolo.rodola, eric.araujo, Trundle, brian.curtin, torsten, nvetoshkin, neologix, mmarkk
messages: + msg130715
2011-03-12 23:47:23pitrousetnosy: loewis, terry.reedy, pitrou, giampaolo.rodola, eric.araujo, Trundle, brian.curtin, torsten, nvetoshkin, neologix, mmarkk
messages: + msg130714
2011-03-12 23:44:59torstensetnosy: + torsten
messages: + msg130713
2011-03-11 21:22:22eric.araujosetnosy: + eric.araujo
2011-03-11 09:53:45nvetoshkinsetnosy: loewis, terry.reedy, pitrou, giampaolo.rodola, Trundle, brian.curtin, nvetoshkin, neologix, mmarkk
messages: + msg130554
2011-03-11 09:43:01loewissetnosy: loewis, terry.reedy, pitrou, giampaolo.rodola, Trundle, brian.curtin, nvetoshkin, neologix, mmarkk
messages: + msg130552
2011-03-11 05:25:45mmarkksetnosy: loewis, terry.reedy, pitrou, giampaolo.rodola, Trundle, brian.curtin, nvetoshkin, neologix, mmarkk
messages: + msg130545
2011-03-10 21:55:00nvetoshkinsetnosy: loewis, terry.reedy, pitrou, giampaolo.rodola, Trundle, brian.curtin, nvetoshkin, neologix, mmarkk
messages: + msg130530
2011-03-10 03:14:11loewissetnosy: loewis, terry.reedy, pitrou, giampaolo.rodola, Trundle, brian.curtin, nvetoshkin, neologix, mmarkk
messages: + msg130485
2011-03-09 21:21:25nvetoshkinsetnosy: loewis, terry.reedy, pitrou, giampaolo.rodola, Trundle, brian.curtin, nvetoshkin, neologix, mmarkk
messages: + msg130467
2011-03-09 10:38:00pitrousetnosy: loewis, terry.reedy, pitrou, giampaolo.rodola, Trundle, brian.curtin, nvetoshkin, neologix, mmarkk
messages: + msg130440
2011-03-09 05:43:13mmarkksetnosy: loewis, terry.reedy, pitrou, giampaolo.rodola, Trundle, brian.curtin, nvetoshkin, neologix, mmarkk
messages: + msg130428
2011-03-09 05:39:40loewissetnosy: loewis, terry.reedy, pitrou, giampaolo.rodola, Trundle, brian.curtin, nvetoshkin, neologix, mmarkk
messages: + msg130426
2011-03-09 03:55:08brian.curtinsetnosy: + brian.curtin
messages: + msg130415
2011-03-09 03:52:06nvetoshkinsetnosy: loewis, terry.reedy, pitrou, giampaolo.rodola, Trundle, nvetoshkin, neologix, mmarkk
messages: + msg130414
2011-03-09 00:39:42loewissetnosy: loewis, terry.reedy, pitrou, giampaolo.rodola, Trundle, nvetoshkin, neologix, mmarkk
messages: + msg130392
2011-03-09 00:31:43terry.reedysetnosy: + terry.reedy
messages: + msg130391
2011-03-07 19:43:54mmarkksetnosy: loewis, pitrou, giampaolo.rodola, Trundle, nvetoshkin, neologix, mmarkk
messages: + msg130286
2011-03-07 13:57:52nvetoshkinsetnosy: loewis, pitrou, giampaolo.rodola, Trundle, nvetoshkin, neologix, mmarkk
messages: + msg130256
2011-03-06 12:01:34nvetoshkinsetnosy: + nvetoshkin
messages: + msg130167
2011-03-05 18:48:35neologixsetnosy: loewis, pitrou, giampaolo.rodola, Trundle, neologix, mmarkk
messages: + msg130129
2011-03-05 18:31:24giampaolo.rodolasetnosy: + giampaolo.rodola
2011-03-05 17:11:38pitrousetnosy: + pitrou, neologix
messages: + msg130125
2011-03-05 14:33:43Trundlesetnosy: + Trundle
2011-03-05 11:02:58mmarkksetmessages: + msg130114
2011-03-05 10:45:23loewissetnosy: + loewis
messages: + msg130112
2011-03-05 10:17:58mmarkkcreate