Author tim.peters
Recipients
Date 2006-02-07.04:07:03
SpamBayes Score
Marked as misclassified
Message-id
In-reply-to
Content
Logged In: YES 
user_id=31435

[/F]
> Except that he's hitting the file system quite heavily,

Except that _without_ the call to touch(), he's hitting it
even more heavily, creating and destroying little files just
as fast as the OS can do it in each of 10 threads -- but
there aren't any errors then.

> and asyncronously.

What's asynch here?  The OP's touch() function waits for the
spawned process to terminate, and the test driver doesn't
try to delete the file until after that.

> My guess is that Windows simply gets behind
> (a quick filemon test indicates that this is indeed the
> case; just before a crash, I see the events
> CREATE/SUCCESS, QUERY/SUCCESS, QUERY/SUCCESS,
> WRITE/SUCCESS, and OPEN/SHARING VIOLATION for the
> failing file, with lots of requests for other files
> interleaved).

That's consistent with the symptom reported:  an exception
raised upon trying to remove the file, but not during any
other file operation.  Does it tell you more than _just_
that?  It doesn't for me.

> Unless someone wants to fix Windows,

As above, because removing the call to the internal `touch`
function makes all problems go away it's not obvious that
this is a Windows problem.

> a simple workaround would be to retry the os.remove a
> few times before giving up (with a time.sleep(0.1) in
> between).

Because of the internal threading errors in subprocess.py
(see my first comment), the threads in the test program
still usually die, but with instances of list.remove(x)
ValueErrors internal to subprocess.py.

If I hack around that, then this change to the test
program's file-removal code appears adequate to eliminate
all errors on my box (which is a zippy 3.4 GHz):

            try:
                os.remove(filename)
            except OSError:
                time.sleep(0.1)
                os.remove(filename)

It's possible that some virus-scanning or file-indexing
gimmick on my box is opening these little files for its own
purposes -- although, if so, I'm at a loss to account for
why a single "os.remove(filename)" never raises an exception
when the `touch()` call is commented out.

OTOH, with the `touch()` call intact, the time.sleep(0.1)
above is not adequate to prevent os.remove() errors if I
change the file-writing code to:

    f.write("test" * 250000)

Even boosting the sleep() to 0.4 isn't enough then.

That does (mildly) suggest there's another process opening
the temp files, and doing something with them that takes
time proportional to the file size.  However, the
os.remove() errors persist when I disable all such gimmicks
(that I know about ;-)) on my box.

It seems likely I'll never determine a cause for that.  The
bad thread behavior in subprocess.py is independent, and
should be repaired regardless.
History
Date User Action Args
2007-08-23 14:37:41adminlinkissue1425127 messages
2007-08-23 14:37:41admincreate