Trace:
----------------------------------------------------------------------
[ 30/354] test_compileall
test test_compileall failed -- Traceback (most recent call last):
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/test_compileall.py", line 253, in test_regexp
    self.assertCompiled(self.initfn)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/test_compileall.py", line 149, in assertCompiled
    self.assertTrue(os.path.exists(imp.cache_from_source(fn)))
AssertionError: False is not true
----------------------------------------------------------------------
http://www.python.org/dev/buildbot/all/builders/x86%20OpenIndiana%203.x/builds/1097/steps/test/logs/stdio

It failed also on "AMD64 FreeBSD 8.2 3.x" buildbot:
-----------------
test test_compileall failed -- Traceback (most recent call last):
  File "/usr/home/buildbot/buildarea/3.x.krah-freebsd/build/Lib/test/test_compileall.py", line 253, in test_regexp
    self.assertCompiled(self.initfn)
  File "/usr/home/buildbot/buildarea/3.x.krah-freebsd/build/Lib/test/test_compileall.py", line 149, in assertCompiled
    self.assertTrue(os.path.exists(imp.cache_from_source(fn)))
AssertionError: False is not true
-----------------
http://www.python.org/dev/buildbot/all/builders/AMD64%20FreeBSD%208.2%203.x/builds/102/steps/test/logs/stdio

How do we debug this?  Does someone have access to a similar box to see whether the pyc files do get created and where?

Given that it happens randomly I suspect a timing issue, but without having reviewed the code in question.  I'm not sure when I'll have time to look at this.

The failing test is launching a subprocess to compile python code, waiting for the subprocess to exit, and then checking to see if the file was created.  So the timing issue would appear to be that the file created by the subprocess doesn't appear to exist to the main program even after the subprocess has exited.  Stefan, Jesus, do either of you know of anything about the file systems used on your buildbots that might give rise to this problem?  Is there some sort of file system sync call we need to make to make sure different processes have the same view of the file system?

However, if my theory is true I would expect that other tests in test_compileall would also fail randomly.  Does anyone know if it is always test_regexp?

R. David Murray <report@bugs.python.org> wrote:
> The failing test is launching a subprocess to compile python code, waiting
> for the subprocess to exit, and then checking to see if the file was
> created.  So the timing issue would appear to be that the file created by
> the subprocess doesn't appear to exist to the main program even after the
> subprocess has exited.  Stefan, Jesus, do either of you know of anything
> about the file systems used on your buildbots that might give rise to this
> problem?  Is there some sort of file system sync call we need to make to
> make sure different processes have the same view of the file system?

The FreeBSD bot has a UFS filesystem with no soft updates:

$ mount
/dev/ad0s1a on / (ufs, local)

If the main process waits for the subprocess to finish, I can't think of
a reason to do any kind of synchronization.

Same here. I am using a standard ZFS configuration. If the child finish doing its job, the file should be there. Are we sure no errors happens in the child?.

Running test_compileall with -F it failed on iteration 588 on my linux box.  So this isn't platform specific.  No idea why that particular test should be fragile, but I'll look in to it when I get some time if nobody beats me to it.

A quick eyeball of the test doesn't reveal anything obvious:

- if the subprocess is failing, it is doing so in such a way that it is still returning a 0 error code and isn't writing anything to stdout (as those assertions are before the check that the expected compiled file exists)
- the invocation of script_helper looks sensible
- if the directory creation fails, it won't get as far as running the subprocess

To instrument this, I would suggest starting by adding an assertion message on the failing assertion that includes "fn". My guess is that with the random directory name in there, the regex is triggering on the directory name and hence excluding the __init__.py file as well.

Hmm, I probably should have revised the first half of my comment after I had the thought about "simple regex" + "random directory name" being potentially problematic. I don't think that theory really counts as obvious, though...

Ah ah, this bug is funny :-) test_regexp doesn't fail because of a race condition or a file system issue, but because of the regex!

The regex "ba.*" is applied on the fullname, not only on the basename. If the temporary contains "ba", the test fails :-) Replace tempfile.mkdtemp() by tempfile.mkdtemp(prefix='bar') in CommandLineTests.setUp() to always reproduce this bug.

New changeset 76bd26565cc7 by R David Murray in branch '3.2':
#11873: Improve test regex so random directory names don't cause test to fail
http://hg.python.org/cpython/rev/76bd26565cc7

New changeset 83e069281810 by R David Murray in branch 'default':
Merge: #11873: Improve test regex so random directory names don't cause test to fail
http://hg.python.org/cpython/rev/83e069281810

I ran the patched version 2945 times without failure.

+        self.assertRunOK('-q', '-x', 'ba[^\/]*$', self.pkgdir)

This regex matchs 'ba(...)\'. I think that you wanted to write r'[^\\/]'.

You are right that I got the regex wrong (I'm bad at regexes), but in fact it is fine the way it is.  Since there's no r, the regex is actually "ba[^/]$", which is exactly what I meant.

> You are right that I got the regex wrong (I'm bad at regexes), but
> in fact it is fine the way it is.  Since there's no r, the regex
> is actually "ba[^/]$", which is exactly what I meant.

What about Windows? tempfile.mkdtemp(prefix='bar') can generate ...\tmpxxbaxx\... Or compileall does first normalize the path?

> You are right that I got the regex wrong (I'm bad at regexes), but in
> fact it is fine the way it is.  Since there's no r, the regex is
> actually "ba[^/]$", which is exactly what I meant.

I would edit the string to reflect your meaning, then, otherwise other people may think too that the r was forgotten.

> What about Windows? tempfile.mkdtemp(prefix='bar') can generate
> ...\tmpxxbaxx\... Or compileall does first normalize the path?

@r.david.murray: You reopened the issue, but you didn't answer to this question. And, is there a bug in your regex or not?

I reopened the issue because I don't know the answer to your question but suspect that it is an issue.  So it needs to be checked.  And the regex should be cleaned up regardless, for clarity's sake.

> What about Windows? tempfile.mkdtemp(prefix='bar') can generate
> ...\tmpxxbaxx\...

AH! The test failed on "x86 XP-4 x86":

http://www.python.org/dev/buildbot/all/builders/x86%20XP-4%203.x/builds/4874/steps/test/logs/stdio

======================================================================
FAIL: test_regexp (test.test_compileall.CommandLineTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\test\test_compileall.py", line 253, in test_regexp
    self.assertCompiled(self.initfn)
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\test\test_compileall.py", line 149, in assertCompiled
    self.assertTrue(os.path.exists(imp.cache_from_source(fn)))
AssertionError: False is not true

@r.david.murray: would you like to fix the regex?

New changeset f8ece8c93918 by R David Murray in branch '3.2':
#11873: fix test regex so it covers windows os.sep as well.
http://hg.python.org/cpython/rev/f8ece8c93918

New changeset e543c0ddec63 by R David Murray in branch 'default':
merge #11873: fix test regex so it covers windows os.sep as well.
http://hg.python.org/cpython/rev/e543c0ddec63

Yeah, that's why I had reopened the issue...hopefully it is fixed now.

I guess I'm just really bad at regexes.

http://www.python.org/dev/buildbot/all/builders/x86%20XP-4%203.x/builds/4885/steps/test/logs/stdio

New changeset 6fdb1f000d36 by R David Murray in branch '3.2':
#11873: another try at fixing the regex, courtesy of Victor Stinner
http://hg.python.org/cpython/rev/6fdb1f000d36

New changeset 775356b583d1 by R David Murray in branch 'default':
merge #11873: another try at fixing the regex, courtesy of Victor Stinner
http://hg.python.org/cpython/rev/775356b583d1

I'm going to be an optimist and close this again.