At Antoine's behest, I tried running

  ./python.exe -m test -j2

in my cpython sandbox and saw several test case failures which
didn't appear when I executed a simple

    make test

He suggested I add the -W flag and try again.  I did, but I don't
see any difference in the output, so either I'm still doing something
wrong or the -W flag is somehow mishandled.  As far as I can tell the
failing tests were not re-run in verbose mode.  script command output
attached.

I ran these tests on Mac OS X Leopard (10.5.8).

S

It should have been '-w', not '-W'.

Thanks.  That worked better.  Here is the tail end of the output
showing the verbose test output.

Brett, -W exists too and it seems it failed working here.

Skip, can you please try "make distclean" and then rebuild from scratch?

I think the pydoc failure is due to haypo’s patch in #3080.  Another patch (by me) in #8754 has a fix.

Antoine> Skip, can you please try "make distclean" and then rebuild from
    Antoine> scratch?

I did.  The last output was after

    make distclean
    ./configure ... my usual suspects ...
    make
    ./python.exe -j2 -w

I will try one more time later this evening with a capital "W".  I have a
train to catch first.

Skip

> Antoine> Skip, can you please try "make distclean" and then rebuild from
>     Antoine> scratch?
> 
> I did.  The last output was after

Oops, sorry.

> I will try one more time later this evening with a capital "W".  I have a
> train to catch first.

Well, if "-W" didn't work once, it probably won't work twice. Not sure
you want to bother.

New changeset 9aa6097131ef by Antoine Pitrou in branch '3.2':
Issue #11653: fix -W with -j in regrtest
http://hg.python.org/cpython/rev/9aa6097131ef

New changeset c381b35e4f31 by Antoine Pitrou in branch 'default':
Issue #11653: fix -W with -j in regrtest.
http://hg.python.org/cpython/rev/c381b35e4f31

Antoine> New changeset 9aa6097131ef by Antoine Pitrou in branch '3.2':
    Antoine> Issue #11653: fix -W with -j in regrtest
    Antoine> http://hg.python.org/cpython/rev/9aa6097131ef

    Antoine> New changeset c381b35e4f31 by Antoine Pitrou in branch 'default':
    Antoine> Issue #11653: fix -W with -j in regrtest.
    Antoine> http://hg.python.org/cpython/rev/c381b35e4f31

Saw these checkins and tried again.  I definitely see different output and
it seems to rerun failed tests immediately in verbose mode.  For example:

    [238/352] test_pydoc
    Re-running test test_pydoc in verbose mode
    test_badimport (test.test_pydoc.PydocDocTest) ... FAIL
    test_help_output_redirect (test.test_pydoc.PydocDocTest) ... ok
    test_html_doc (test.test_pydoc.PydocDocTest) ... ok
    test_input_strip (test.test_pydoc.PydocDocTest) ... ok
    test_issue8225 (test.test_pydoc.PydocDocTest) ... ok
    test_not_here (test.test_pydoc.PydocDocTest) ... ok
    test_stripid (test.test_pydoc.PydocDocTest) ... ok
    test_text_doc (test.test_pydoc.PydocDocTest) ... ok
    test_class (test.test_pydoc.TestDescriptions) ... ok
    test_module (test.test_pydoc.TestDescriptions) ... ok
    test_server (test.test_pydoc.PydocServerTest) ... ok
    test_content_type_err (test.test_pydoc.PydocUrlHandlerTest) ... ok
    test_url_requests (test.test_pydoc.PydocUrlHandlerTest) ... ok

However, I also see lots of tests crashing in imp.  Here's one example:

    test test_ttk_textonly crashed -- Traceback (most recent call last):
      File "/Users/skip/src/hgpython/cpython/Lib/test/regrtest.py", line 988, in runtest_inner
        the_package = __import__(abstest, globals(), locals(), [])
      File "/Users/skip/src/hgpython/cpython/Lib/test/test_ttk_textonly.py", line 7, in <module>
        from tkinter.test import runtktests
      File "/Users/skip/src/hgpython/cpython/Lib/importlib/_bootstrap.py", line 437, in load_module
        return self._load_module(fullname)
      File "/Users/skip/src/hgpython/cpython/Lib/importlib/_bootstrap.py", line 141, in decorated
        return fxn(self, module, *args, **kwargs)
      File "/Users/skip/src/hgpython/cpython/Lib/importlib/_bootstrap.py", line 330, in _load_module
        code_object = self.get_code(name)
      File "/Users/skip/src/hgpython/cpython/Lib/importlib/_bootstrap.py", line 407, in get_code
        imp._fix_co_filename(found, source_path)
    AttributeError: 'module' object has no attribute '_fix_co_filename'

Skip

The import failures are because you didn't rebuild (I added a new function to imp). Re-build and the import issues will go away.

Brett> The import failures are because you didn't rebuild (I added a new
    Brett> function to imp). Re-build and the import issues will go away.

An easy mistake to make.  I'm used to running "make test" which would have
guaranteed python.exe was up-to-date.

I just executed

    hg pull -u
    make
    ./python.exe -m test -j2 -W

and I am seeing the same AttributeError exceptions.  So:

    make distclean
    ./configure ...
    make
    ./python.exe -m test -j2 -W

and I still get the AttributeError exceptions.  According to hg id I am
using this:

    changeset:   68889:f09f7ab40ce6
    tag:         tip
    user:        Raymond Hettinger <python@rcn.com>
    date:        Wed Mar 23 20:33:30 2011 -0700
    summary:     Add tests for _source to importable and exec'able.

Is that what I should be using?

Skip

Skip, what parameters are you using with ./configure ?

You have a build problem, Skip. On my system and at least all of the stable buildbots for 3.3 there are no test failures from imp._fix_co_filename().

Ned> Skip, what parameters are you using with ./configure ?

Pretty vanilla.  Install in my directory tree, get libraries from MacPorts:

  --prefix=/Users/skip/local --enable-shared LDFLAGS=-L/opt/local/lib CPPFLAGS=-I/opt/local/include

I thought you had fixed the --enable-shared linkage problems, but taking it
out seems to solve this issue.  I'm guessing nothing has been checked in yet
to solve that problem.

S

> Pretty vanilla.  Install in my directory tree, get libraries from MacPorts:
> 
>   --prefix=/Users/skip/local --enable-shared LDFLAGS=-L/opt/local/lib CPPFLAGS=-I/opt/local/include
> 
> I thought you had fixed the --enable-shared linkage problems, but taking it
> out seems to solve this issue.  I'm guessing nothing has been checked in yet
> to solve that problem.

It's not --enable-shared, it's that you have to set "LD_LIBRARY_PATH=.",
otherwise your new "./python.exe" will use some other Python 3.3 shared
library installed on your system. So you're basically testing an old
Python with the latest test suite :)

We do have a buildbot testing shared builds btw:
http://www.python.org/dev/buildbot/all/buildslaves/bolen-ubuntu

Antoine is correct in principle, however LD_LIBRARY_PATH does not work on OS X.  To test without installing, you would need to use DYLD_LIBRARY_PATH instead.  Also keep in mind that, unlike most other systems, OS X binaries embed absolute paths to dynamic libraries.

As a learning exercise for myself, I constructed this example.  I added a printf to the errno module, a module that is included in the shared library, to help see what's happening:

$ cd /tmp/a/py3k
$ ./configure --prefix=/tmp/a/local --enable-shared LDFLAGS=-L/opt/local/lib CPPFLAGS=-I/opt/local/include && make
$ file ./libpython3.3m.dylib 
./libpython3.3m.dylib: Mach-O 64-bit dynamically linked shared library x86_64
$ otool -L ./python
./python:
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 550.42.0)
	/tmp/a/local/lib/libpython3.3m.dylib (compatibility version 3.3.0, current version 3.3.0)
[...]
# note, "make install" not yet run so lib not available in configured path
$ ls /tmp/a/local/lib/libpython3.3m.dylib
ls: /tmp/a/local/lib/libpython3.3m.dylib: No such file or directory
$ ./python -c 'pass'
dyld: Library not loaded: /tmp/a/local/lib/libpython3.3m.dylib
  Referenced from: /private/tmp/a/py3k/./python
  Reason: image not found
Trace/BPT trap
# override path so that uninstalled shared lib is found in build dir
$ DYLD_LIBRARY_PATH=. ./python -c 'pass'
errno module version 0
$ make install
$ ls /tmp/a/local/lib/libpython3.3m.dylib
/tmp/a/local/lib/libpython3.3m.dylib*
# now lib is installed and found
$ ./python -c 'pass'
errno module version 0
# edit message in errno module to "version 1"
$ make
# have not done a new install so old shared lib is found
$ ./python -c 'pass'
errno module version 0
# enable dynamic loader printing of loads
$ export DYLD_PRINT_LIBRARIES=1
$ ./python -c 'pass'
dyld: loaded: /private/tmp/a/py3k/./python
dyld: loaded: /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
dyld: loaded: /tmp/a/local/lib/libpython3.3m.dylib
[...]
errno module version 0
# override loads from build directory -> works correctly
$ DYLD_LIBRARY_PATH=. ./python -c 'pass'
dyld: loaded: /private/tmp/a/py3k/./python
dyld: loaded: /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
dyld: loaded: /private/tmp/a/py3k/libpython3.3m.dylib
errno module version 1
# install updated lib
$ make install
# also works correctly
$ ./python -c 'pass'
dyld: loaded: /private/tmp/a/py3k/./python
dyld: loaded: /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
dyld: loaded: /tmp/a/local/lib/libpython3.3m.dylib
[...]
errno module version 1

So the moral of the story is: if you use --enable-shared, you need to be careful to either "make install" or to use DYLD_LIBRARY_PATH=/path/to/build_directory.

Skip, are you OK with closing this issue?