unittest.loader has the following snippet:

                    if realpath.lower() != fullpath_noext.lower():
                        module_dir = os.path.dirname(realpath)
                        mod_name = os.path.splitext(os.path.basename(full_path))[0]
                        expected_dir = os.path.dirname(full_path)
                        msg = ("%r module incorrectly imported from %r. Expected %r. "
                               "Is this module globally installed?")


Unfortunately, this will break with virtualenv on Ubuntu, which creates
a "local" directory full of symlinks. You end with this kind of error message:

======================================================================
ERROR: __main__ (unittest.loader.LoadTestsFailure)
----------------------------------------------------------------------
ImportError: 'test_asyncagi' module incorrectly imported from '/home/antoine/obelus/.tox/py27/local/lib/python2.7/site-packages/obelus/test'. Expected '/home/antoine/obelus/.tox/py27/lib/python2.7/site-packages/obelus/test'. Is this module globally installed?


Instead of (rather stupidly) calling lower(), realpath() and normcase()
should be called instead, to make sure the canonical paths are compared.

Attaching patch.

Good catch and fix Antoine. Thanks.

New changeset d7ec961cea1c by Antoine Pitrou in branch '2.7':
Issue #19352: Fix unittest discovery when a module can be reached through several paths (e.g. under Debian/Ubuntu with virtualenv).
http://hg.python.org/cpython/rev/d7ec961cea1c

New changeset a830cc1c0565 by Antoine Pitrou in branch '3.3':
Issue #19352: Fix unittest discovery when a module can be reached through several paths (e.g. under Debian/Ubuntu with virtualenv).
http://hg.python.org/cpython/rev/a830cc1c0565

New changeset ebbe87204114 by Antoine Pitrou in branch 'default':
Issue #19352: Fix unittest discovery when a module can be reached through several paths (e.g. under Debian/Ubuntu with virtualenv).
http://hg.python.org/cpython/rev/ebbe87204114

Fixed!

re-opening. the patch did break autopilot running with 2.7 on Ubuntu. I don't yet understand what this patch is supposed to fix.

see https://launchpad.net/bugs/1255505

More precisely, it broke unittest's discovery (not specific to autopilot). For any installed test, you now get:

$ python -m unittest discover lazr
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/lib/python2.7/unittest/__main__.py", line 12, in <module>
    main(module=None)
  File "/usr/lib/python2.7/unittest/main.py", line 94, in __init__
    self.parseArgs(argv)
  File "/usr/lib/python2.7/unittest/main.py", line 113, in parseArgs
    self._do_discovery(argv[2:])
  File "/usr/lib/python2.7/unittest/main.py", line 214, in _do_discovery
    self.test = loader.discover(start_dir, pattern, top_level_dir)
  File "/usr/lib/python2.7/unittest/loader.py", line 206, in discover
    tests = list(self._find_tests(start_dir, pattern))
  File "/usr/lib/python2.7/unittest/loader.py", line 287, in _find_tests
    for test in self._find_tests(full_path, pattern):
  File "/usr/lib/python2.7/unittest/loader.py", line 287, in _find_tests
    for test in self._find_tests(full_path, pattern):
  File "/usr/lib/python2.7/unittest/loader.py", line 267, in _find_tests
    raise ImportError(msg % (mod_name, module_dir, expected_dir))
ImportError: 'test_error' module incorrectly imported from '/usr/lib/python2.7/dist-packages/lazr/restfulclient/tests'. Expected '/usr/lib/python2.7/dist-packages/lazr/restfulclient/tests'. Is this module globally installed?

I reverted this patch in Ubuntu for now as a quickfix.

This might be specific how Debian installs Python 2 modules. Packages ship them in /usr/share/pyshared/<modulename>/, and for every supported 2.X version, ship /usr/lib/python2.X/dist-packages/<modulename>/ which contains only the directories and *.pyc files, but symlinks the *.py files to /usr/share/pyshared/<modulename>/../*.py

So, it might be that upstream does not support this symlink layout, but it's the best thing to avoid having to install multiple *.py copies (this is all solved in a much more elegant way with Python 3, of course). But it would be nice if unittest's discovery could still cope with this.

Thanks!

> ImportError: 'test_error' module incorrectly imported from '/usr/lib/python2.7/dist-packages/lazr/restfulclient/tests'. Expected '/usr/lib/python2.7/dist-packages/lazr/restfulclient/tests'. Is this module globally installed?

Well, this looks like the same path to me. Can you investigate a bit?

This can happen when the code used to compare the paths is different from the code used to generate the failure message. This of course masks the real problem. I can look at where that might be possible and then hopefully we can get a genuine error message telling us the problem.

In this new code:

                    mod_file = os.path.abspath(getattr(module, '__file__', full_path))
                    realpath = os.path.splitext(os.path.realpath(mod_file))[0]
                    fullpath_noext = os.path.splitext(os.path.realpath(full_path))[0]

we get

modfile == /usr/lib/python2.7/dist-packages/lazr/restfulclient/tests/test_error.pyc
realpath == /usr/lib/python2.7/dist-packages/lazr/restfulclient/tests/test_error
fullpath_noext == /usr/share/pyshared/lazr/restfulclient/tests/test_error

for this file:

lrwxrwxrwx 1 root root 71 Mai 26  2013 /usr/lib/python2.7/dist-packages/lazr/restfulclient/tests/test_error.py -> ../../../../../../share/pyshared/lazr/restfulclient/tests/test_error.py

Which is as expected in Debian/Ubuntu as the *.pyc file is a real file in /usr/lib/python2.7, but the *.py is symlinked to /usr/share/.

This new patch essentially enforces that the *.py file is not a symlink, which breaks the Debian-ish way of installing python 2 modules.

On ven., 2013-11-29 at 16:23 +0000, Martin Pitt wrote:
> This new patch essentially enforces that the *.py file is not a
> symlink, which breaks the Debian-ish way of installing python 2
> modules.

So it doesn't help that Debian/Ubuntu likes to put symlinks everywhere,
then... (the original issue is due to another peculiarity you've added)

So, how about the following algorithm:
- check that both paths are equal
- if they aren't, call realpath() and check again
- if they are still unequal, raise an error

I suppose this is 2.7-only, btw? In 3.x, __file__ points to the py file
and not the pyc file.

Yes, this affects python 2.7 only; as I said, this is all solved in python3 by introducing the explicit extensions like __pycache__/*..cpython-33.pyc so that multiple versions can share one directory. With that these symlink hacks aren't necessary any more.

> this affects python 2.7 only;