classification
Title: unittest loader barfs on symlinks
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.4, Python 3.3, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: acapnotic, barry, doko, michael.foord, pitrou, pitti, python-dev
Priority: normal Keywords: patch

Created on 2013-10-22 14:32 by pitrou, last changed 2014-02-06 23:58 by acapnotic.

Files
File name Uploaded Description Edit
unittest_loader_symlinks.patch pitrou, 2013-10-22 14:49
Messages (14)
msg200964 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-10-22 14:32
unittest.loader has the following snippet:

                    if realpath.lower() != fullpath_noext.lower():
                        module_dir = os.path.dirname(realpath)
                        mod_name = os.path.splitext(os.path.basename(full_path))[0]
                        expected_dir = os.path.dirname(full_path)
                        msg = ("%r module incorrectly imported from %r. Expected %r. "
                               "Is this module globally installed?")


Unfortunately, this will break with virtualenv on Ubuntu, which creates
a "local" directory full of symlinks. You end with this kind of error message:

======================================================================
ERROR: __main__ (unittest.loader.LoadTestsFailure)
----------------------------------------------------------------------
ImportError: 'test_asyncagi' module incorrectly imported from '/home/antoine/obelus/.tox/py27/local/lib/python2.7/site-packages/obelus/test'. Expected '/home/antoine/obelus/.tox/py27/lib/python2.7/site-packages/obelus/test'. Is this module globally installed?


Instead of (rather stupidly) calling lower(), realpath() and normcase()
should be called instead, to make sure the canonical paths are compared.
msg200965 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-10-22 14:49
Attaching patch.
msg200966 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2013-10-22 14:50
Good catch and fix Antoine. Thanks.
msg201046 - (view) Author: Roundup Robot (python-dev) Date: 2013-10-23 17:13
New changeset d7ec961cea1c by Antoine Pitrou in branch '2.7':
Issue #19352: Fix unittest discovery when a module can be reached through several paths (e.g. under Debian/Ubuntu with virtualenv).
http://hg.python.org/cpython/rev/d7ec961cea1c
msg201047 - (view) Author: Roundup Robot (python-dev) Date: 2013-10-23 17:16
New changeset a830cc1c0565 by Antoine Pitrou in branch '3.3':
Issue #19352: Fix unittest discovery when a module can be reached through several paths (e.g. under Debian/Ubuntu with virtualenv).
http://hg.python.org/cpython/rev/a830cc1c0565

New changeset ebbe87204114 by Antoine Pitrou in branch 'default':
Issue #19352: Fix unittest discovery when a module can be reached through several paths (e.g. under Debian/Ubuntu with virtualenv).
http://hg.python.org/cpython/rev/ebbe87204114
msg201048 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-10-23 17:17
Fixed!
msg204701 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2013-11-28 21:23
re-opening. the patch did break autopilot running with 2.7 on Ubuntu. I don't yet understand what this patch is supposed to fix.
msg204702 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2013-11-28 21:34
see https://launchpad.net/bugs/1255505
msg204710 - (view) Author: Martin Pitt (pitti) Date: 2013-11-29 07:58
More precisely, it broke unittest's discovery (not specific to autopilot). For any installed test, you now get:

$ python -m unittest discover lazr
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/lib/python2.7/unittest/__main__.py", line 12, in <module>
    main(module=None)
  File "/usr/lib/python2.7/unittest/main.py", line 94, in __init__
    self.parseArgs(argv)
  File "/usr/lib/python2.7/unittest/main.py", line 113, in parseArgs
    self._do_discovery(argv[2:])
  File "/usr/lib/python2.7/unittest/main.py", line 214, in _do_discovery
    self.test = loader.discover(start_dir, pattern, top_level_dir)
  File "/usr/lib/python2.7/unittest/loader.py", line 206, in discover
    tests = list(self._find_tests(start_dir, pattern))
  File "/usr/lib/python2.7/unittest/loader.py", line 287, in _find_tests
    for test in self._find_tests(full_path, pattern):
  File "/usr/lib/python2.7/unittest/loader.py", line 287, in _find_tests
    for test in self._find_tests(full_path, pattern):
  File "/usr/lib/python2.7/unittest/loader.py", line 267, in _find_tests
    raise ImportError(msg % (mod_name, module_dir, expected_dir))
ImportError: 'test_error' module incorrectly imported from '/usr/lib/python2.7/dist-packages/lazr/restfulclient/tests'. Expected '/usr/lib/python2.7/dist-packages/lazr/restfulclient/tests'. Is this module globally installed?

I reverted this patch in Ubuntu for now as a quickfix.

This might be specific how Debian installs Python 2 modules. Packages ship them in /usr/share/pyshared/<modulename>/, and for every supported 2.X version, ship /usr/lib/python2.X/dist-packages/<modulename>/ which contains only the directories and *.pyc files, but symlinks the *.py files to /usr/share/pyshared/<modulename>/../*.py

So, it might be that upstream does not support this symlink layout, but it's the best thing to avoid having to install multiple *.py copies (this is all solved in a much more elegant way with Python 3, of course). But it would be nice if unittest's discovery could still cope with this.

Thanks!
msg204730 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-11-29 15:29
> ImportError: 'test_error' module incorrectly imported from '/usr/lib/python2.7/dist-packages/lazr/restfulclient/tests'. Expected '/usr/lib/python2.7/dist-packages/lazr/restfulclient/tests'. Is this module globally installed?

Well, this looks like the same path to me. Can you investigate a bit?
msg204732 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2013-11-29 15:44
This can happen when the code used to compare the paths is different from the code used to generate the failure message. This of course masks the real problem. I can look at where that might be possible and then hopefully we can get a genuine error message telling us the problem.
msg204738 - (view) Author: Martin Pitt (pitti) Date: 2013-11-29 16:23
In this new code:

                    mod_file = os.path.abspath(getattr(module, '__file__', full_path))
                    realpath = os.path.splitext(os.path.realpath(mod_file))[0]
                    fullpath_noext = os.path.splitext(os.path.realpath(full_path))[0]

we get

modfile == /usr/lib/python2.7/dist-packages/lazr/restfulclient/tests/test_error.pyc
realpath == /usr/lib/python2.7/dist-packages/lazr/restfulclient/tests/test_error
fullpath_noext == /usr/share/pyshared/lazr/restfulclient/tests/test_error

for this file:

lrwxrwxrwx 1 root root 71 Mai 26  2013 /usr/lib/python2.7/dist-packages/lazr/restfulclient/tests/test_error.py -> ../../../../../../share/pyshared/lazr/restfulclient/tests/test_error.py

Which is as expected in Debian/Ubuntu as the *.pyc file is a real file in /usr/lib/python2.7, but the *.py is symlinked to /usr/share/.

This new patch essentially enforces that the *.py file is not a symlink, which breaks the Debian-ish way of installing python 2 modules.
msg204740 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-11-29 16:52
On ven., 2013-11-29 at 16:23 +0000, Martin Pitt wrote:
> This new patch essentially enforces that the *.py file is not a
> symlink, which breaks the Debian-ish way of installing python 2
> modules.

So it doesn't help that Debian/Ubuntu likes to put symlinks everywhere,
then... (the original issue is due to another peculiarity you've added)

So, how about the following algorithm:
- check that both paths are equal
- if they aren't, call realpath() and check again
- if they are still unequal, raise an error

I suppose this is 2.7-only, btw? In 3.x, __file__ points to the py file
and not the pyc file.
msg204741 - (view) Author: Martin Pitt (pitti) Date: 2013-11-29 16:58
Yes, this affects python 2.7 only; as I said, this is all solved in python3 by introducing the explicit extensions like __pycache__/*..cpython-33.pyc so that multiple versions can share one directory. With that these symlink hacks aren't necessary any more.
History
Date User Action Args
2014-02-06 23:58:01acapnoticsetnosy: + acapnotic
2013-11-29 16:58:32pittisetmessages: + msg204741
2013-11-29 16:52:03pitrousetmessages: + msg204740
2013-11-29 16:23:54pittisetmessages: + msg204738
2013-11-29 15:44:37michael.foordsetmessages: + msg204732
2013-11-29 15:29:34pitrousetmessages: + msg204730
2013-11-29 07:58:58pittisetnosy: + pitti
messages: + msg204710
2013-11-28 21:34:59dokosetmessages: + msg204702
2013-11-28 21:23:28dokosetstatus: closed -> open

nosy: + doko
messages: + msg204701

resolution: fixed ->
2013-10-23 17:17:15pitrousetstatus: open -> closed
resolution: fixed
messages: + msg201048

stage: needs patch -> resolved
2013-10-23 17:16:41python-devsetmessages: + msg201047
2013-10-23 17:13:04python-devsetnosy: + python-dev
messages: + msg201046
2013-10-22 14:50:14michael.foordsetmessages: + msg200966
2013-10-22 14:49:12pitrousetfiles: + unittest_loader_symlinks.patch
keywords: + patch
messages: + msg200965
2013-10-22 14:32:55pitroucreate