classification
Title: unittest discovery doesn't detect namespace packages when given no parameters
Type: behavior Stage: needs patch
Components: Documentation Versions: Python 3.10, Python 3.9, Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: Claudiu.Popa, Florian.Apolloner, Zbynek.Winkler, ashkop, barry, docs@python, eric.smith, eric.snow, ezio.melotti, mdjonyhossainhabib, methane, michael.foord, miss-islington, miss-islington, miss-islington, miss-islington, miss-islington, pconnell, rbcollins, rgammans, rgammans@gammascience.co.uk, terry.reedy
Priority: normal Keywords: patch

Created on 2015-04-07 11:36 by Florian.Apolloner, last changed 2021-07-08 16:36 by eric.smith.

Files
File name Uploaded Description Edit
console_log.txt Florian.Apolloner, 2015-04-07 11:36
issue_23882_test_case.sh ashkop, 2015-04-20 07:12
issue23882_find_all.patch ashkop, 2015-04-28 09:28 review
issue23882_find_one_level.patch ashkop, 2015-04-28 09:29 review
Pull Requests
URL Status Linked Edit
PR 21560 merged methane, 2020-07-20 09:14
PR 24619 merged miss-islington, 2021-02-22 06:14
PR 24619 merged miss-islington, 2021-02-22 06:14
PR 24619 merged miss-islington, 2021-02-22 06:14
PR 24619 merged miss-islington, 2021-02-22 06:14
PR 24620 merged miss-islington, 2021-02-22 06:14
PR 24621 closed miss-islington, 2021-02-22 06:15
Messages (27)
msg240205 - (view) Author: Florian Apolloner (Florian.Apolloner) Date: 2015-04-07 11:36
Unittest discovery does not seem to work if the tests package is  a namespace package. The attached file should have all details to reproduce, as soon as I readd an __init__.py everything works fine. 

Test was done using python3.4.2 and 3.4.3
msg240254 - (view) Author: Alex Shkop (ashkop) * Date: 2015-04-08 09:57
Spent some time looking into this one. Looks like the problem is in TestLoader.discover() method. There are couple of issues I found in it, all caused by same assumption.

Documentation [1] states that all test modules found by discover() method should be importable from top_level_dir. Whenever this method finds a subdirectory of start_dir it checks for __init__.py file. If there's no __init__.py then finder assumes that files within this package is not importable and stops recursion. This kind of 'importablity' check is not valid since we have namespace packages.

I'm not sure what should be done to fix this issue. We can change documentation to state that only regular packages with tests will be discovered. Or we can fix 'importability' checks, which will mean that all tests in all subdirectories will be discovered.

[1] https://docs.python.org/3.4/library/unittest.html#unittest.TestLoader.discover
msg240264 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2015-04-08 13:00
Is there any reason for unittest to not use pkgutil.iter_modules or pkgutil.walk_packages?  Either should work.
msg241331 - (view) Author: PCManticore (Claudiu.Popa) * (Python triager) Date: 2015-04-17 13:58
Isn't this already supported by the patch added in issue17457?
msg241334 - (view) Author: Alex Shkop (ashkop) * Date: 2015-04-17 14:25
Not fully. Patch for issue17457 fixed discovery when you explicitly specified namespace package as a target for discovery. I.e. when you run

python -m unittest namespace_pkg

But it didn't recurse into any namespace packages inside namespace_pkg. So when you run

python -m unittest

it doesn't go under all namespace packages (basically folders) in cwd.
msg241351 - (view) Author: Robert Collins (rbcollins) * (Python committer) Date: 2015-04-17 18:40
Hi. I'd be happy enough to use pkgutil helpers if they (like walkdirs) allowed trimming the output: part of the definition of discovery is that one can control it, to stop it importing or processing part of the tree that one doesn't want processed.

Alex: can you create a test case please - just a small script, python or shell or whatever, that will setup a demonstration of the bug? Thanks.
msg241619 - (view) Author: Alex Shkop (ashkop) * Date: 2015-04-20 07:12
This script creates following directory structure

issue_23882\
  tests\
    test_fail.py

If you run from issue_23882 directory

python -m unittest

it doesn't find any tests. If there is __init__.py inside tests folder, same command finds test_fail.py and runs it.

I'm not sure yet about using pkgutil cause looks like iter_modules and walk_packages do not find namespace packages as well.
msg241621 - (view) Author: Robert Collins (rbcollins) * (Python committer) Date: 2015-04-20 07:51
Ok, so here's whats happening:
the default behaviour is to do discovery of '.', which bypasses the namespace support code.

Running with tests as the first parameter works because it doesn't require the directory being directly scanned to be a package.

Equally, if the namespace isn't mapped to a local directory, 'python -m unittest tests' will load tests from the tests namespace.

So - this seems to be acting as designed. Namespace packages working fine.

However, I think what you're asking for is that namespace packages in your cwd are picked up and tested, without you having to supply the namespace name?

I think what we'd need for that to work sanely is a patch to teach loader.discover to determine the local namespaces which are present under start_dir. Right now this code:
        elif os.path.isdir(full_path):
            if (not namespace and
                not os.path.isfile(os.path.join(full_path, '__init__.py'))):
                return None, False

gets in your way: we assume that either we're processing a namespace, or there are no namespaces at all.

I think some more detangling of the generator could address this fairly cleanly.
msg241627 - (view) Author: Alex Shkop (ashkop) * Date: 2015-04-20 09:50
Thanks. I understand the code pretty well and I saw issue17457 that fixes discovery for explicitly specified namespace package.

What I need is to know how discovery has to work. Do we need to discover namespace packages inside discovery path? And should we do that recursively? I.e. should we pick namespace packages inside namespace packages? This will lead to recursive scan of all directories under discovery path.
msg242170 - (view) Author: Alex Shkop (ashkop) * Date: 2015-04-28 09:28
I'm still not sure which solution is the best. So I attach two simple patches.

First one enables full recursive scan of start_dir for namespace packages.

Second one loads tests only from top level namespace packages.
msg242203 - (view) Author: Robert Collins (rbcollins) * (Python committer) Date: 2015-04-28 19:02
Ah the user model? 

I think the following:

If I run 'python -m unittest' in a directory, then I expect to run all of the tests contained within that directory tree, and no others.

Does that definition help?
msg242213 - (view) Author: Alex Shkop (ashkop) * Date: 2015-04-29 06:13
Yes. That is how issue23882_find_all.patch works. I just removed hte condition

        if (not namespace and
            not os.path.isfile(os.path.join(full_path, '__init__.py'))):
            return None, False
 
This makes namespace parameter redundant. Can I remove it?
msg242967 - (view) Author: Alex Shkop (ashkop) * Date: 2015-05-12 12:57
Please, review the patch.
msg248935 - (view) Author: Robert Collins (rbcollins) * (Python committer) Date: 2015-08-21 00:57
reviewed in rietvald, but here too just in case.

The hunk that saves/restores _top_level_dir feels wrong to me - and not part of this bug, please remove it.

The rest of the patch is fine today.

But it also needs to add two specifically namespace tests: concretely we need the following cases covered:


a) namespace subdir/namespace subdir/test_foo.py gets loaded
b) namespace subdir/test_foo.py with another instance of the namespace subdir on sys.path also containing e.g. test_bar.py, test_bar.py is not loaded.
msg248936 - (view) Author: Robert Collins (rbcollins) * (Python committer) Date: 2015-08-21 00:57
(for the trivial case of CLI discover without a parameter - so translate that to the lower level API and then test that)
msg373925 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2020-07-19 03:38
I had rejected this idea in #29642. This is a copy of my comments in the issue.

---

I'm afraid this change makes testloader searches unrelated directory contains massive files (like node_modules).

I don't think loading all tests from whole namespace package is not usual use case.

When using import, (namespace) package name is explicitly specified.
Only specified name is searched.

In test loader's case, there are no such limit.
Loader may search millions of completely unrelated directories.
It may include directories in NFS or samba over slow network.
msg373926 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2020-07-19 03:42
I think people misunderstanding and misusing PEP 420, withouth knowing what is namespace package for.

I had wrote an article to describe namespace package is not a regular package.
https://dev.to/methane/don-t-omit-init-py-3hga
msg373927 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2020-07-19 03:55
Searching into directory without __init__.py recursively is not only inefficient, but also dangerous.

 project/
   - mylib/
      - __init__.py
      - foo.py
   - tests/
      - __init__.py
      - test_foo.py
   - tools/
      - bin/
          - dangerous_script.py

What happens if `python -m unittest` is run in the project root?
Who excepts tools/bin/dangarous.py is executed?

My conclution is:

* People shouldn't abuse PEP 420.  Omitting __init__.py is allowed only for namespace package.
* Namespace package should be searched based on PEP 420 rule. Don't search into regular directory unless it is specified.
msg373952 - (view) Author: Roger Gammans (rgammans@gammascience.co.uk) Date: 2020-07-19 10:06
But namespace packages are still useful for what PEP420 envisages and they should be able to have runnable tests.

For instance

 projectX/
   - interfaces/
         - proxyX.py
         - testX.py
 projectY/
   - intefaces/
         - proxyY.py
         - testY.py

Here interfaces is a namespace package and projectX and projectY are kept separate, perhaps to reduce dependency and/or for separation of concerns.

I don't think this is insurmountable - even taking into account Inada's very good point about dangerous_scripts. A full tests/ packages in on both projectX and projectY would allow their test to run. However, in some environments, it is normal to put the test alongside the code as shown here.

But some better documentation on this issue would advisable, and some guidance on the best practice.

Python has a complex ecosystem, and so an individual developer might hit this problem but be using a third party `framework`, which hasn't taken all of this on board. Eg. https://code.djangoproject.com/ticket/30403 . (I'm not really singling django out, although it's the one I hit) So it important that there is a clear understanding of the best way to deal with this case.
msg373961 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2020-07-19 11:29
In such case, both directories must be specified explicitly.

Test discovery doesn't know that a directory is namespace package or
namespace package. So it shouldn't assume it is namespace package unless
specified.

Note that PEP 420 searches namespace package only when it is specified.
msg387500 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2021-02-22 06:14
New changeset 5a4aa4c03e27ca5007b86c9c1ee62c77ad08a120 by Inada Naoki in branch 'master':
bpo-23882: Doc: Clarify unittest discovery document (GH-21560)
https://github.com/python/cpython/commit/5a4aa4c03e27ca5007b86c9c1ee62c77ad08a120
msg387501 - (view) Author: miss-islington (miss-islington) Date: 2021-02-22 06:25
New changeset 9dd018e35cce30bc2545290b6083dbf6e50d7b61 by Miss Islington (bot) in branch '3.8':
bpo-23882: Doc: Clarify unittest discovery document (GH-21560)
https://github.com/python/cpython/commit/9dd018e35cce30bc2545290b6083dbf6e50d7b61
msg387502 - (view) Author: miss-islington (miss-islington) Date: 2021-02-22 06:37
New changeset 30fe3ee6d39fba8183db779f15936fe64cc5ec85 by Miss Islington (bot) in branch '3.9':
bpo-23882: Doc: Clarify unittest discovery document (GH-21560)
https://github.com/python/cpython/commit/30fe3ee6d39fba8183db779f15936fe64cc5ec85
msg387532 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-02-22 18:57
From the merge:

+++ b/Doc/library/unittest.rst
@@ -330,7 +330,9 @@ Test modules and packages can customize test loading and discovery by through
 the `load_tests protocol`_.
 
 .. versionchanged:: 3.4
-   Test discovery supports :term:`namespace packages <namespace package>`.
+   Test discovery supports :term:`namespace packages <namespace package>`
+   for start directory. Note that you need to the top level directory too.
+   (e.g. ``python -m unittest discover -s root/namespace -t root``).
 
The last sentence is missing a verb after 'you need to' saying what to do about "the top level directory.  "be in"?  "go to"?  "later destroy"? (just kidding ;-)
msg387542 - (view) Author: Md Jony Hossain Habib (mdjonyhossainhabib) Date: 2021-02-22 20:14
diff -r 293d9964cf6e Lib/unittest/loader.py
--- a/Lib/unittest/loader.py	Tue Apr 28 00:04:53 2015 -0400
+++ b/Lib/unittest/loader.py	Tue Apr 28 10:12:07 2015 +0300
@@ -338,7 +338,7 @@
             raise ImportError('Start directory is not importable: %r' % start_dir)
 
         if not is_namespace:
-            tests = list(self._find_tests(start_dir, pattern))
+            tests = list(self._find_tests(start_dir, pattern, namespace=True))
         return self.suiteClass(tests)
 
     def _get_directory_containing_module(self, module_name):
@@ -403,7 +403,7 @@
                 name = self._get_name_from_path(full_path)
                 self._loading_packages.add(name)
                 try:
-                    yield from self._find_tests(full_path, pattern, namespace)
+                    yield from self._find_tests(full_path, pattern, False)
                 finally:
                     self._loading_packages.discard(name)
 
diff -r 293d9964cf6e Lib/unittest/test/test_program.py
--- a/Lib/unittest/test/test_program.py	Tue Apr 28 00:04:53 2015 -0400
+++ b/Lib/unittest/test/test_program.py	Tue Apr 28 10:12:07 2015 +0300
@@ -16,7 +16,7 @@
         expectedPath = os.path.abspath(os.path.dirname(unittest.test.__file__))
 
         self.wasRun = False
-        def _find_tests(start_dir, pattern):
+        def _find_tests(start_dir, pattern, namespace):
             self.wasRun = True
             self.assertEqual(start_dir, expectedPath)
             return tests
msg387549 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2021-02-23 02:31
I noticed that namespace package support has been broken since this commit.
https://github.com/python/cpython/commit/bbbcf8693b876daae4469765aa62f8924f39a7d2

Now namespace pacakge has __file__ attribute which is None. But...

                try:
                    start_dir = os.path.abspath(
                       os.path.dirname((the_module.__file__)))
                except AttributeError:
                    # look for namespace packages

`the_module.__file__` doesn't raise AttributeError for now. But os.path.dirname(None) raise TypeError.

The commit is backported to 3.7 branch. So namespace package support has been broken since Python 3.7.

Shouldn't we drop namespace package support?
It is misleading. And we could not maintain it. We didn't notice that it is broken for 3 years!
msg397150 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2021-07-08 16:36
As the author of PEP 420, I think having test discovery support namespace packages is a mistake, at least in the sense of pretending every directory is a namespace package.
History
Date User Action Args
2021-07-08 16:36:37eric.smithsetnosy: + eric.smith
messages: + msg397150
2021-02-23 02:31:59methanesetnosy: + barry
messages: + msg387549

resolution: fixed ->
stage: resolved -> needs patch
2021-02-22 20:14:28mdjonyhossainhabibsetnosy: + mdjonyhossainhabib
messages: + msg387542
2021-02-22 18:57:38terry.reedysetstatus: closed -> open
nosy: + terry.reedy
messages: + msg387532

2021-02-22 08:00:01methanesetstatus: open -> closed
stage: patch review -> resolved
resolution: fixed
versions: + Python 3.8, Python 3.9
2021-02-22 06:37:24miss-islingtonsetmessages: + msg387502
2021-02-22 06:25:01miss-islingtonsetmessages: + msg387501
2021-02-22 06:15:05miss-islingtonsetpull_requests: + pull_request23405
2021-02-22 06:14:59miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request23402
2021-02-22 06:14:59miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request23404
2021-02-22 06:14:59miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request23403
2021-02-22 06:14:58miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request23401
2021-02-22 06:14:49miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request23400
2021-02-22 06:14:42methanesetmessages: + msg387500
2020-07-20 09:14:55methanesetstage: needs patch -> patch review
pull_requests: + pull_request20706
2020-07-20 08:46:03methanesetassignee: docs@python

nosy: + docs@python
components: + Documentation
stage: patch review -> needs patch
2020-07-19 11:29:16methanesetmessages: + msg373961
2020-07-19 10:06:19rgammans@gammascience.co.uksetnosy: + rgammans@gammascience.co.uk
messages: + msg373952
2020-07-19 04:37:03methanelinkissue36723 superseder
2020-07-19 03:55:53methanesetmessages: + msg373927
2020-07-19 03:49:02methanelinkissue35617 superseder
2020-07-19 03:42:43methanesetmessages: + msg373926
2020-07-19 03:38:03methanesetnosy: + methane

messages: + msg373925
versions: + Python 3.10, - Python 3.5, Python 3.6
2020-07-19 03:35:55methanelinkissue29642 superseder
2020-03-24 09:37:47Zbynek.Winklersetnosy: + Zbynek.Winkler
2019-10-28 20:48:25pconnellsetnosy: + pconnell
2019-08-19 21:43:04rgammanssetnosy: + rgammans
2015-08-21 00:57:46rbcollinssetmessages: + msg248936
2015-08-21 00:57:17rbcollinssetmessages: + msg248935
2015-05-12 20:10:50ned.deilysetstage: needs patch -> patch review
2015-05-12 12:57:41ashkopsetmessages: + msg242967
2015-04-29 06:13:41ashkopsetmessages: + msg242213
2015-04-28 19:02:22rbcollinssetmessages: + msg242203
2015-04-28 09:29:03ashkopsetfiles: + issue23882_find_one_level.patch
2015-04-28 09:28:54ashkopsetfiles: + issue23882_find_all.patch
keywords: + patch
messages: + msg242170
2015-04-20 09:50:08ashkopsetmessages: + msg241627
2015-04-20 07:51:42rbcollinssettitle: unittest discovery and namespaced packages -> unittest discovery doesn't detect namespace packages when given no parameters
stage: needs patch
messages: + msg241621
versions: + Python 3.5, Python 3.6, - Python 3.4
2015-04-20 07:12:50ashkopsetfiles: + issue_23882_test_case.sh

messages: + msg241619
2015-04-17 18:40:21rbcollinssetmessages: + msg241351
2015-04-17 14:25:30ashkopsetmessages: + msg241334
2015-04-17 13:58:26Claudiu.Popasetnosy: + Claudiu.Popa
messages: + msg241331
2015-04-08 13:00:44eric.snowsetmessages: + msg240264
2015-04-08 09:57:53ashkopsetnosy: + ashkop
messages: + msg240254
2015-04-07 14:54:53eric.snowsetnosy: + eric.snow
2015-04-07 11:43:33ezio.melottisetnosy: + rbcollins, ezio.melotti, michael.foord
type: behavior
2015-04-07 11:36:35Florian.Apollonercreate